Time-varying properties of asymmetric volatility and multifractality in
Bitcoin

Tetsuya Takaishi1*,

1 Hiroshima University of Economics, Hiroshima, Japan

*<EMAIL_ADDRESS>
## Abstract

This study investigates the volatility of daily Bitcoin returns and
multifractal properties of the Bitcoin market by employing the rolling window
method and examines relationships between the volatility asymmetry and market
efficiency. Whilst we find an inverted asymmetry in the volatility of Bitcoin,
its magnitude changes over time, and recently, it has become small. This
asymmetric pattern of volatility also exists in higher frequency returns.
Other measurements, such as kurtosis, skewness, average, serial correlation,
and multifractal degree, also change over time. Thus, we argue that properties
of the Bitcoin market are mostly time dependent. We examine efficiency-related
measures: the Hurst exponent, multifractal degree, and kurtosis. We find that
when these measures represent that the market is more efficient, the
volatility asymmetry weakens. For the recent Bitcoin market, both efficiency-
related measures and the volatility asymmetry prove that the market becomes
more efficient.

##  1 Introduction

Bitcoin, advocated by Satoshi Nakamoto[1], was launched in 2009 as the first
decentralized cryptocurrency. Its system is based on a peer-to-peer network.
Whilst many other cryptocurrencies have been created since its launch, and the
cryptocurrency market has grown rapidly, Bitcoin remains the dominant
cryptocurrency in terms of market capitalization. Fig 1 represents the market
capitalizations of the largest 10 cryptocurrencies. Bitcoin dominates about
70% of the total capitalization of 10 cryptocurrencies.

Fig 1:  Market capitalization of the largest 10 cryptocurrencies (as of 31
August). The solid line represents a Pareto chart. The data are taken from
”https://coinmarketcap.com/.”

In recent years, Bitcoin has attracted interest of many researchers. Various
aspects of Bitcoin, including hedging capabilities[2], bubbles[3], liquidity
and efficiency[4], Taylor effect[5], structural breaks [6], transaction
activity[7], complexity synchronization[8], long memory effects[9], price
clustering[10], rough volatility[11] power-law cross-correlation[12], market
structure[13] have been investigated.

Similar to other assets, stylized facts[14, 15], such as volatility
clustering, fat-tailed return distribution, and long memory in absolute
returns, are observed in Bitcoin (e.g., [16, 17]). An aggregational
Gaussianity that the fat-tailed return distributions change to the Gausian
distribution on large time scales is another stylized fact observed in various
assets[14, 15]. This aggregational Gaussianity is also observed in Bitcoin and
a minimum time scale required to recover the Gaussianity is estimated to be
two weeks[17]. The skewness is negative on a short time scale, but it moves to
zero on a larger time scale[17].

Despite the similar stylized facts in Bitcoin, researchers report a distinct
property: inverted volatility asymmetry. By using generalized autoregressive
conditional heteroscedasticity (GARCH) models[18, 19, 20, 21], several studies
report an ”inverted asymmetry” in which volatility reacts more to positive
returns than negative ones[22, 23, 24, 25]. This contrasts sharply with the
observation that the volatility of stocks reacts to negative returns more than
positive ones[26, 27, 28].

However, some studies report that there is no significant asymmetry in
volatility [29, 17]. An opposite result to the inverted asymmetry, that is,
the same volatility reaction as stocks, is documented[30]. Moreover, whilst
inverted asymmetry is also observed in other cryptocurrencies[31, 32], the
asymmetry in Bitcoin is insignificant[31]. Therefore, a consistent picture of
volatility asymmetry in Bitcoin has not been obtained. We infer that the
discrepancy observed in the volatility asymmetry is caused, in part, by the
time-varying property of volatility asymmetry. We attribute the differing
conclusions to the various data periods used in earlier studies. The
possibility of time-varying asymmetry has been already noted[22], and it has
been reported that whilst inverted asymmetry is observed before 2014, no
significant asymmetry is observed after 2014.

This study aims to investigate time-varying properties of the Bitcoin market,
especially volatility asymmetry. In emprical finance, one of popular models to
analyze volatility is the GARCH model[18, 19] which can successfully capture
some stylized facts such as volatility clustring and fat-tailed distribution.
There exist many variants of the GARCH model designed to capture more
properties of financial time series[33]. To assess the volatility asymmetry,
we use the threshold GARCH (TGARCH) model[21], which is widely accepted in
empirical finance and has been used in the previous studies on the Bitcoin
volatility asymmetry[22, 30, 31, 17, 25, 34, 32]. We estimate model parameters
by employing the rolling window method, which enables us to see time
variation. The rolling window method is commonly used in econometrics or
empirical finance for time-series analysis in a limited amount of financial
data. We also investigate the Hurst exponent and multifractality of the
Bitcoin market by the multifractal detrended fluctuation analysis (MF-DFA)
method[35], which is a powerful method to study multifractal properties and
has been applied for various assets in econophysics; see, for example, [36].
The Hurst exponent and multifractality of the Bitcoin time series are examined
intensively in connection with the market efficiency (e.g., [37, 38, 39, 40,
41, 17, 42, 43, 44, 45]). Here note that according to the efficient market
hypothesis[46], there are three types of market efficiencies: weak, semi-
strong, and strong forms. Since we use the time-series data only, the market
efficiency in this study means the weak-form market efficiency.

A remarkable feature that the Bitcoin time series exhibits is presence of
anti-persistency, that is, the Hurst exponent less than 1/2 [37]. The anti-
persistency means that the time series reverses its moving direction more
often than a random time series. On the other hand, the time series with the
Hurst exponent greater than 1/2 persists the same moving direction more than a
random time series. This anti-persistency behavior, however, turns out to be
temporary. It is observed that the Hurst exponent and the multifractality
degree vary over time, and the anti-persistency appears repeatedly[40, 44].
Then, it seems that the Hurst exponent approaches the value of 0.5, which
might be an indication toward a maturity market[37, 47].

Since the efficient market should be free from any type of volatility
asymmetry that results in predicting a certain market property to help gaining
profits, the volatility asymmetry is expected to induce some inefficiency and
to relate with efficiency-related measures such as multifractality. Thus, we
also examine a possible relationship between volatility asymmetry and
efficiency-related measures. We find that efficiency-related measures are
related to the volatility asymmetry. When the efficiency-related measures
indicate that the market is more efficient, the volatility asymmetry weakens.

To fully understand the dynamics of Bitcoin time series, we need to
investigate various aspects of Bitcoin. This study investigates not only the
volatility asymmetry but also the multifractality, and combines them to
advance the understanding of properties of the Bitcoin market. Our results
reveal that the Bitcoin market efficiency has improved in recent years. It has
been claimed that the market efficiency is related with market size and
economic development[48]. In accordance with this, our results suggest that
the Bitcoin market is more mature than ever.

The rest of this paper is organized as follows. Section 2 describes the
methodology. In Section 3, we describe the data, and in Section 4, we present
the empirical results and discuss the results. Finally, we conclude our study
in Section 5.

##  2 Methodology

Let $p_{t_{i}};t_{i}=i\Delta t;i=0,1,2,...,N$ be the Bitcoin price time series
with sampling period $\Delta t$. We define the return $r_{t_{i+1}}$ by the
logarithmic price difference:

$r_{t_{i+1}}=100\times(\log p_{t_{i+1}}-\log p_{t_{i}}).$ (1)

Since in empirical finance the volatility analysis using the GARCH-type models
mainly focuses on daily volatility, here we also focus on daily returns, i.e.
$\Delta t=1440-min$.

The TGARCH model[21] is used to investigate volatility asymmetry in the
Bitcoin market; the return $r_{t_{i}}$ and the volatility $\sigma^{2}_{t_{i}}$
at ${t_{i}}$ are modelled as follows:

$\displaystyle r_{t_{i}}$ $\displaystyle=$
$\displaystyle\mu+c_{1}r_{t_{i-1}}+\epsilon_{t_{i}},$ (2)
$\displaystyle\sigma^{2}_{t_{i}}$ $\displaystyle=$
$\displaystyle\omega+\alpha\epsilon_{t_{i-1}}+\beta\sigma^{2}_{t_{i-1}}+\gamma\epsilon_{t_{i-1}}^{2}I(\epsilon_{t_{i-1}}),$
(3)

where $\epsilon_{t_{i}}$ is defined by
$\epsilon_{t_{i}}=\sigma_{t_{i}}\eta_{t_{i}}$, and $I(\epsilon_{t_{i-1}})$ is
an indicator function, implying that it is 1 if $\epsilon_{t_{i-1}}<0$ and 0
otherwise. $\eta_{t_{i}}$ is an unobservable random variable from an
independent and identically distributed (IID) process. Here, we use the
Student t distribution as an IID process. To check the robustness on choice of
distributions, we use the normal distribution and the generalized error
distribution[20].

It is empirically well-known that stock return volatility increases after
negative returns more than positive returns[26, 27]. This volatility asymmetry
is called ”the leverage effect” and causes a negative correlation between
stock returns and volatility. To capture the leverage effect, various GARCH-
type models with the volatility asymmetry are introduced, e.g. [21, 20, 49,
50, 51]. For the TGARCH model, the volatility asymmetry is measured by the
$\gamma$ parameter in Eq (3), and when the leverage effect exists, the
$\gamma$ parameter takes a positive value. On the other hand, for the inverted
volatility asymmetry observed in the Bitcoin market the $\gamma$ parameter
takes a negative value and volatility reacts more to positive returns than
negative ones, leading to the inverted volatility asymmetry. $c_{1}$ is the
coefficient of an autoregressive model of order 1 (AR(1)) that captures the
serial correlation.

To investigate the time-varying properties of volatility asymmetry, we use the
rolling window method to estimate the model parameters. For the parameter
estimation, we use the ”urgarch” package of R. First, we set a window size of
548 days ($\approx$ one and a half years). We choose this window size because
the smaller window size such as one year leads to more noisy results in
estimating model parameters, and perform a parameter estimation for the window
containing the first 548 data samples for the time series. Next, we shift the
window 30 days ($\approx$ one month) and perform a parameter estimation for
the data in the next window. We repeat this process until the end the time
series.

Multifractal analysis is a useful method to quantify properties of complex
system, and it has been applied in many different fields, e.g.[52, 53, 54, 55,
56]. Multifractal analysis is also popular in studies of financial markets and
multifractal properties have been intensively studied[56]. To investigate the
multifractal properties of the Bitcoin market, we apply the MF-DFA method[35].
The MF-DFA method is described by the following steps.

(i) Determine the profile $Y(i)$.

$Y(i)=\sum_{j=1}^{i}(r_{t_{j}}-\langle r\rangle),$ (4)

where $\langle r\rangle$ stands for the average of returns.

(ii) Divide the profile $Y(i)$ into $N_{s}$ non-overlapping segments of equal
length $s$, where $N_{s}\equiv{int}(N/s)$. Since the length of the time series
is not always a multiple of $s$, a short time period at the end of the profile
may remain. For this part, the same procedure is repeated starting from the
end of the profile. Therefore, total $2N_{s}$ segments are obtained.

(iii) Calculate the variance.

$F^{2}(\nu,s)=\frac{1}{s}\sum_{i=1}^{s}(Y[(\nu-1)s+i]-P_{\nu}(i))^{2},$ (5)

for each segment $\nu,\nu=1,...,N_{s}$ and

$F^{2}(\nu,s)=\frac{1}{s}\sum_{i=1}^{s}(Y[N-(\nu-N_{s})s+i]-P_{\nu}(i))^{2},$
(6)

for each segment $\nu,\nu=N_{s}+1,...,2N_{s}$. Here, $P_{\nu}(i)$ is the
fitting polynomial to remove the local trend in segment $\nu$; we use a cubic
order polynomial.

(iv) Average over all segments and obtain the $q$th order fluctuation
function.

$F_{q}(s)=\left\\{\frac{1}{2N_{s}}\sum_{\nu=1}^{2N_{s}}(F^{2}(\nu,s))^{q/2}\right\\}^{1/q}.$
(7)

For $q=0$, the averaging procedure in Eq(7) cannot be directly applied.
Instead, we employ the following logarithmic averaging procedure.

$F_{0}(s)=\exp\left[\frac{1}{4N_{s}}\sum_{\nu=1}^{2N_{s}}\ln(F^{2}(\nu,s))\right].$
(8)

(v) Determine the scaling behavior of the fluctuation function. If the time
series $r_{t_{i}}$ are long-range power-law correlated, $F_{q}(s)$ is expected
to be the following functional form for large $s$:

$F_{q}(s)\sim s^{h(q)}.$ (9)

In calculating the fluctuation function $F_{q}(s)$, we take $q$ varying
between -25 and 25, with a step of 0.2. The scaling exponent $h(q)$ is called
the generalized Hurst exponent, and the usual Hurst exponent is given by
$h(2)$. When $h(q)$ is constant, the time series is called ”monofractal.” For
example, the random Gaussian time series is monofractal[35]. On the other
hand, when $h(q)$ varies depending on $q$, the time series is called
”multifractal.”

Following [57], we define the multifractality degree $\Delta h(q)$ by

$\Delta h(q)=h(q_{min})-h(q_{max}),$ (10)

where $q_{min}=-q$ and $q_{max}=q$.

Since for the random Gaussian time series, $\Delta h(q)$ takes zero, the
magnitude of $\Delta h(q)$ is expected to relate with the strength of the
market inefficiency.

We also calculate the singularity spectrum $f(\alpha)$, which is another way
to characterize a multifractal time series. It is defined by

$\alpha(q)=h(q)+qh^{\prime}(q),$ (11) $f(\alpha)=q[\alpha-h(q)]+1,$ (12)

where $\alpha(q)$ is the Hölder exponent or singularity strength[35]. The
multifractality degree through $f(\alpha)$ is defined by

$\Delta\alpha(q)=\alpha(q_{min})-\alpha(q_{max}),$ (13)

which takes zero for monofractal. In this study we take $q=[-25,25]$. However,
there is a caution that when $|q|$ is large, the moments may diverse and the
estimated results are unreliable[36]. Thus, to avoid such a situation,
following [47], for the calculations of the multifractal degree $\Delta h(q)$
and $\Delta\alpha(q)$, we take $q=4$. The strength of $\Delta\alpha(q)$ is
also related with the market inefficiency, and we call $\Delta h(q)$ and
$\Delta\alpha(q)$ including kurtosis ”efficiency-related measures.” We use the
same rolling window method to investigate time-varying properties of
multifractality and take the same window size (548 days) with the volatility
analysis by the TGARCH model to compare the results.

##  3 Data

We use Bitcoin tick data (in dollars) traded on Bitstamp from September 10,
2011, to June 06, 2020, downloaded from Bitcoincharts
(http://api.bitcoincharts.com/v1/csv/). Due to a hacking incident, no data are
available from January 4, 2015 to January 9, 2015. For these missing data, we
treat them as the price is unchanged.

###  3.1 Price and return

Fig 2 illustrates daily ($\Delta t=1440$-min) price and returns $r_{t_{j}}$
constructed from the Bitcoin tick data. We eliminate the data that are larger
than 40, i.e., $r_{t_{i}}>40$, as outliers. This manipulation keeps the
results almost unchanged except kurtosis. In our data set, we find four
outliers. Fig 2(d) shows the volatility series $s_{t}$ defined by
$s_{t}=s_{t-1}+|r_{t}|-\bar{r}$, where $r_{t}$ and $\bar{r}$ stand for the
return at $t$ and the average of $r_{t}$, respectively. The volatility series
introduced in [58, 59] can be utilized to identify the volatility clustering.
Namely, the increasing (decreasing) trend of the volatility series implies the
existence of the high (low) volatility clustering. Such trends indicating high
and low volatility clusterings are seen in Fig 2(d).

Fig 2:  (a): Daily price, (b): daily price (semi-log plot), (c): returns and
(d) volatility series.

Table 1 provides descriptive statistics for the whole sample of returns, and
we find a positive average, high kurtosis, and negative skewness. We also
explore the time variation in these using the rolling window method.

Table 1:  Descriptive statistics for the whole sample of daily returns.
SD stands for ”standard deviation.” The values in parentheses indicate one sigma errors estimated by the Jackknife method.  Mean | SD | Kurtosis | Skewness | Nobs
---|---|---|---|---
0.27(11) | 4.63(41) | 11.9(22) | -0.267(99) | 3188

###  3.2 Time-varying properties of descriptive statistics

Fig 3 illustrates the average, standard deviation (SD), kurtosis, and skewness
calculated with a 548-day rolling window. Interestingly, they vary
considerably over time. Whilst the kurtosis before 2017 is very high (i.e.,
more than 10), it decreases after 2017. Recently, it has taken a value of
around 6, which is still higher than the Gaussian kurtosis. The origin of high
kurtosis could be a fat-tailed return distribution that means higher price
variations are observed more often. At the early stage of Bitcoin market, the
tail index $\mu$ of the cumulative return distribution is found to be
$\mu\approx 2$, which is referred to as the inverse square law[60]. The
similar tail indces are have also been reported in [61] This is sharply
contrast to the well-known inverse cubic law for other assets[62, 63, 64, 65],
in which the tail index $\mu$ is $\mu\approx 3$. The inverse square law
observed in the Bitcoin market, however, is not permanent. The recent Bitcoin
data up to 2017 show that the tail index comes close to 3, which suggests that
the Bitcoin market is becoming more mature[47]. Further studies[66, 67] also
indicate the change of the tail index to 3. It is also worth noting that the
recent COVID-19 pandemic considerably affects the cryptocurrency market and as
a result the market experiences a volatile period in which the tail index
varies[13].

These observations imply that the tail index $\mu$ of the cumulative return
distribution in the Bitcoin market varies over time and moves from $\mu\approx
2$ (fatter tail) to $\mu\approx 3$ (thinner tail), which could account for the
decrease in kurtosis. Although the skewness is mostly negative over the whole
period, the magnitude of the negative skewness decreases gradually with time,
and the skewness seems to disappear in recent returns. The disappearance of
skewness possibly means the efficiency improvement of the Bitcoin market, and
agrees with the observation in efficiency-related measures such as the Hurst
exponent and the multifractal degree that indicate that the market efficiency
of the Bitcoin market varies over time and the recent market efficiency is
being improved. We describe the results on the market efficiency in more
detail later. Overall, these findings in descriptive statistics also suggest
that properties of the Bitcoin market vary over time. Thus, it is important to
consider time variation in analysis.

Fig 3:  Fig 3. Average, SD, kurtosis, and skewness as a function of time. SD
stands for standard deviation. These are calculated using the rolling window
method with a window size of 548 days. Bars in the data points represent one
sigma error, estimated by the Jackknife method.

###  3.3 Aggregational Gaussianity

Fig 4 illustrates the aggregational Gaussianity of returns, namely the
kurtosis of returns sampled at $\Delta t$ as a function of $\Delta t$. The
figure is plotted in log-log scale, and we find that the kurtosis decreases
according to a power-law up to $\Delta t\approx 20000$ min $\approx$ two
weeks, with an exponent $\sim-0.62$, and the kurtosis of returns at $\Delta t$
longer than two weeks is consistent with the Gaussian kurtosis, in agreement
with the previous result[17]. This finding suggests that the time series of
returns at $\Delta t$ longer than two weeks most likely becomes the random
Gaussian time series.

Fig 4:  Kurtosis at various sampling period $\Delta t$ as a function of
$\Delta t$. Bars in the data points represent one sigma error, estimated by
the Jackknife method. The red line displays the Gaussian kurtosis ($=3$).

##  4 Empirical results and discussion

Table 2 presents the TGARCH parameters estimated for the whole dataset. The
$\gamma$ parameter is negative, which indicates an inverted asymmetry in the
volatility. Its magnitude, however, is not large, which is consistent with the
result of [31]. This is probably because the strength of time-varying
asymmetry of $\gamma$ weakens in the parameter estimation for the whole
dataset which may include both periods of positive and negative asymmetries.
In the following, we find that the $\gamma$ parameter varies over time
considerably.

Table 2:  TGARCH parameter estimates for the whole dataset. $\nu$ is the shape
parameter of the Student t distribution.
The values in parentheses indicate standard errors.  $\alpha$ | $\beta$ | $\omega$ | $\gamma$ | $c_{1}$ | $\mu$ | $\nu$
---|---|---|---|---|---|---
0.181(36) | 0.853(17) | 0.235(77) | -0.058(21) | -0.044(17) | 0.181(36) | 3.21(14)

Figs 5 and 6 show the results of the TGARCH parameters in the 548-day rolling
window. It is evident that the parameters vary over time. Although parameter
$\gamma$ is mostly negative, we find some exceptions. For instance, parameter
$\gamma$ takes positive or zero values around 2015; and after 2019, its
magnitude becomes small or consistent with zero. We also observe a strong
inverted asymmetry between 2016 and 2018. For the robustness check on the IID
distribution in Eq (2), we perform the parameter estimation with the normal
and generalized error distributions, and find that the similar asymmetric
volatility patterns to that from the Student t distribution are obtained.
Therefore, the choice of distributions in the IID process is irrelevant.

Fig 5:  Estimation results of $\alpha,\beta$, and $\omega$. The error bars
show the standard errors.  Fig 6:  Estimation results of $\gamma,c_{1}$ and
$\mu$. The error bars show the standard errors.

The AR(1) parameter $c_{1}$, which captures serial correlation, also varies
considerably. It is argued that non-zero serial correlation implies that
uninformed investors dominate in trading and that price changes due to
uninformed investors will increase volatility more than price changes caused
by informed investors[68]. In line with [68], it is claimed that non-zero
AR(1) coefficients are found for cryptocurrencies; and the inverted asymmetry
due to uniformed investors is consistent with phenomena such as fear of
missing out, pump and dump schemes, and the disposition effect[31]. Our
results of $c_{1}$ indicate that there are periods in which $c_{1}$ is
consistent with zero, which suggests that the Bitcoin market is not always
dominated by uninformed investors. We find both strong inverted asymmetry and
non-zero $c_{1}$ from 2016 to 2018. Thus, the period from 2016 to 2018 is
considered to be dominated by uninformed investors, thereby affecting the
volatility asymmetry. As seen in Fig 2, in this period, the Bitcoin price
increases considerably and recorded the highest value on December 2017.
Consequently, Bitcoin price movement in this period is associated with the
inverted asymmetry induced dominantly by uninformed investors.

We also estimate the model parameters for high-frequency returns (6h and 12h).
Fig 7 represents the results of $\gamma$ together with those from the daily
returns. We find that high-frequency returns exhibit similar variation with
daily returns except that the 6h returns for which no significant inverted
asymmetry is seen before 2015. These results imply that whilst the asymmetric
volatility pattern remains for higher frequency returns, the detail of the
asymmetry pattern depends on the frequency of returns.

Fig 7:  Asymmetry parameter $\gamma$ for 6h, 12h, and daily returns. The error
bars show the standard errors.

To calculate the multifractal degree $\Delta h(q)$ and $\Delta\alpha(q)$, we
first determine $h(q)$ by fitting the fluctuation function $F_{q}(s)$ to Eq(9)
in a range of $s=[20,100]$. Then, we determine $\Delta h(q)$ and
$\Delta\alpha(q)$ by Eq(10) and (13), respectively. As representatives, Fig 8
(a)-(b) shows fluctuation functions $F_{q}(s)$ calculated using the first
window data and $h(q)$, respectively.

Fig 8:  (a) Fluctuation functions $F_{q}(s)$ (b) The generalized Hurst
exponent $h(q)$. In Fig 8 (a), the results are plotted from $q=-25$ (bottom)
to $q=25$ with a step of 1.0.

Fig 9 (a) represents the Hurst exponent $h(2)$ as a function of time, showing
some anti-persistent periods ($h(2)<1/2$). The recent $h(2)$ around 2020 is
consistent with or slightly above 1/2, which suggests that the recent Bitcoin
market is becoming more efficient. This finding is consistent with the results
of kurtosis and skewness in Fig 3 that shows low kurtosis and insignificant
skewness for the recent Bitcoin market, meaning that the return distribution
becomes more Gaussian shaped than before. For $\Delta h(q)$ and
$\Delta\alpha(q)$, we show the results of $q=4$ in Fig 9 (b) as a function of
time. Two measures of the multifractal degree calculated here show the same
time-varying pattern. Broadly speaking, the multifractal degree decreases with
time, which suggests that the Bitcoin market gradually approaches the
efficient market. Using the Amihud illiquidity measure[69], it is argued that
the market inefficiency in the cryptocurrency market is caused by illiquidity
and the illiquidity is related with anti-persistency[4]. For the Bitcoin
market, liquidity in terms of the Amihud illiquidity measure turns out to be
improving[44], which agrees with the improved efficiency of the Bitcoin market
in recent years.

Fig 9:  (a) the Hurst exponent $h(2)$, (b) multifractal degrees $\Delta h(4)$
and $\Delta\alpha(4)$. These results are obtained by the rolling window method
with a 548-day window and a step of 1 day.

In Fig 10 (a)-(b), we examine the relationship between kurtosis and the
efficiency-related measures (a) $h(2)$ and (b) $\Delta\alpha(4)$. Fig 10 (a)
shows that the results of $h(2)$ near 1/2 takes kurtosis smaller values, for
example, less than 7. However, most larger kurtoses more likely correspond to
$h(2)$ far from 1/2. In Fig 10 (b), we find that the smaller kurtoses opt to
take small $\Delta\alpha(4)$. Therefore, the results of Fig 10 (a)-(b) imply
that in the more efficient market, the kurtosis takes smaller values close to
the Gaussian one. Here, note that since returns at short scale are usually
fat-tailed distributions, the kurtosis of daily returns may not reach the
Gaussian one, and it rather reaches a certain minimum value ($>3$) at the
efficiency-improved market.

Fig 10:  (a) Kurtosis versus $h(2)$ and (b) Kurtosis versus $\Delta\alpha(4)$.

Fig 11 (a)-(c) illustrates relationships between the asymmetric parameter
$\gamma$ and three measurements, (a) $h(2)$, (b) $\Delta\alpha(4)$, and (c)
kurtosis, and some remarks are in order. In Fig 11 (a), it is observed that in
an anti-persistent domain ($h(2)<1/2$), the asymmetric parameter $\gamma$
takes negative value more than positive ones, and the parameter $\gamma$ near
zero tends to cluster in a region that $h(2)$ takes values near 1/2 or
slightly above 1/2.

Fig 11:  Relationships between $\gamma$ and (a) $h(2)$, (b) $\Delta\alpha(4)$,
and (c) kurtosis.

In Fig 11 (b), we find that in a region of small $\Delta\alpha(4)$, for
example, $\Delta\alpha(4)<0.5$, the parameter $\gamma$ mostly takes values
near zero. However, for larger $\Delta\alpha(4)$, the parameter $\gamma$ tends
to take negative values. Fig 11 (c) indicates that for a region near the
Gaussian kurtosis (e.g., kurtosis $<7$), the parameter $\gamma$ comes to
values near zero. Moreover, mostly the strong negative $\gamma$ comes to a
region with higher kurtoses. The overall results from Fig 10 (a)-(c) indicate
that for the market being more efficient, that is, for $h(2)$ near 1/2, small
$\Delta\alpha(4)$, and kurtosis closer to the Gaussian one, the volatility
asymmetry likely disappears. This is consistent with the consequence of the
efficient market that any predictable patterns such as the asymmetric
volatility should not exist.

###  4.1 An application on investment strategy

Correct understanding of the market state is of great importance for investors
who change trading strategy depending on the state of the market. This study
could contribute to offer such information. For example, according to the
efficient market hypothesis[46], the technical analysis is not supported on
the efficient market. Our results imply that the recent Bitcoin market is
being efficient and it might be difficult to make high profits by the
technical analysis. By monitoring the market efficiency of Bitcoin, if the
Bitcoin market becomes inefficient substantially again, one could use the
technical analysis to gain profits. Another suggestion from the efficient
market hypothesis is that on the efficient market, the most efficient
portfolio is a market portfolio consisting of every asset weighted in
proportion to its market capitalization. Of course, it is difficult to form a
completely diversified portfolio in practice[70]. However, it may be possible
to make an approximate diversified portfolio including the efficient Bitcoin.
Furthermore, although not all cryptocurrencies are efficient, when the
cryptocurrency markets become more mature and more efficient, an index or
portfolio consisting of cryptocurrencies could be a proxy of a fully
diversified portfolio on the cryptocurrency markets.

##  5 Conclusions

We use the rolling window method to investigate time-varying properties of
Bitcoin. We find that various measurements, such as volatility asymmetry,
kurtosis, skewness, serial correlation, and multifractality, are time varying.
Thus, the Bitcoin market may have inherently variable properties. Although the
inverted asymmetry is observed in Bitcoin and the strong inverted asymmetry is
found around 2016-2018, the recent volatility asymmetry is weak. The magnitude
of the volatility asymmetry may relate with the market state, especially the
market efficiency.

To investigate a relationship between the volatility asymmetry and the market
efficiency, we examine efficiency-related measures: the Hurst exponent,
multifractal degree, and kurtosis. We find that when these efficiency-related
measures indicate that the market is more efficient, the volatility asymmetry
is more likely to weaken. The efficiency-related measures indicate that the
recent Bitcoin market has become more efficient.

Whilst we use one of the GARCH-type models, that is, TGARCH model, which is
commonly used in volatility analysis of Bitcoin, other types of volatility
models such as stochastic volatility model[71, 72, 73] exist. It might be
interesting to investigate whether these models lead to the similar results on
the volatility asymmetry.

In addition to the inverted asymmetry, other remarkable properties are
observed in the Bitcoin market. For example, whilst it is claimed that for
stock markets, time series of the log-volatility increments shows
”monofractal” anti-persistence behavior[74, 75, 76], for the Bitcoin market
”multifractal” anti-persistence behavior is reported[11]. This difference is
important to construct a correct volatility model with monofractal or
multifractal behavior. Another interesting property is observed in return-
volatility cross-correlation. For stock markets, it is found that return-
volatility cross-correlation function exhibits an exponential decay that
indicates that the return-volatility cross-correlation is short-ranged[77, 78,
79]. However, for the Bitcoin market, the return-volatility cross-correlation
function shows a power law meaning that the cross-correlation is long-
ranged[12]. The mechanism that originates these remarkable properties
including the inverted asymmetry are not established yet. Toward the complete
understanding of the Bitcoin dynamics, we should further study these
properties in more detail in future direction.

## Acknowledgment

Numerical calculations for this work were carried out at the Yukawa Institute
Computer Facility and at the facilities of the Institute of Statistical
Mathematics.

## References

  * 1. Nakamoto S. Bitcoin: A Peer-to-Peer Electronic Cash System; 2008.
  * 2. Dyhrberg AH.  Hedging capabilities of Bitcoin. Is it the virtual gold?  Finance Research Letters. 2016;16:139–144.
  * 3. Cheah ET, Fry J.  Speculative bubbles in Bitcoin markets? An empirical investigation into the fundamental value of Bitcoin.  Economics Letters. 2015;130:32–36.
  * 4. Wei WC.  Liquidity and market efficiency in cryptocurrencies.  Economics Letters. 2018;168:21–24.
  * 5. Takaishi T, Adachi T.  Taylor effect in Bitcoin time series.  Economics Letters. 2018;172:5–7.
  * 6. Thies S, Molnár P.  Bayesian change point analysis of Bitcoin returns.  Finance Research Letters. 2018;27:223–227.
  * 7. Koutmos D.  Bitcoin returns and transaction activity.  Economics Letters. 2018;167:81–85.
  * 8. Fang W, Tian S, Wang J.  Multiscale fluctuations and complexity synchronization of Bitcoin in China and US markets.  Physica A. 2018;512:109–120.
  * 9. Phillip A, Chan J, Peiris S.  On long memory effects in the volatility measure of cryptocurrencies.  Finance Research Letters. 2019;28:95–100.
  * 10. Urquhart A.  Price clustering in Bitcoin.  Economics Letters. 2017;159:145–148.
  * 11. Takaishi T.  Rough volatility of Bitcoin.  Finance Research Letters. 2020;32:101379.
  * 12. Takaishi T.  Power-law return-volatility cross-correlations of Bitcoin.  EPL (Europhysics Letters). 2020;129(2):28001.
  * 13. Drożdż S, Kwapień J, Oświȩcimka P, Stanisz T, Wa̧torek M.  Complexity in economic and social systems: cryptocurrency market at around COVID-19.  Entropy. 2020;22(9):1043.
  * 14. Cont R.  Empirical Properties of Asset Returns: Stylized Facts and Statistical Issues.  Quantitative Finance. 2001;1:223–236.
  * 15. Kwapień J, Drożdż S.  Physical approach to complex systems.  Physics Reports. 2012;515(3-4):115–226.
  * 16. Chu J, Nadarajah S, Chan S.  Statistical analysis of the exchange rate of Bitcoin.  PloS one. 2015;10(7):e0133678.
  * 17. Takaishi T.  Statistical properties and multifractality of Bitcoin.  Physica A. 2018;506:507–519.
  * 18. Engle RF.  Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation.  Econometrica: Journal of the Econometric Society. 1982; p. 987–1007.
  * 19. Bollerslev T.  Generalized Autoregressive Conditional Heteroskedasticity.  Journal of Econometrics. 1986;31:307–327.
  * 20. Nelson DB.  Conditional Heteroskedasticity in Asset Returns: A New Approach.  Econometrica. 1991;59:347–370.
  * 21. Glosten LR, Jaganathan R, Runkle D.  On the Relation Between the Expected Value and the Volatility of the Nominal Excess on Stocks.  Journal of Finance. 1993;48:1779–1801.
  * 22. Bouri E, Azzi G, Dyhrberg AH.  On the return-volatility relationship in the Bitcoin market around the price crash of 2013.  Economics: The Open-Access, Open-Assessment E-Journal. 2017;11:1–16.
  * 23. Katsiampa P.  Volatility estimation for Bitcoin: A comparison of GARCH models.  Economics Letters. 2017;158:3–6.
  * 24. Stavroyiannis S, Babalos V.  Dynamic properties of the Bitcoin and the US market.  Available at SSRN 2966998. 2017;.
  * 25. Ardia D, Bluteau K, Rüede M.  Regime changes in Bitcoin GARCH volatility dynamics.  Finance Research Letters. 2019;29:266–271.
  * 26. Black F.  Studies of Stock Market Volatility Changes.  1976 Proceedings of the American Statisticsl Association, Business and Economic Statistics Section. 1976; p. 177–181.
  * 27. Christie AA.  The stochastic behavior of common stock variances: Value, leverage and interest rate effects.  Journal of Financial Economics. 1982;10:407–432.
  * 28. Wu G.  The determinants of asymmetric volatility.  The Review of Financial Studies. 2001;14(3):837–859.
  * 29. Dyhrberg AH.  Bitcoin, gold and the dollar–A GARCH volatility analysis.  Finance Research Letters. 2016;16:85–92.
  * 30. Bouoiyour J, Selmi R.  Bitcoin: A beginning of a new phase.  Economics Bulletin. 2016;36(3):1430–1440.
  * 31. Baur DG, Dimpfl T.  Asymmetric volatility in cryptocurrencies.  Economics Letters. 2018;173:148–151.
  * 32. Fakhfekh M, Jeribi A.  Volatility dynamics of crypto-currencies’ returns: Evidence from asymmetric and long memory GARCH models.  Research in International Business and Finance. 2020;51:101075.
  * 33. Bollerslev T, Chou RY, Kroner KF.  ARCH modeling in finance: A review of the theory and empirical evidence.  Journal of econometrics. 1992;52(1-2):5–59.
  * 34. Kyriazis NA, Daskalou K, Arampatzis M, Prassa P, Papaioannou E.  Estimating the volatility of cryptocurrencies during bearish markets by employing GARCH models.  Heliyon. 2019;5(8):e02239.
  * 35. Kantelhardt JW, Zschiegner SA, Koscielny-Bunde E, Havlin S, Bunde A, Stanley HE.  Multifractal detrended fluctuation analysis of nonstationary time series.  Physica A. 2002;316(1):87–114.
  * 36. Jiang ZQ, Xie WJ, Zhou WX, Sornette D.  Multifractal analysis of financial markets.  Rep Prog Phys. 2019;82(12):125901.
  * 37. Urquhart A.  The inefficiency of Bitcoin.  Economics Letters. 2016;148:80–82.
  * 38. Bariviera AF, Basgall MJ, Hasperué W, Naiouf M.  Some stylized facts of the Bitcoin market.  Physica A. 2017;484:82–90.
  * 39. Bariviera AF.  The inefficiency of Bitcoin revisited: A dynamic approach.  Economics Letters. 2017;161:1–4.
  * 40. Alvarez-Ramirez J, Rodriguez E, Ibarra-Valdez C.  Long-range correlations and asymmetry in the Bitcoin market.  Physica A. 2018;492:948–955.
  * 41. Kristoufek L.  On Bitcoin markets (in) efficiency and its evolution.  Physica A. 2018;503:257–262.
  * 42. da Silva Filho AC, Maganini ND, de Almeida EF.  Multifractal analysis of Bitcoin market.  Physica A. 2018;512:954–967.
  * 43. Zhang W, Wang P, Li X, Shen D.  The inefficiency of cryptocurrency and its cross-correlation with Dow Jones Industrial Average.  Physica A. 2018;510:658–670.
  * 44. Takaishi T, Adachi T.  Market efficiency, liquidity, and multifractality of Bitcoin: A dynamic study.  Asia-Pacific Financial Markets. 2020;27:145–154.
  * 45. Dimitrova V, Fernández-Martínez M, Sánchez-Granero MA, Trinidad Segovia JE.  Some comments on Bitcoin market (in) efficiency.  PloS one. 2019;14(7):e0219243.
  * 46. Fama EF.  Efficient capital markets: A review of theory and empirical work.  The journal of Finance. 1970;25(2):383–417.
  * 47. Drożdż S, Gȩbarowski R, Minati L, Oświȩcimka P, Wa̧torek M.  Bitcoin market route to maturity? Evidence from return fluctuations, temporal correlations and multiscaling effects.  Chaos: An Interdisciplinary Journal of Nonlinear Science. 2018;28(7):071101.
  * 48. Zunino L, Bariviera AF, Guercio MB, Martinez LB, Rosso OA.  On the efficiency of sovereign bond markets.  Physica A. 2012;391(18):4342–4349.
  * 49. Sentana E.  Quadratic ARCH Models.  Review of Economic Studies. 1995;62:639–661.
  * 50. Heston SL, Nandi S.  A closed-form GARCH option valuation model.  The review of financial studies. 2000;13(3):585–625.
  * 51. Takaishi T.  Rational GARCH model: An empirical test for stock returns.  Physica A. 2017;473:451–460.
  * 52. Ivanov PC, Amaral LAN, Goldberger AL, Havlin S, Rosenblum MG, Struzik ZR, et al.  Multifractality in human heartbeat dynamics.  Nature. 1999;399(6735):461–465.
  * 53. Stanley HE, Meakin P.  Multifractal phenomena in physics and chemistry.  Nature. 1988;335(6189):405–409.
  * 54. Lennartz S, Livina V, Bunde A, Havlin S.  Long-term memory in earthquakes and the distribution of interoccurrence times.  EPL (Europhysics Letters). 2008;81(6):69001.
  * 55. Ihlen EA, Vereijken B.  Multifractal formalisms of human behavior.  Human movement science. 2013;32(4):633–651.
  * 56. Jafari G, Pedram P, Hedayatifar L.  Long-range correlation and multifractality in Bach’s inventions pitches.  Journal of Statistical Mechanics: Theory and Experiment. 2007;2007(04):P04012.
  * 57. Zunino L, Tabak BM, Figliola A, Pérez D, Garavaglia M, Rosso O.  A multifractal approach for stock market inefficiency.  Physica A. 2008;387(26):6558–6566.
  * 58. Trinidad Segovia JE, Fernández-Martínez M, Sánchez-Granero MA.  A novel approach to detect volatility clusters in financial time series.  Physica A. 2019;535:122452.
  * 59. Nikolova V, Trinidad Segovia JE, Fernández-Martínez M, Sánchez-Granero MA.  A Novel Methodology to Calculate the Probability of Volatility Clusters in Financial Series: An Application to Cryptocurrency Markets.  Mathematics. 2020;8(8):1216.
  * 60. Easwaran S, Dixit M, Sinha S.  Bitcoin dynamics: the inverse square law of price fluctuations and other stylized facts.  In: Econophysics and data driven modelling of market dynamics. Springer; 2015. p. 121–128.
  * 61. Begušić S, Kostanjčar Z, Stanley HE, Podobnik B.  Scaling properties of extreme price fluctuations in Bitcoin markets.  Physica A. 2018;510:400–406.
  * 62. Gopikrishnan P, Meyer M, Amaral LN, Stanley HE.  Inverse cubic law for the distribution of stock price variations.  The European Physical Journal B-Condensed Matter and Complex Systems. 1998;3(2):139–140.
  * 63. Gopikrishnan P, Plerou V, Amaral LAN, Meyer M, Stanley HE.  Scaling of the distribution of fluctuations of financial market indices.  Physical Review E. 1999;60(5):5305.
  * 64. Plerou V, Gopikrishnan P, Amaral LAN, Meyer M, Stanley HE.  Scaling of the distribution of price fluctuations of individual companies.  Physical review E. 1999;60(6):6519.
  * 65. Pan RK, Sinha S.  Self-organization of price fluctuation distribution in evolving markets.  EPL (Europhysics Letters). 2007;77(5):58004.
  * 66. Drożdż S, Minati L, Oświȩcimka P, Stanuszek M, Wa̧torek M.  Signatures of the crypto-currency market decoupling from the Forex.  Future Internet. 2019;11(7):154.
  * 67. Takaishi T.  Recent scaling properties of Bitcoin price returns.  arXiv:200906874. 2020;.
  * 68. Avramov D, Chordia T, Goyal A.  The impact of trades on daily volatility.  The Review of Financial Studies. 2006;19(4):1241–1277.
  * 69. Amihud Y.  Illiquidity and stock returns: cross-section and time-series effects.  Journal of financial markets. 2002;5(1):31–56.
  * 70. Roll R.  A critique of the asset pricing theory’s tests Part I: On past and potential testability of the theory.  Journal of financial economics. 1977;4(2):129–176.
  * 71. Taylor SJ.  Financial Returns Modelled by the Product of Two Stochastic Processes, a Study of Daily Sugar Prices 1961-79.  North-Holland, Amsterdam; 1982.
  * 72. Taylor SJ.  Modelling Financial Time Series.  John Wiley & New jersy; 1986.
  * 73. Asai M, McAleer M, Yu J.  Multivariate stochastic volatility: a review.  Econometric Reviews. 2006;25(2-3):145–175.
  * 74. Gatheral J, Jaisson T, Rosenbaum M.  Volatility is rough.  Quantitative Finance. 2018;18(6):933–949.
  * 75. Bennedsen M, Lunde A, Pakkanen MS.  Decoupling the short-and long-term behavior of stochastic volatility.  arXiv:161000332. 2016;.
  * 76. Livieri G, Mouti S, Pallavicini A, Rosenbaum M.  Rough volatility: evidence from option prices.  IISE Transactions. 2018;50(9):767–776.
  * 77. Bouchaud JP, Matacz A, Potters M.  Leverage effect in financial markets: The retarded volatility model.  Physical Review Letters. 2001;87(22):228701.
  * 78. Qiu T, Zheng B, Ren F, Trimper S.  Return-volatility correlation in financial dynamics.  Physical Review E. 2006;73(6):065103.
  * 79. Chen JJ, Zheng B, Tan L.  Agent-based model with asymmetric trading and herding for complex financial systems.  PloS one. 2013;8(11):e79531.


11institutetext: BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain,
11email<EMAIL_ADDRESS>22institutetext: University of Adelaide,
Adelaide, Australia

# Performance Metrics for
Probabilistic Ordinal Classifiers

Adrian Galdran${}^{\textrm{,{\char 0\relax}}}$ 1122

###### Abstract

Ordinal classification models assign higher penalties to predictions further
away from the true class. As a result, they are appropriate for relevant
diagnostic tasks like disease progression prediction or medical image grading.
The consensus for assessing their categorical predictions dictates the use of
distance-sensitive metrics like the Quadratic-Weighted Kappa score or the
Expected Cost. However, there has been little discussion regarding how to
measure performance of probabilistic predictions for ordinal classifiers. In
conventional classification, common measures for probabilistic predictions are
Proper Scoring Rules (PSR) like the Brier score, or Calibration Errors like
the ECE, yet these are not optimal choices for ordinal classification. A PSR
named Ranked Probability Score (RPS), widely popular in the forecasting field,
is more suitable for this task, but it has received no attention in the image
analysis community. This paper advocates the use of the RPS for image grading
tasks. In addition, we demonstrate a counter-intuitive and questionable
behavior of this score, and propose a simple fix for it. Comprehensive
experiments on four large-scale biomedical image grading problems over three
different datasets show that the RPS is a more suitable performance metric for
probabilistic ordinal predictions. Code to reproduce our experiments can be
found at github.com/agaldran/prob_ord_metrics.

###### Keywords:

Ordinal Classification Proper Scoring Rules Model Calibration Uncertainty
Quantification

##  1 Introduction and Related Work

The output of predictive machine learning models is often presented as
categorical values, i.e. “hard” class membership decisions. Nonetheless,
understanding the faithfulness of the underlying probabilistic predictions
giving rise to such hard class decisions can be essential in some critical
applications. Meaningful probabilities enable not only high model accuracy,
but also more reliable decisions: a doctor may choose to order further
diagnostic tests if a binary classifier gives a $p=45\%$ probability of
disease, even if the hard prediction is “healthy” [2]. This is particularly
true for ordinal classification problems, e.g. disease severity staging [6, 7]
or medical image grading [14, 21]. In these problems, predictions should be as
close as possible to the actual category; further away predictions must incur
in heavier penalties, as they have increasingly worse consequences.

There is a large body of research around performance metrics for medical image
analysis [20]. Most existing measures, like accuracy or the F1-score, focus on
assessing hard predictions in specific ways that capture different aspects of
a problem. In ordinal classification, the recommended metrics are Quadratic-
Weighted Kappa and the Expected Cost [5, 16]. In recent years, measuring the
performance of “soft” probabilistic predictions has attracted an increasing
research interest [12, 19]. For this purpose, the current consensus is to
employ Calibration Errors like the ECE and Proper Scoring Rules like the Brier
score [16]. We will show that other metrics can instead be a better choice for
assessing probabilistic predictions in the particular case of ordinal
classification problems.

How to measure the correctness of probabilistic predictions is a decades-old
question, naturally connected to forecasting, i.e. predicting the future state
of a complex system [9]. A key aspect of forecasting is that, contrary to
classifiers, forecasters do not output hard decisions, but probability
distributions over possible outcomes. Weather forecasts do not tell us whether
it will rain tomorrow or not, they give us a probability estimate about the
likelihood of raining, leaving to us the decision of taking or not an
umbrella, considering the personal cost of making such decision. The same
applies for financial investments or sports betting, where it is also the
final user who judges risks and makes decisions based on probabilistic
forecasts. In this context, Proper Scoring Rules (PSRs) have been long used by
the forecasting community to measure predictive performance [10]. PSRs are the
focus of this paper, and will be formally defined in section 2.1.

#### Relation to Calibration:

A popular approach to assess the quality of probabilistic predictions is
measuring calibration. A model is well calibrated if its probabilistic
predictions are aligned with its accuracy on average. PSRs and calibration are
intertwined concepts: PSRs can be decomposed into a calibration and a
resolution component [8]. Therefore, a model needs to be both calibrated and
resolved (i.e. having sharp, or concentrated probabilities) in order to have a
good PSR value. For example, if a disease appears in 60% of the population,
and our model is just “return p=0.6”, in the long run the model is correct
$60\%$ of the time, and it is perfectly calibrated, as its confidence is fully
aligned with its accuracy, despite having zero predictive ability. If the
model predicted in a “resolved” manner with $p=0.99$ the presence of the
disease, but being correct only $70\%$ of the time, then it would be
overconfident, which is a form of miscalibration. Only when the model is
simultaneously confident and correct can it attain a good PSR value.

The two most widely adopted PSRs are the Brier and the Logarithmic Score [1,
11]. Unfortunately, none of these is appropriate for the assessment of ordinal
classification probabilities [3]. A third PSR, long used by forecasting
researchers in this scenario, the Ranked Probability Score (RPS, [4]), appears
to have been neglected so far in biomedical image grading applications. This
paper first covers the definition and basic properties of PSRs, and then
motivates the use the RPS for ordinal classifiers. We also illustrate a
counter-intuitive behavior of the RPS, and propose a simple modification to
solve it. Our experiments cover two relevant biomedical image grading problems
and illustrate how the RPS can better assess probabilistic predictions of
ordinal classification models.

##  2 Methods

###  2.1 Scoring Rules - Notation, Properties, Examples

We consider a $K$-class classification problem, and a classifier that takes an
image $\mathbf{x}$ and maps it into a vector of probabilities
$\mathbf{p}\in[0,1]^{K}$. Typically, $\mathbf{p}$ is the result of applying a
softmax operation on the output of a neural network. Suppose $\mathbf{x}$
belongs to class $y\in\\{1,...,K\\}$, and denote by $\mathbf{y}$ its one-hot
representation. A Scoring Rule (SR) $\mathcal{S}$ is any function taking the
probabilistic prediction $\mathbf{p}$ and the label $\mathbf{y}$ and producing
a number $\mathcal{S}(\mathbf{p},\mathbf{y})\in\mathbb{R}$ (a score). Here we
consider negatively oriented SRs, which assign lower values to better
predictions.

Of course, the above is an extremely generic definition, to which we must now
attach additional properties in order to encode our understanding of what
better predictions means for a particular problem.

#### Property 1:

A Scoring Rule (SR) is proper if its value is minimal when the probabilistic
prediction coincides with the ground-truth in expectation.

#### Example:

The Brier Score [1] is defined as the sum of the squared differences between
probabilities and labels:

$\textrm{Brier}(\mathbf{p},\mathbf{y})=\|\mathbf{p}-\mathbf{y}\|_{2}^{2}=\sum_{i=1}^{K}(p_{i}-y_{i})^{2}.$
(1)

Since its value is always non-negative, and it decreases to 0 when
$\mathbf{p}=\mathbf{y}$, we conclude that the Brier Score is indeed proper.

#### Property 2:

A Proper Scoring Rule (PSR) is local if its value only depends on the
probability assigned to the correct category.

#### Example:

The Brier Score is non-local, as its value depends on the probability placed
by the model on all classes. The Logarithmic Score [11], given by:

$\mathcal{L}(\mathbf{p},\mathbf{y})=-\log(p_{c})$ (2)

where $c$ is the correct category of $\mathbf{x}$, rewards the model by
placing as much probability mass as possible in $c$, regardless of how the
remaining probability is distributed. It is, therefore, a local PSR. The
Logarithmic Score is also known, when taken on average over a dataset, as the
Negative Log-Likelihood.

#### Property 3:

A PSR is sensitive to distance if its value takes into account the order of
the categories, in such a way that probability placed in categories further
away from the correct class is more heavily penalized.

#### Example:

Both the Brier and the Logarithmic scores are insensitive to distance
(shuffling $\mathbf{p}$ and $\mathbf{y}$ won’t affect the score). Sensitivity
to distance is essential for assessing ordinal classifiers. Below we define
the Ranked Probability Score (RPS) [4, 18], which has this property, and is
therefore more suitable for our purposes.

###  2.2 The Ranked Probability Score for Ordinal Classification

Consider a test sample $(\mathbf{x},\mathbf{y})$ in a 3-class classification
problem, with label $\mathbf{y}$ and two probabilistic predictions
$\mathbf{p}_{1},\mathbf{p}_{2}$:

$\mathbf{y}=[\,1,0,0\,],\ \ \mathbf{p}_{1}=[\,\frac{1}{4},\frac{3}{4},0\,],\ \
\mathbf{p}_{2}=[\,\frac{1}{4},0,\frac{3}{4}\,]$ (3)

In this scenario, both the Brier and the Logarithmic scores produce the same
penalty for each prediction, whereas a user might prefer $\mathbf{p}_{1}$ over
$\mathbf{p}_{2}$ due to the latter assigning more probability to the second
category. Indeed, if we use the arg-max operator to generate a hard-decision
for this sample, we will obtain a prediction of class 2 and class 3
respectively, which could result in the second model declaring a patient as
severely unhealthy with serious consequences. In this context, we would like
to have a PSR that takes into account distance to the true category, such as
the Ranked Probability Score (RPS, [4]), given by:

$\textrm{RPS}(\mathbf{p},\mathbf{y})=\frac{1}{K-1}\sum_{i=1}^{K-1}\left[\sum_{j=1}^{i}(p_{j}-y_{j})\right]^{2}=\frac{1}{K-1}\|\mathbf{P}-\mathbf{Y}\|^{2}_{2}.$
(4)

The RPS is the squared $\bm{\ell_{2}}$ distance between the cumulative
distributions $\mathbf{Y}$ of the target label $\mathbf{y}$ and $\mathbf{P}$
of the probabilistic prediction $\mathbf{p}$, discounting their last component
(as they are both always one) and normalizing so that it varies in the unit
interval. In the above example, the RPS would give for each prediction a
penalty of $\textrm{RPS}(\mathbf{p}_{1},\mathbf{y})=\nicefrac{{1}}{{8}}$,
$\textrm{RPS}(\mathbf{p}_{2},\mathbf{y})=\nicefrac{{1}}{{4}}$, as shown in
Figure 1.

Figure 1: The RPS is sensitive to distance, suitable for assessing
probabilistic predictions on biomedical image grading problems. It is the
difference between the cumulative probability distributions of the label and a
probabilistic prediction.

Among many interesting properties, one can show that the RPS is proper [17],
and reduces to the Brier score for $K=2$. Despite the RPS dating back more
than 50 years [4], and enjoying great popularity in the weather forecasting
community, it appears to be much less known in the image analysis and computer
vision areas, where we could not find any trace of it. The first goal of this
paper is to bring to the attention of computer vision researchers this tool
for measuring the performance of probabilistic predictions in ordinal
classification.

Figure 2: The Ranked Probability Score displays some counter-intuitive
behavior that the proposed sa-RPS can fix. Here, $\mathbf{p}_{2}$ places more
probability on the correct class but $\mathbf{p}_{1}$ is preferred due to its
symmetry.

###  2.3 The Squared Absolute RPS

Our second goal in this paper is to identify and then fix certain failure
modes of the RPS that might lead to counter-intuitive behaviors. First, in
disease grading and other ordinal classification problems it is customary to
assign penalties to mistakes that grow quadratically with the distance to the
correct category. This is the reason why most works utilize the Quadratic-
Weighted Kappa Score (QWK) instead of the linearly weighted version of this
metric. However, the RPS increases the penalty linearly, as can be quickly
seen with a simple 3-class problem and an example
$(\mathbf{x}_{1},\mathbf{y}_{1})$ of class 1 ($\mathbf{y}_{1}=[\,1,0,0\,]$):

$\textrm{RPS}([\,1,0,0\,],\mathbf{y}_{1})=0,\ \
\textrm{RPS}([\,0,1,0\,],\mathbf{y}_{1})=1/2.\ \
\textrm{RPS}([\,0,0,1\,],\mathbf{y}_{1})=1.$ (5)

Also, the RPS has a hidden preference for symmetric predictions. To see this,
consider a second example $(\mathbf{x}_{2},\mathbf{y}_{2})$ in which the
correct category is now the middle one ($\mathbf{y}_{2}=[\,0,1,0\,]$), and two
probabilistic predictions: $p_{\mathrm{sym}}=[\,3/10,4/10,3/10\,]$,
$p_{\mathrm{asym}}=[\,1/10,5/10,9/10\,]$. In principle, there is no reason to
prefer $p_{\mathrm{sym}}$ over $p_{\mathrm{asym}}$, unless certain
prior/domain knowledge tells us that symmetry is a desirable property. In this
particular case, $p_{\mathrm{asym}}$ is actually more confident on the correct
class than $p_{\mathrm{sym}}$, which is however the preferred prediction for
the RPS:

$\textrm{RPS}([\,0.30,0.40,0.30\,],\mathbf{y}_{2})=0.09<0.1025=\textrm{RPS}([\,0.45,0.50,0.05\,],\mathbf{y}_{2}).$
(6)

In order to address these aspects of the conventional RPS, we propose to
implement instead the Squared Absolute RPS (sa-RPS), given by:

$\textrm{sa-
RPS}(\mathbf{p},\mathbf{y})=\frac{1}{K-1}\left[\sum_{i=1}^{K}\left|\sum_{j=1}^{i}(p_{j}-y_{j})\right|\right]^{2}$
(7)

Replacing the inner square in eq. (4) by an absolute value, we manage to break
the preference for symmetry of the RPS, and squaring the overall result we
build a metric that still varies in [0,1] but gives a quadratic penalty to
further away predictions. This is illustrated in Fig. 2 above.

###  2.4 Evaluating Evaluation Metrics

Our third goal is to demonstrate how the (sa-)RPS is useful for evaluating
probabilistic ordinal predictions. In the next section we will show some
illustrative examples that qualitatively demonstrate its superiority over the
Brier and logarithmic score. However, it is hard to quantitatively make the
case for one performance metric over another, since metrics themselves are
what quantify modeling success. We proceed as follows: we first train a neural
network to solve a biomedical image grading problem. We generate probabilistic
predictions on the test set and apply distance sensitive metrics to (arg-
maxed) hard predictions (QWK and EC, as recommended in [16]), verifying model
convergence.

Here it is important to stress that, contrary to conventional metrics (like
accuracy, QWK, or ECE) PSRs can act on an individual datum, without averaging
over sets of samples. We exploit this property to design the following
experiment: we sort the probabilistic predictions of the test set according to
a score $\mathcal{S}$, and then progressively remove samples that are of worst
quality according to $\mathcal{S}$. We take the arg-max on the remaining
probabilistic predictions and compute QWK and EC. If $\mathcal{S}$ prefers
better ordinal predictions, we must see a performance increase on that subset.
We repeat this process, each time removing more of the worse samples, and
graph the evolution of QWK and EC for different scores $\mathcal{S}$: a better
score should result in a faster QWK/EC-improving trend.

Lastly, in order to derive a single number to measure performance, we compute
the area under the remaining samples vs QWK/EC curve, which we call Area under
the Retained Samples Curve (AURSC). In summary:

What we expect to see: As we remove test set samples considered as worse
classified by RPS, we expect to more quickly improve QWK/EC on the resulting
subsets. We measure this with the Area under the Retained Samples Curve
(AURSC)

##  3 Experimental Results

We now give a description of the data we used for experimentation, analyze
performance for each considered problem, and close with a discussion of
results.

###  3.1 Datasets and Architecture

Our experiments are on two different medical image grading tasks: 1) the
TMED-v2 dataset ([13], link) contains 17,270 images from 577 patients, with an
aortic stenosis (AS) diagnostic label from three categories (none, early AS,
or significant AS). The authors provide an official train/test distribution of
the data that we use here. 2) Eyepacs (link) contains retinal images and
labels for grading Diabetic Retinopathy (DR) stage into five categories,
ranging from healthy to proliferative DR. Ithas 35,126 images for training and
53,576 in the test set.

We train a ConvNeXt [15], minimizing the CE loss with the adam algorithm for
10 epochs starting with a learning rate of $l=1e$-$4$, decaying to zero over
the training. We report average Area under the Retained Samples Curve (AURSC)
for $50$ bootstrap iterations in each dataset below, and also plot the
evolution of performance as we remove more samples considered to be worse by
four PSRs: the Brier score, the Logarithmic score (Neg-Log), RPS and sa-RPS.

Figure 3: For the same test set and predictions, the RPS finds wrong samples
that are more incorrect from the point of view of ordinal classification.

###  3.2 How is RPS useful? Qualitative Error Analysis

The obvious application of RPS would be to train better ordinal classification
models. But beyond this, RPS also enables improved, fine-grained error
analysis. Let us see this through a simple experiment. Since PSRs assess
samples individually, we can sort our test set using RPS, NLL, and Brier
score. The worst-scored items are what the model considers the wrongest
probabilistic predictions. The result of sorting predictions on the Eyepacs
test set with the Brier, Neg-Log and RPS rules is show on Fig. 3. We can see
that the prediction identified as worst by the RPS does indeed violate more
heavily the order of categories, placing more probability on class 5 for a
sample of class 1. On the other hand, for the same test set and predictions,
the Brier score finds worst a prediction with 99% of the probability on class
3 and a label of class 5, and the Neg-Log score identifies a sample of class 1
for which the model wrongly predicts class 2.

###  3.3 Quantitative Experimental Analysis

Quantitative results of the experiment described in section 2.4, computing
AURSC values for all PSRs, are shown in Table 2, with dispersion measures
obtained from 50 bootstraped performance measurements.

| TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.46 $\pm$ 0.35 | 3.76 $\pm$ 0.21 | 17.36 $\pm$ 0.04 | 2.84 $\pm$ 0.07
Neg-Log | 13.56 $\pm$ 0.35 | 3.62 $\pm$ 0.2 | 17.44 $\pm$ 0.04 | 2.67 $\pm$ 0.07
RPS | 14.76 $\pm$ 0.28 | 2.68 $\pm$ 0.14 | 17.81 $\pm$ 0.03 | 1.99 $\pm$ 0.04
sa-RPS | 14.95 $\pm$ 0.25 | 2.53 $\pm$ 0.12 | 17.86 $\pm$ 0.03 | 1.88$\pm$ 0.04
Table 1: Areas under the Retained Samples Curve for TMED and Eyepacs, with a
ConvNeXt, for each PSR; best and second best values are marked.

We see that for the considered ordinal classification problems, distance-
sensitive scores consistently outperform the Brier and Neg-Log scores. Also,
the Square-Absolute Ranked Probability Score always outperforms the
conventional Ranked Probability Score. It is worth stressing that when
observing bootstrapped performance intervals, neither the Brier nor the
Logarithmic scores manage to overlap the SA-RPS interval in any of the two
datasets, and in the Eyepacs dataset not even the best RPS result reaches the
performance of worst SA-RPS result.

(a)  (b)

Figure 4: We sort probabilistic predictions in each test set using several
PSRs: Brier, Neg-Log, RPS, sa-RPS. We progressively discard worse-scored
samples, improving the metric of interest (only QWK shown). Removing worse
samples according to RPS and sa-RPS leads to better QWK, implying that they
both capture better ordinal classification performance at the probabilistic
level.

For a visual analysis, Fig. 4 shows the full Sample Retention Curves from
which AURSC-QWK values in Table 1 were computed. These curves show how PSRs
can indeed take a single probabilistic prediction and return a score that is
correlated to QWK, which is computed over sets of samples. This is because as
we remove samples according to any PSR, performance in the remaining test set
improves in all cases. The curves in Fig. 4 also tell a more complete story of
how the two distance-sensitive scores outperform the Brier and Neg-Log scores,
particularly for TMED and Eyepacs. Just by removing a 5%-6% of samples with
worse (higher) RPS, we manage to improve QWL and EC to a greater extent.

##  4 Conclusion and Future Work

We have shown that Proper Scoring Rules are useful tools for diagnosing
probabilistic predictions, but the standard Brier and Logarithmic scores
should not be preferred in ordinal classification problems like medical image
grading. Instead, the Ranked Probability Score, popular in the forecasting
community, should be favoured. We have also proposed sa-RPS, an extension of
the RPS that can better handle some pathological cases. Future work will
involve using the RPS to learn ordinal classifiers, and investigating its
impact in calibration problems.

## Acknowledgments

This work was supported by a Marie Skłodowska-Curie Fellowship (No 892297).

## References

  * [1] Brier, G.W.: Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78(1), 1–3 (Jan 1950). https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2, publisher: American Meteorological Society Section: Monthly Weather Review
  * [2] Cahan, A., Gilon, D., Manor, O., Paltiel, O.: Probabilistic reasoning and clinical decision-making: do doctors overestimate diagnostic probabilities? QJM: An International Journal of Medicine 96(10), 763–769 (Oct 2003). https://doi.org/10.1093/qjmed/hcg122, https://doi.org/10.1093/qjmed/hcg122
  * [3] Constantinou, A.C., Fenton, N.E.: Solving the Problem of Inadequate Scoring Rules for Assessing Probabilistic Football Forecast Models. Journal of Quantitative Analysis in Sports 8(1) (Mar 2012). https://doi.org/10.1515/1559-0410.1418, publisher: De Gruyter
  * [4] Epstein, E.S.: A Scoring System for Probability Forecasts of Ranked Categories. Journal of Applied Meteorology and Climatology 8(6), 985–987 (Dec 1969). https://doi.org/10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2
  * [5] Ferrer, L.: Analysis and Comparison of Classification Metrics (Nov 2022). https://doi.org/10.48550/arXiv.2209.05355, http://arxiv.org/abs/2209.05355, arXiv:2209.05355 [cs]
  * [6] Galdran, A., Chelbi, J., Kobi, R., Dolz, J., Lombaert, H., ben Ayed, I., Chakor, H.: Non-uniform Label Smoothing for Diabetic Retinopathy Grading from Retinal Fundus Images with Deep Neural Networks. Translational Vision Science & Technology 9(2), 34 (Jun 2020). https://doi.org/10.1167/tvst.9.2.34
  * [7] Galdran, A., Dolz, J., Chakor, H., Lombaert, H., Ben Ayed, I.: Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 665–674. Lecture Notes in Computer Science, Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_64
  * [8] Gneiting, T., Balabdaoui, F., Raftery, A.E.: Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69(2), 243–268 (2007). https://doi.org/10.1111/j.1467-9868.2007.00587.x, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9868.2007.00587.x
  * [9] Gneiting, T., Katzfuss, M.: Probabilistic Forecasting. Annual Review of Statistics and Its Application 1(1), 125–151 (2014). https://doi.org/10.1146/annurev-statistics-062713-085831
  * [10] Gneiting, T., Raftery, A.E.: Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477), 359–378 (Mar 2007). https://doi.org/10.1198/016214506000001437
  * [11] Good, I.J.: Rational Decisions. Journal of the Royal Statistical Society: Series B (Methodological) 14(1), 107–114 (1952). https://doi.org/10.1111/j.2517-6161.1952.tb00104.x
  * [12] Gruber, S.G., Buettner, F.: Better Uncertainty Calibration via Proper Scores for Classification and Beyond (Oct 2022), https://openreview.net/forum?id=PikKk2lF6P
  * [13] Huang, Z., Long, G., Wessler, B., Hughes, M.C.: TMED 2: A Dataset for Semi-Supervised Classification of Echocardiograms. In: Unpublished Technical Report (2022), https://tmed.cs.tufts.edu/papers/HuangEtAl_TMED2_DataPerf_2022.pdf
  * [14] Jaroensri, R., et al.: Deep learning models for histologic grading of breast cancer and association with disease prognosis. npj Breast Cancer 8(1), 1–12 (Oct 2022). https://doi.org/10.1038/s41523-022-00478-y
  * [15] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. pp. 11976–11986 (2022)
  * [16] Maier-Hein, L., et al.: Metrics reloaded: Pitfalls and recommendations for image analysis validation (Feb 2023). https://doi.org/10.48550/arXiv.2206.01653, http://arxiv.org/abs/2206.01653, arXiv:2206.01653 [cs]
  * [17] Murphy, A.H.: On the “Ranked Probability Score”. Journal of Applied Meteorology and Climatology 8(6), 988–989 (Dec 1969). https://doi.org/10.1175/1520-0450(1969)008<0988:OTPS>2.0.CO;2
  * [18] Murphy, A.H.: A Note on the Ranked Probability Score. Journal of Applied Meteorology and Climatology 10(1), 155–156 (Feb 1971). https://doi.org/10.1175/1520-0450(1971)010<0155:ANOTRP>2.0.CO;2
  * [19] Perez-Lebel, A., Morvan, M.L., Varoquaux, G.: Beyond calibration: estimating the grouping loss of modern neural networks (Jan 2023). https://doi.org/10.48550/arXiv.2210.16315, http://arxiv.org/abs/2210.16315, arXiv:2210.16315 [cs, stat]
  * [20] Reinke, A., et al.: Understanding metric-related pitfalls in image analysis validation (Feb 2023). https://doi.org/10.48550/arXiv.2302.01790
  * [21] Silva-Rodríguez, J., Colomer, A., Sales, M.A., Molina, R., Naranjo, V.: Going deeper through the Gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer Methods and Programs in Biomedicine 195, 105637 (Oct 2020). https://doi.org/10.1016/j.cmpb.2020.105637

##  Appendix A Further Experimental Results

In the main paper we reported results without dispersion measures due space
constraints. Below we show expanded tables with standard deviation, and we
include several more architectures (plus ConvNeXt in Tab. 2, here also with
dispersion measures). Specifically, we include results obtained wtih a Swin-
Transformer (Tab. 3), a Densnet121 (Tab. 4), a Resnet50 (Tab. 5), a Resnet18
(Tab. 7, and a Mobilent V2 (Tab. 8). In general, we observe similar trends as
in the main paper. For all six considered neural networks, distance-sensitive
PSRs always achieve greater performance than the Brier and the Logarithmic
scores, and also the Square-Absolute Ranked Probability Score always
outperforms the conventional Ranked Probability Score. It is worth stressing
that when observing bootstrapped performance intervals, the neither the Brier
nor the Logarithmic scores manage to overlap the SA-RPS interval in any of the
two datasets, and in the Eyepacs dataset not even the best RPS result reaches
the performance of worst SA-RPS result.

| TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.46 $\pm$ 0.35 | 3.76 $\pm$ 0.21 | 17.36 $\pm$ 0.04 | 2.84 $\pm$ 0.07
Neg-Log | 13.56 $\pm$ 0.35 | 3.62 $\pm$ 0.2 | 17.44 $\pm$ 0.04 | 2.67 $\pm$ 0.07
RPS | 14.76 $\pm$ 0.28 | 2.68 $\pm$ 0.14 | 17.81 $\pm$ 0.03 | 1.99 $\pm$ 0.04
sa-RPS | 14.95 $\pm$ 0.25 | 2.53 $\pm$ 0.12 | 17.86 $\pm$ 0.03 | 1.88$\pm$ 0.04
Table 2: Areas under the Retained Samples Curve for TMED and Eyepacs, with a ConvNeXt, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.60 $\pm$ 0.39 | 3.56 $\pm$ 0.25 | 17.33 $\pm$ 0.04 | 2.84 $\pm$ 0.07
Neg-Log | 13.71 $\pm$ 0.39 | 3.43 $\pm$ 0.25 | 17.40 $\pm$ 0.04 | 2.68 $\pm$ 0.06
RPS | 14.59 $\pm$ 0.30 | 2.75 $\pm$ 0.18 | 17.77 $\pm$ 0.03 | 2.02 $\pm$ 0.05
sa-RPS | 14.79 $\pm$ 0.26 | 2.58 $\pm$ 0.15 | 17.82 $\pm$ 0.03 | 1.91 $\pm$ 0.04
Table 3: Areas under the Retained Samples Curve for TMED and Eyepacs, with a Swin-T, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.25 $\pm$ 0.35 | 3.76 $\pm$ 0.2 | 17.16 $\pm$ 0.05 | 3.07 $\pm$ 0.08
Neg-Log | 13.31 $\pm$ 0.34 | 3.66 $\pm$ 0.2 | 17.23 $\pm$ 0.05 | 2.91 $\pm$ 0.07
RPS | 14.43 $\pm$ 0.28 | 2.81 $\pm$ 0.15 | 17.67 $\pm$ 0.03 | 2.15 $\pm$ 0.05
sa-RPS | 14.70 $\pm$ 0.23 | 2.59 $\pm$ 0.11 | 17.72 $\pm$ 0.03 | 2.02$\pm$ 0.04
Table 4: Areas under the Retained Samples Curve for TMED and Eyepacs, with a DenseNet, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.14 $\pm$ 0.44 | 3.93 $\pm$ 0.27 | 16.91 $\pm$ 0.05 | 3.5 $\pm$ 0.08
Neg-Log | 13.24 $\pm$ 0.43 | 3.80 $\pm$ 0.26 | 16.97 $\pm$ 0.05 | 3.34 $\pm$ 0.07
RPS | 14.32 $\pm$ 0.34 | 2.98 $\pm$ 0.19 | 17.50 $\pm$ 0.03 | 2.41 $\pm$ 0.04
sa-RPS | 14.46 $\pm$ 0.30 | 2.84 $\pm$ 0.16 | 17.54 $\pm$ 0.03 | 2.30 $\pm$ 0.04
Table 5: Areas under the Retained Samples Curve for TMED and Eyepacs, with a ResNet50, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.41 $\pm$ 0.41 | 3.61 $\pm$ 0.23 | 16.87 $\pm$ 0.05 | 3.59 $\pm$ 0.08
Neg-Log | 13.47 $\pm$ 0.41 | 3.51 $\pm$ 0.22 | 16.93 $\pm$ 0.05 | 3.44 $\pm$ 0.08
RPS | 14.49 $\pm$ 0.33 | 2.73 $\pm$ 0.16 | 17.48 $\pm$ 0.04 | 2.46 $\pm$ 0.05
sa-RPS | 14.82 $\pm$ 0.29 | 2.47 $\pm$ 0.13 | 17.54 $\pm$ 0.03 | 2.31 $\pm$ 0.05
Table 6: Areas under the Retained Samples Curve for TMED and Eyepacs, with a ResNet34, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 13.12 $\pm$ 0.33 | 4.33 $\pm$ 0.23 | 16.74 $\pm$ 0.06 | 3.66 $\pm$ 0.08
Neg-Log | 13.14 $\pm$ 0.34 | 4.25 $\pm$ 0.24 | 16.83 $\pm$ 0.05 | 3.45 $\pm$ 0.08
RPS | 14.41 $\pm$ 0.27 | 3.21 $\pm$ 0.17 | 17.36 $\pm$ 0.04 | 2.55 $\pm$ 0.05
sa-RPS | 14.73 $\pm$ 0.23 | 2.93 $\pm$ 0.14 | 17.41 $\pm$ 0.04 | 2.42 $\pm$ 0.05
Table 7: Areas under the Retained Samples Curve for TMED and Eyepacs, with a ResNet18, for each PSR; best and second best values are marked. | TMED | Eyepacs
---|---|---
| AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$ | AURSC-QWK$\uparrow$ | AURSC-EC$\downarrow$
Brier | 11.13 $\pm$ 0.43 | 4.92 $\pm$ 0.23 | 16.62 $\pm$ 0.05 | 3.85 $\pm$ 0.07
Neg-Log | 11.13 $\pm$ 0.43 | 4.84 $\pm$ 0.23 | 16.71 $\pm$ 0.05 | 3.63 $\pm$ 0.07
RPS | 12.61 $\pm$ 0.36 | 3.74 $\pm$ 0.17 | 17.28 $\pm$ 0.04 | 2.66 $\pm$ 0.05
sa-RPS | 12.86 $\pm$ 0.33 | 3.46 $\pm$ 0.15 | 17.34 $\pm$ 0.03 | 2.52 $\pm$ 0.04
Table 8: Areas under the Retained Samples Curve for TMED and Eyepacs, with a
Mobilenet, for each PSR; best and second best values are marked.