disposition

# Measurability of functionals and of ideal point forecasts

Tobias Fissler Vienna University of Economics and Business (WU), Department of
Finance, Accounting and Statistics, Welthandelsplatz 1, 1020 Vienna, Austria,
e-mail<EMAIL_ADDRESS>Hajo Holzmann Philipps-Universität Marburg,
Fachbereich Mathematik und Informatik, Hans-Meerwein-Straße, 35043 Marburg,
Germany, e-mail<EMAIL_ADDRESS>
###### Abstract

Abstract. The ideal probabilistic forecast for a random variable $Y$ based on
an information set $\mathcal{F}$ is the conditional distribution of $Y$ given
$\mathcal{F}$. In the context of point forecasts aiming to specify a
functional $T$ such as the mean, a quantile or a risk measure, the ideal point
forecast is the respective functional applied to the conditional distribution.
This paper provides a theoretical justification why this ideal forecast is
actually a forecast, that is, an $\mathcal{F}$-measurable random variable. To
that end, the appropriate notion of measurability of $T$ is clarified and this
measurability is established for a large class of practically relevant
functionals, including elicitable ones. More generally, the measurability of
$T$ implies the measurability of any point forecast which arises by applying
$T$ to a probabilistic forecast. Similar measurability results are established
for proper scoring rules, the main tool to evaluate the predictive accuracy of
probabilistic forecasts.

Keywords: Bayes act; elicitability; forecast; information set; scoring
function; scoring rule

AMS 2020 Subject Classification:  62C99; 91B06

##  1 Introduction

A typical aspect of decision making, be it in business, politics, or private
life, is that decisions should account for unknown or future events. Hence,
these decisions commonly base on predictions or forecasts for these events.
Fortunately, one regularly has at least partial information about the events
of interest, such as regressors or feature variables in a cross-sectional
setting or past observations of the event in a time series framework, which
help to improve the predictions. Expressed in terms of mathematical
statistics, the event of interest is a random element $Y$ on a probability
space which often attains values in the real numbers, and the given
information is modelled as a sub-$\sigma$-algebra $\mathcal{F}$ of the
$\sigma$-algebra of events.

It has been argued in the literature that probabilistic forecasts, taking the
form of probability measures, distributions, or densities, are most
informative and should be preferred, since they directly express the
uncertainty about the future event (Dawid,, 1984; Gneiting et al.,, 2007). The
_ideal_ or truthful probabilistic forecast of $Y$ based on $\mathcal{F}$ is
the conditional distribution of $Y$ given $\mathcal{F}$, denoted by
$\mathbb{P}_{Y|\mathcal{F}}$. “However, practical situations may require
single-valued point forecasts, for reasons of decision making, reporting
requirements, or communications, among others” (Gneiting and Katzfuss,, 2014).
In such situations, one summarises the uncertainty in a typically real-valued
functional $T$ of the distribution of $Y$ such as the mean, a quantile, an
expectile, or a law-determined risk measure such as Expected Shortfall. Then,
the ideal or truthful point forecast for $Y$ based on $\mathcal{F}$ is the
functional $T$ applied to the conditional distribution of $Y$ given
$\mathcal{F}$, that is, $T(Y|\mathcal{F}):=T(\mathbb{P}_{Y|\mathcal{F}})$
(Nolde and Ziegel,, 2017). This concatenation also turns out to be the main
idea in building dynamic or conditional risk measures (Weber,, 2006).

Ideal forecasts based on $\mathcal{F}$ need to be $\mathcal{F}$-measurable
random elements, or random variables, for the case of point forecasts. On the
one hand, this ensures that they actually only exploit the information in
$\mathcal{F}$ and no additional sources. From a more technical perspective, on
the other hand, measurability ensures that one can assign probabilities to
events induced by forecasts and, more generally, that forecasts are amenable
to statistical analysis.

This paper studies when an ideal point forecast based on $\mathcal{F}$, or
more generally, a forecast which arises by applying the functional to a
probabilistic forecast based on $\mathcal{F}$, is indeed
$\mathcal{F}$-measurable. To this end, the appropriate notion of measurability
of the target functional $T$, viewed as a map from a class of probability
measures to the reals, is clarified. Moreover, we establish corresponding
measurability results for many practically relevant examples, including the
class of elicitable ones, which can be written as the minimiser of an expected
loss function (Theorem 4.1). Our results provide a formal justification for
the $\mathcal{F}$-measurability of ideal $\mathcal{F}$-based point forecasts
which seems to have been taken for granted in large parts of the forecast
evaluation literate, sometimes tacitly, sometimes with somewhat incomplete
arguments, or which has been established under restrictive assumptions
(Weber,, 2006; Holzmann and Eulert,, 2014; Nolde and Ziegel,, 2017; Pohle,,
2020; Fissler and Hoga,, 2021; Gneiting and Resin,, 2021; Fissler et al.,,
2021; Hoga and Dimitriadis,, 2021). From a regression or machine learning
perspective where $Y$ is observed together with explanatory variables $X$, our
results justify the measurability of the oracle regression function $x\mapsto
T(Y|X=x)$ beyond the classical situation of mean regression to quantile
regression (Koenker,, 2005), in expectile regression (Newey and Powell,, 1987)
or in Expected Shortfall regression (Dimitriadis and Bayer,, 2019). In the
field of sensitivity analysis, measuring the information content with score
improvements also calls for the measurability results established in our paper
(Borgonovo et al.,, 2021; Fissler and Pesenti,, 2022).

In Section 2, we provide the formal definitions and concepts of the article.
The appropriate concept of measurability of functionals is discussed in
Section 3 which leads to the $\mathcal{F}$-measurability of the ideal
$\mathcal{F}$-based forecast. We show measurability of important functionals
such as moments or weighted quantiles and discuss the connection to continuity
and robustness results discussed in statistics and finance. Measurability of
elicitable functionals is discussed and established in Section 4. Finally, we
provide sufficient conditions for the measurability of scoring rules in
Section 5. They are used to measure the predictive accuracy of probabilistic
forecasts, and map a pair of a predictive distribution $\mathrm{P}$ and an
observation $y$ to the real number $\mathrm{S}(\mathrm{P},y)$. Hence,
technically speaking, for fixed $y$ they constitute a real-valued statistical
functional, which aligns them with the main topic of this paper. Regarding the
application, the measurability of scoring rules formally justifies common
statistical practices such as computing Diebold–Mariano tests which test
predictive dominance (Diebold and Mariano,, 1995).

##  2 Statistical functionals and forecasting

Real-valued statistical functionals map probability measures on the
Borel-$\sigma$-algebra of events of the observation domain to real numbers.
More formally, the _observation domain_ $O\subseteq\mathbb{R}^{d}$ is a Borel-
measurable subset of a Euclidean space $\mathbb{R}^{d}$, where we will mainly
consider $d=1$, with _Borel- $\sigma$-algebra_ $\mathcal{O}$ of $O$. A
statistical functional on some family $\mathcal{P}$ of (Borel-)probability
measures on $(O,\mathcal{O})$ is a map $T:\mathcal{P}\to\mathbb{R}$ from
$\mathcal{P}$ to the real numbers.

A prediction space setting (Gneiting and Ranjan,, 2013) consists of a
probability space $(\Omega,\mathcal{A},\mathbb{P})$ together with a random
variable $Y:\Omega\to O$, the response variable modelling the quantity of
interest, and a sub-$\sigma$-algebra $\mathcal{F}$ of $\mathcal{A}$, the
information set on which the forecast is based. $\mathcal{F}$ can be
generated, for example, by observable explanatory variables, often called,
regressors or features, which can also contain past observations of $Y$ in a
time series setting. Then, the conditional distribution
$\mathbb{P}_{Y|\mathcal{F}}$ of $Y$ given $\mathcal{F}$ is the _ideal
probabilistic_ forecast. If a real-valued quantity shall be issued as a
forecast, it is essential to specify which statistical functional $T$ of the
conditional distribution $\mathbb{P}_{Y|\mathcal{F}}$ is the target.

A general forecast for $T$ of $\mathbb{P}_{Y|\mathcal{F}}$ is an
$\mathcal{F}$-measurable random variable, and the ideal forecast is
$T(Y|\mathcal{F}):=T\big{(}\mathbb{P}_{Y|\mathcal{F}}\big{)}$. For
$T(Y|\mathcal{F})$ to be defined, we first require that conditional
distributions in $\mathbb{P}_{Y|\mathcal{F}}$ are contained in $\mathcal{P}$,
the domain of $T$. Second, we require the $\mathcal{F}$-measurability of the
forecast $T(Y|\mathcal{F})$. Our aim is to provide conditions on $T$ which
guarantee that, when well-defined, $T(Y|\mathcal{F})$ is automatically a
forecast, that is, $\mathcal{F}$-measurable, and to show that these conditions
are satisfied by essentially all statistical functionals which arise in
applications.

To proceed, let us recall the formal definition of the conditional
distribution $\mathbb{P}_{Y|\mathcal{F}}$. A _Markov kernel_ from
$(\Omega,\mathcal{F})$ to $(O,\mathcal{O})$ is a map
$\kappa:\Omega\times\mathcal{O}\to[0,1]$ such that

  1. (i)

for each $\omega\in\Omega$, the map $B\mapsto\kappa(\omega,B)$,
$B\in\mathcal{O}$, is a probability measure on $(O,\mathcal{O})$,

  2. (ii)

for each $B\in\mathcal{O}$, the map $\omega\mapsto\kappa(\omega,B)$,
$\omega\in\Omega$, is $\mathcal{F}-\mathcal{B}[0,1]$-measurable.

A _regular version_ of the conditional probability of $Y$ given $\mathcal{F}$,
denoted by $\mathbb{P}_{Y|\mathcal{F}}$, is a Markov kernel from
$(\Omega,\mathcal{F})$ to $(O,\mathcal{O})$ such that for each
$F\in\mathcal{F}$ and $B\in\mathcal{O}$,

$\mathbb{P}\big{(}F\cap\\{Y\in
B\\}\big{)}=\int_{F}\mathbb{P}_{Y|\mathcal{F}}(\omega;B)\,{\mathrm{d}}\mathbb{P}(\omega).$

Recall that $\mathbb{P}_{Y|\mathcal{F}}$ is unique only up to almost sure
equality.

When forecasting the functional $T$ for $Y$ based on $\mathcal{F}$, we aim at
the map

$\omega\mapsto
T\big{(}\mathbb{P}_{Y|\mathcal{F}}(\omega;\cdot)\big{)}=:T(Y|\mathcal{F})(\omega),\qquad\omega\in\Omega,$
(1)

and we shall assume that the version $\mathbb{P}_{Y|\mathcal{F}}$ can be and
is chosen such that $\mathbb{P}_{Y|\mathcal{F}}(\omega;\cdot)\in\mathcal{P}$
for all $\omega\in\Omega$. If $\kappa$ is a probabilistic forecast based on
$\mathcal{F}$, that is, a Markov kernel from $(\Omega,\mathcal{F})$ to
$(O,\mathcal{O})$ for which $\kappa(\omega,\cdot)\in\mathcal{P}$ for
$\omega\in\Omega$, we may form the resulting point forecast by

$\omega\mapsto T\big{(}\kappa(\omega;\cdot)\big{)},\qquad\omega\in\Omega.$

To show that $T(Y|\mathcal{F})$ or, more generally, that
$T\big{(}\kappa(\cdot;\cdot)\big{)}$ is a forecast, that is
$\mathcal{F}$-measurable, it suffices to find a $\sigma$-algebra
$\mathcal{A}(\mathcal{P})$ such that (a) $\mathbb{P}_{Y|\mathcal{F}}$ or
$\kappa$, considered as a map from $\Omega$ to $\mathcal{P}$, is
$\mathcal{F}-\mathcal{A}(\mathcal{P})$-measurable, and (b) to show that
$T:\mathcal{P}\to\mathbb{R}$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable, where $\mathcal{B}$ is the
Borel-$\sigma$-algebra of $\mathbb{R}$.

##  3 Measurability of functionals

We shall choose $\mathcal{A}(\mathcal{P})$ as the projection $\sigma$-algebra.
More formally, let $\mathcal{Q}$ be the family of all probability measures
$\mathrm{P}$ on $(O,\mathcal{O})$, and for each $B\in\mathcal{O}$, consider
the evaluation map

$\pi_{B}(\mathrm{P})=\mathrm{P}(B),\qquad\mathrm{P}\in\mathcal{Q}.$

Then, on $\mathcal{Q}$ we can consider the smallest $\sigma$-algebra
$\mathcal{A}(\mathcal{Q})$ which makes all evaluation maps
$\mathcal{A}(\mathcal{Q})-\mathcal{B}[0,1]$-measurable. For a subset
$\mathcal{P}\subseteq\mathcal{Q}$ we denote by
$\mathcal{A}(\mathcal{P})=\\{\mathcal{P}\cap A\mid
A\in\mathcal{A}(\mathcal{Q})\\}$ the trace $\sigma$-algebra. Note that
$\mathcal{A}(\mathcal{P})$ is also the $\sigma$-algebra generated by the
restrictions of the evaluation maps $\pi_{B}$ to $\mathcal{P}$, which follows
from the simple fact that for any $C\in\mathcal{B}[0,1]$ and any
$B\in\mathcal{O}$ we have
$\pi_{B|\mathcal{P}}^{-1}(C)=\mathcal{P}\cap\pi_{B}^{-1}(C)$. Moreover,
$\mathcal{A}(\mathcal{P})$ can be regarded as the Borel-$\sigma$-algebra
induced by the topology of setwise convergence.

######  Lemma 3.1.

Let $\kappa:\Omega\times\mathcal{O}\to[0,1]$ be a map such that for each
$\omega\in\Omega$, $\kappa(\omega,\cdot)\in\mathcal{P}$. Then $\kappa$ is a
Markov kernel from $(\Omega,\mathcal{F})$ to $(O,\mathcal{O})$ if and only if
the map $\omega\to\kappa(\omega,\cdot)$ from $\Omega$ to $\mathcal{P}$ is
$\mathcal{F}-\mathcal{A}(\mathcal{P})$-measurable.

###### Proof.

$\kappa$ is a Markov kernel if and only if for each $B\in\mathcal{O}$, the map
$\omega\to\kappa(\omega,B)=\pi_{B}(\kappa(\omega,\cdot))$ is
$\mathcal{F}-\mathcal{B}[0,1]$-measurable, which is the case if and only if
$\omega\to\kappa(\omega,\cdot)$ is
$\mathcal{F}-\mathcal{A}(\mathcal{P})$-measurable. ∎

Concerning the measurability of the forecasts $T(Y|\mathcal{F})$ of
functionals $T$, we observe the following.

######  Lemma 3.2.

If the functional $T:\mathcal{P}\to\mathbb{R}$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable, then $T(Y|\mathcal{F})$
defined in (1) is $\mathcal{F}-\mathcal{B}$-measurable.

###### Proof.

This is clear since by Lemma 3.1, the conditional distribution
$\mathbb{P}_{Y|\mathcal{F}}$ is
$\mathcal{F}-\mathcal{A}(\mathcal{P})$-measurable as a map from $\Omega$ to
$\mathcal{P}$, and $T$ is $\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable by
assumption. ∎

Similarly to Lemma 3.2, the general forecast $\omega\mapsto
T\big{(}\kappa(\omega;\cdot)\big{)}$, $\omega\in\Omega$, where $\kappa$ is a
Markov kernel from $(\Omega,\mathcal{F})$ to $(O,\mathcal{O})$, is
$\mathcal{F}-\mathcal{B}$-measurable. Thus, to clarify the measurability of
$T\big{(}\kappa(\cdot;\cdot)\big{)}$ and of $T(Y|\mathcal{F})$ in (1) it
suffices to investigate $T$. Before we turn to a general result for elicitable
functionals, let us investigate several important examples. In the following
we identify any probability distribution $\mathrm{P}\in\mathcal{Q}$ with its
distribution function $F=F_{\mathrm{P}}$ defined by
$F(x)=\mathrm{P}(O\cap(-\infty,x])$, $x\in O$.

######  Example 3.3 (Moments).

Let $h:O\to\mathbb{R}$ be a measurably function. Let $\mathcal{P}_{h}$ be a
family of probability measures on $(O,\mathcal{O})$ satisfying
$\int_{O}\,|h(y)|\,{\mathrm{d}}\mathrm{P}(y)<\infty$, and consider the mean
functional

$T_{h}(\mathrm{P})=\int_{O}\,h(y)\,{\mathrm{d}}\mathrm{P}(y),\qquad\mathrm{P}\in\mathcal{P}_{h}.$

Then $T_{h}$ is $\mathcal{A}(\mathcal{P}_{h})-\mathcal{B}$-measurable.

Indeed, if $h(y)=\mathds{1}_{B}(y)$ for some $B\in\mathcal{O}$, then
$T_{h}=\pi_{B}$ and the claim follows from the measurability of the evaluation
map. If $h$ is a simple function, the resulting functional will be a finite
linear combination of evaluation maps, and hence also measurable. For $h\geq
0$, there is a sequence of non-negative simple functions $(h_{n})$ with
$h_{n}\uparrow h$ pointwise, and then $T_{h_{n}}(\mathrm{P})\uparrow
T_{h}(\mathrm{P})$ for every $\mathrm{P}$. Hence, as a pointwise limit of the
measurable functions $T_{h_{n}}$, $T_{h}$ is also measurable. For a general
$h$, the measurability follows by the decomposition
$T_{h}=T_{h^{+}}-T_{h^{-}}$.

Example 3.3 directly yields the measurability of Borel-measurable functions of
multiple moments. This yields, for example, the measurability of the variance,
the skewness, the kurtosis, or of the Sharpe ratio.

######  Example 3.4 (Quantiles).

Let $O=\mathbb{R}$, fix $\alpha\in[0,1]$ and let

$\displaystyle T(F)=q_{\alpha}^{-}(F)$
$\displaystyle=\inf\\{x\in\mathbb{R}\mid
F(x)\geq\alpha\\}\in\bar{}\mathbb{R}=\mathbb{R}\cup\\{-\infty,\infty\\},\qquad
F\in\mathcal{P},$

be the lower $\alpha$-quantile of $F$, which is its essential supremum if
$\alpha=1$. Recall that $\mathcal{B}(\bar{}\mathbb{R})=\\{A\cup E\mid
A\in\mathcal{B}(\mathbb{R}),\ E\subseteq\\{-\infty,\infty\\}\\}$.

To show that $T$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}(\bar{}\mathbb{R})$-measurable, consider
the family of evaluation maps $\pi_{(-\infty,x]}(F)=F(x)$, for
$F\in\mathcal{P}$, $x\in\mathbb{R}$. Each $\pi_{(-\infty,x]}$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}([0,1])$-measurable.

Now, given $p\in\mathbb{R}$, the lower $\alpha$-quantile $q_{\alpha}^{-}(F)$
of $F$ is strictly larger than $p$, $q_{\alpha}^{-}(F)>p$, if and only if
$\alpha>F(p)$. Therefore

$\displaystyle\\{F\in\mathcal{P}\mid q_{\alpha}^{-}(F)>p\\}$
$\displaystyle=\\{F\in\mathcal{P}\mid\pi_{(-\infty,p]}(F)<\alpha\\},$

showing measurability of the lower $\alpha$-quantile, where we exploit the
fact that it suffices to show measurability on a generator of
$\mathcal{B}(\bar{}\mathbb{R})$.

Similar considerations yield the
$\mathcal{A}(\mathcal{P})-\mathcal{B}(\bar{}\mathbb{R})$-measurability of the
upper $\alpha$-quantile, $\alpha\in[0,1]$,

$\displaystyle T(F)=q_{\alpha}^{+}(F)$
$\displaystyle=\sup\\{x\in\mathbb{R}\mid\lim_{t\uparrow
x}F(t)\leq\alpha\\}\in\bar{}\mathbb{R},\qquad F\in\mathcal{P},$

where $q_{0}^{+}(F)$ corresponds to the essential infimum of $F$. For
$\alpha\in(0,1)$, we can also consider the $\alpha$-quantile, $q_{\alpha}$, as
an interval-valued functional
$q_{\alpha}(F)=[q_{\alpha}^{-}(F),q_{\alpha}^{+}(F)]\subset\mathbb{R}$. Then,
the previously established measurability results yield that $q_{\alpha}$ is an
_Effros measurable_ closed-valued multifunction in the sense that for all open
sets $G\subseteq\mathbb{R}$, $\\{F\in\mathcal{P}\mid q_{\alpha}(F)\cap
G\neq\emptyset\\}\in\mathcal{A}(\mathcal{P})$, see Definition 1.3.1 in
Molchanov, (2017). Therefore, the ideal forecast $q_{\alpha}(Y|\mathcal{F})$
is a random closed set in the sense of Definition 1.1.1 in Molchanov, (2017).
This complements the measurability argument of the quantile provided recently
in de Castro et al., (2021).

In the risk management literature, the lower $\alpha$-quantile is known as
Value at Risk at level $\alpha$, $\mathrm{VaR}_{\alpha}$. The following
example is concerned with another important quantitative risk measure, the
Expected Shortfall.

######  Example 3.5 (Weighted averages of quantiles).

Let $w\colon[0,1]\to[0,\infty)$ be a measurable weight function such that the
left-sided and the right-sided limits exists for all $x\in[0,1]$. Consider the
functional

$T_{w}(F)=\int_{0}^{1}q_{\gamma}^{-}(F)w(\gamma)\,{\mathrm{d}}\gamma\in\bar{}\mathbb{R},\qquad
F\in\mathcal{P}_{w},$

where $\mathcal{P}_{w}$ is the domain such that $T_{w}$ is well-defined. The
measurability of $q_{\gamma}^{-}$ from Example 3.4 and a Riemann approximation
argument yield that $T_{w}$ is
$\mathcal{A}(\mathcal{P}_{w})-\mathcal{B}(\bar{}\mathbb{R})$-measurable.

This directly yields the measurability of the lower and upper Expected
Shortfall (by choosing $w=\mathds{1}_{[0,\alpha]}/\alpha$ or
$w=\mathds{1}_{[\alpha,1]}/(1-\alpha)$) and of the Range Value at Risk (Cont
et al.,, 2010; Fissler and Ziegel,, 2021) (by choosing
$w=\mathds{1}_{[\alpha,\beta]}/(\beta-\alpha)$ for $0<\alpha<\beta<1$).

To treat the next Example 3.7, we first need to establish the following
technical lemma.

######  Lemma 3.6.

Let $T\colon\mathcal{P}\to\mathbb{R}$ be an
$\mathcal{A}(\mathcal{P})-\mathcal{B}(\mathbb{R})$-measurable functional. Then
the composed evaluation maps $\mathcal{P}\to[0,1]$,

$F\mapsto\pi_{(T(F),\infty)}(F),\qquad F\mapsto\pi_{(-\infty,T(F))}(F),\qquad
F\mapsto\pi_{\\{T(F)\\}}(F)$ (2)

are $\mathcal{A}(\mathcal{P})-\mathcal{B}([0,1])$-measurable.

###### Proof.

For the first map in (2), let $p\in[0,1]$ and consider

$\displaystyle\\{F\in\mathcal{P}\mid\pi_{(T(F),\infty)}(F)\leq p\\}$
$\displaystyle=\\{F\in\mathcal{P}\mid F(T(F))\geq
1-p\\}=\\{F\in\mathcal{P}\mid T(F)\geq q_{1-p}^{-}(F)\\}$
$\displaystyle=\\{F\in\mathcal{P}\mid(T-q_{1-p}^{-})(F)\geq
0\\}\in\mathcal{A}(\mathcal{P}).$

The last assertion comes from the fact that $q_{1-p}^{-}$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}(\mathbb{R})$-measurable and that
differences of measurable functionals are also measurable.
For the second map in (2), the arguments are similar. Using the shorthand
$F(x-):=\lim_{t\uparrow x}F(t)$ for the left-sided limit, we obtain for
$p\in[0,1]$

$\displaystyle\\{F\in\mathcal{P}\mid\pi_{(-\infty,T(F))}(F)\leq p\\}$
$\displaystyle=\\{F\in\mathcal{P}\mid F(T(F)-)\leq p\\}=\\{F\in\mathcal{P}\mid
T(F)\leq q_{p}^{+}(F)\\}$
$\displaystyle=\\{F\in\mathcal{P}\mid(T-q_{p}^{+})(F)\leq
0\\}\in\mathcal{A}(\mathcal{P}).$

Finally, note that
$\pi_{\\{T(F)\\}}(F)=1-\pi_{(T(F),\infty)}(F)-\pi_{(-\infty,T(F))}(F)$. ∎

######  Example 3.7.

Conditional Value at Risk (CoVaR) and conditional Expected Shortfall (CoES)
are influential systemic risk measures due to Adrian and Brunnermeier, (2016).
In a nutshell, they assess the riskiness of a position $Y$ given that a
reference position $X$ is at risk. Following Girardi and Tolga Ergün, (2013),
the latter conditioning event is interpreted as $X$ exceeding its Value at
Risk at level $\beta$, $\mathrm{VaR}_{\beta}$. Hence, these risk measures can
be viewed as functionals, first mapping a bivariate distribution
$\mathrm{P}_{X,Y}$ to the univariate conditional distribution
$\mathrm{P}_{Y\mid X\geq\mathrm{VaR}_{\beta}(X)}$ and then applying a risk
measure such as $\mathrm{VaR}_{\alpha}$ (in the case of CoVaR) or
$\mathrm{ES}_{\alpha}$ (in the case of CoES) to this univariate distribution.
Since $\mathrm{VaR}_{\alpha}$ and $\mathrm{ES}_{\alpha}$ are measurable
functionals from univariate distributions to $\mathbb{R}$, we obtain the
measurability of these conditional risk measures if the map
$\mathrm{P}_{X,Y}\mapsto\mathrm{P}_{Y\mid X\geq\mathrm{VaR}_{\beta}(X)}$ is
measurable. To account for possible discontinuities in the marginal
distribution of the first component $X$, we follow Remark C.1 in Fissler and
Hoga, (2021) and incorporate a correction term to this map. Hence, we formally
consider $\eta\colon\mathcal{P}^{2}\to\mathcal{P}$, where $\mathcal{P}^{2}$ is
a set of Borel-distributions on $\mathbb{R}^{2}$ and $\mathcal{P}$ is a set of
such distributions on $\mathbb{R}$, such that for a Borel set
$B\in\mathcal{B}(\mathbb{R})$ we get

$\displaystyle\eta(\mathrm{P}_{X,Y})(B)=\mathrm{P}_{Y\mid
X\succcurlyeq\mathrm{VaR}_{\beta}(X)}(B)$
$\displaystyle:=\frac{1}{1-\beta}\Big{[}\mathrm{P}_{X,Y}\big{(}(\mathrm{VaR}_{\beta}(\mathrm{P}_{X}),\infty)\times
B\big{)}$
$\displaystyle\qquad+\Big{(}1-\beta-\mathrm{P}_{X}\big{(}(\mathrm{VaR}_{\beta}(\mathrm{P}_{X}),\infty)\big{)}\Big{)}\mathrm{P}_{X,Y}\big{(}\\{\mathrm{VaR}_{\beta}(\mathrm{P}_{X})\\}\times
B\big{)}/\mathrm{P}_{X}\big{(}\\{\mathrm{VaR}_{\beta}(\mathrm{P}_{X})\\}\big{)}\Big{]}$

For the last term, note that
$\mathrm{P}_{X}\big{(}\\{\mathrm{VaR}_{\beta}(\mathrm{P}_{X})\\}\big{)}=0$
implies that
$1-\beta-\mathrm{P}_{X}\big{(}(\mathrm{VaR}_{\beta}(\mathrm{P}_{X}),\infty)\big{)}=0$
for which case we set $0/0=0$. To show the
$\mathcal{A}(\mathcal{P}^{2})-\mathcal{A}(\mathcal{P})$-measurability of
$\eta$ it is sufficient to show that for any $B\in\mathcal{B}(\mathbb{R})$,
the concatenation with the evaluation map $\pi_{B}\circ\eta$ is
$\mathcal{A}(\mathcal{P}^{2})-\mathcal{B}([0,1])$-measurable. This
measurability follows from Lemma 3.6 and the fact that for any
$C\in\mathcal{B}(\mathbb{R})$ the map
$\mathrm{P}_{X,Y}\mapsto\mathrm{P}(\cdot\times C)$ is
$\mathcal{A}(\mathcal{P}^{2})-\mathcal{A}(\mathcal{P})$-measurable.

######  Remark 3.8 (Continuity of statistical functionals).

Apart from mere measurability, continuity or even differentiability properties
of statistical functionals are of interest in statistics, finance and
econometrics. A statistical functional is weakly continuous at
$\mathrm{P}\in\mathcal{P}$ if the weak convergence
$\mathrm{P}_{n}\Rightarrow\mathrm{P}$ of a sequence $(\mathrm{P}_{n})$ in
$\mathcal{P}$ to $\mathrm{P}$ implies that $T(\mathrm{P}_{n})\to
T(\mathrm{P})$. Let $\mathcal{B}(\mathcal{P})$ denote the
Borel-$\sigma$-algebra generated by the weak topology on $\mathcal{P}$. If a
statistical functional $T$ is weakly continuous at every
$\mathrm{P}\in\mathcal{P}$ it is also $\mathcal{B}(\mathcal{P})$-measurable.
Since $\mathcal{B}(\mathcal{P})\subseteq\mathcal{A}(\mathcal{P})$, this
readily implies that $T$ is also measurable in the sense defined above.
However, apart from the Range Value at Risk the functionals discussed above
are known not to be continuous at every $\mathrm{P}$ under the general
assumptions that we impose: For the mean, the discontinuity follows from Huber
and Ronchetti, (2009, Lemma 2.1), while the lower $\alpha$-quantile,
$\alpha\in(0,1)$, is weakly continuous at $F$ if and only if the quantile
function $\gamma\mapsto q_{\gamma}^{-}(F)$ is continuous at $\alpha$, that is,
if and only if $q_{\alpha}^{-}(F)=q_{\alpha}^{+}(F)$; see van der Vaart,
(1998, Lemma 21.2).

In the finance literature, statistical functionals arise from law-determined
(often called law-invariant) risk measures and are often called risk
functionals (Krätschmer et al.,, 2014). Assuming monotonicity with respect to
first order stochastic dominance and translation equivariance, Weber, (2006,
Lemma 2.1 and Corollary 2.1) establishes the weak continuity of risk
functionals on the class of probability distributions with compact support,
implying the measurability. While this applies to quantiles and weighted
averages of quantiles with weight function $w$ satisfying $\int
w(\gamma)\,{\mathrm{d}}\gamma=1$, we do not require compactly supported
distributions.
Generalizing Weber, (2006), Krätschmer et al., (2014) show that monotone and
translation equivariant risk functionals arising from convex risk measures on
$\mathcal{P}\subseteq\mathcal{Q}^{p}=\\{\mathrm{P}\in\mathcal{Q}\mid\int|x|^{p}\,{\mathrm{d}}\mathrm{P}(x)\\}$
are continuous in the Wasserstein metric of order $p\in[1,\infty)$. Similarly,
Kiesel et al., (2016) show the continuity of $L$-functionals in the
Wasserstein metric of order $p\in[1,\infty)$. Since a sequence of probability
measures $(\mathrm{P}_{n})$ in $\mathcal{P}$ converges to $\mathrm{P}$ in the
Wasserstein metric of order $p$ if and only if it converges weakly and
$\int|x|^{p}\,{\mathrm{d}}\mathrm{P}_{n}(x)\to\int|x|^{p}\,{\mathrm{d}}\mathrm{P}(x)$,
the continuity in the Wasserstein metric of order $p$ implies
$\mathcal{A}(\mathcal{P})$-measurability. This yields an alternative argument
for the $\mathcal{A}(\mathcal{P})$-measurability of the lower and upper
expected shortfall.

##  4 Elicitable functionals

Let us now turn to general elicitable functionals. These are functionals such
that the ideal forecast corresponds to a Bayes act in the sense that it
minimizes an expected loss function. Again, let $O\subseteq\mathbb{R}$ be a
measurable subset, let $A$ be an interval, and let $\mathcal{P}$ be a family
of Borel probability measures on $(O,\mathcal{O})$. Let

$\mathrm{L}:A\times O\to\mathbb{R}$

be a scoring or loss function with the following properties:

  1. (i)

For each $a\in A$ and $\mathrm{P}\in\mathcal{P}$, $\mathrm{L}(a,\cdot)$ is
$\mathrm{P}$-integrable. Set

$\overline{\mathrm{L}}(a,\mathrm{P})=\int_{O}\mathrm{L}(a,y)\,{\mathrm{d}}\mathrm{P}(y).$

We also write $\overline{\mathrm{L}}(a,F)$ if $F$ is a distribution function
of a distribution $\mathrm{P}$ in $\mathcal{P}$.

  2. (ii)

For each $F\in\mathcal{P}$, the function $a\mapsto\overline{\mathrm{L}}(a,F)$,
$a\in A$,

    1. (a)

is continuous,

    2. (b)

has a compact, non-empty interval of minimizers, denoted by
$I_{\mathrm{L}}(F)$,

    3. (c)

is monotonically decreasing to the left of $I_{\mathrm{L}}(F)$, and
monotonically increasing to the right.

Then $\mathrm{L}$ is _$\mathcal{P}$ -consistent_ for the functionals
$T_{\min}(F)=\min I_{\mathrm{L}}(F)$ and $T_{\max}(F)=\max I_{\mathrm{L}}(F)$
meaning that $T_{\min}(F)$ and $T_{\max}(F)$ minimize the expected loss
function $a\mapsto\overline{\mathrm{L}}(a,F)$ for each $F\in\mathcal{P}$. If
$I_{\mathrm{L}}(F)$ is a singleton for each $F\in\mathcal{P}$, then
$T_{\min}=T_{\max}=:T$, then $\mathrm{L}$ is _strictly
$\mathcal{P}$-consistent_ for $T$, meaning that $T$ uniquely minimizes the
expected loss. A functional is _elicitable_ on $\mathcal{P}$ if there exists a
strictly $\mathcal{P}$-consistent scoring function for it.

If the class $\mathcal{P}$ is convex and if the continuity assumption a) on
the expected loss holds, then the strict $\mathcal{P}$-consistency of
$\mathrm{L}$ for $T_{\min}=T_{\max}$ already implies properties b) and c) as
shown in Fissler and Ziegel, (2019, Proposition 2.2), Nau, (1985, Proposition
3), Bellini and Bignozzi, (2015, Proposition 3.4), Lambert, (2019, Proposition
11); see also Steinwart et al., (2014, Theorem 5) for a related result.
Properties b) and c) are known as _order-sensitivity_ or _accuracy-rewarding_
in the literature; see Lambert et al., (2008) and the references above. As
noted in the discussion below Proposition 2.2 in Fissler and Ziegel, (2019), a
sufficient condition for the continuity of the expected loss
$a\mapsto\overline{\mathrm{L}}(a,F)$ for each $F\in\mathcal{P}$ are conditions
a), b) and c) on the level of the loss $a\mapsto\mathrm{L}(a,y)$ itself for
each $y\in O$. The continuity of $\mathrm{L}(a,y)$ in its first argument is a
common regularity condition; see e.g. Gneiting, (2011).

Define the entropy, uncertainty, or Bayes risk of the $\mathcal{P}$-consistent
loss $\mathrm{L}$ for $T$ as
$T_{\mathrm{ent}}(F)=\overline{\mathrm{L}}\big{(}T(F),F\big{)}$,
$F\in\mathcal{P}$.

######  Theorem 4.1.

Under the above assumptions, the functionals $T_{\min}$, $T_{\max}$ and
$T_{\mathrm{ent}}$ are $\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable.

###### Proof.

First consider $T_{\min}$. By Example 3.3, for each $a\in A$ the functional
$F\mapsto\overline{\mathrm{L}}(a,F)$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable. We then argue similarly as
in Example 3.4: Let $a\in A$ and $F\in\mathcal{P}$. We claim that under our
assumptions, $T_{\min}(F)>a$ holds if and only if there exists $b\in
A\cap\mathbb{Q}$ with $b>a$ such that
$\overline{\mathrm{L}}(a,F)>\overline{\mathrm{L}}(b,F)$. Indeed, by definition
of $T_{\min}$, $T_{\min}(F)>a$ holds if and only if $a$ is to the left of
$I_{\mathrm{L}}(F)$. By monotonicity and continuity of
$\overline{\mathrm{L}}(\cdot,F)$, this holds if and only if there exists
$b\in(a,\infty)\cap\mathbb{Q}\cap A$ such that
$\overline{\mathrm{L}}(a,F)>\overline{\mathrm{L}}(b,F)$. Therefore,

$\displaystyle\\{F\in\mathcal{P}\mid T_{\min}(F)>a\\}$
$\displaystyle=\bigcup_{b\in(a,\infty)\cap\mathbb{Q}\cap
A}\\{F\in\mathcal{P}\mid\overline{\mathrm{L}}(a,F)-\overline{\mathrm{L}}(b,F)>0\\},$

showing measurability of $T_{\min}(F)$. The arguments for $T_{\max}$ work
similarly.

For $T_{\mathrm{ent}}$, we obtain the measurability upon noting that for any
$x\in\mathbb{R}$

$\displaystyle\\{F\in\mathcal{P}\mid T_{\mathrm{ent}}(F)<x\\}$
$\displaystyle=\bigcup_{a\in\mathbb{Q}\cap
A}\\{F\in\mathcal{P}\mid\overline{\mathrm{L}}(a,F)<x\\}.$

∎

######  Example 4.2 (Expectiles and generalized quantiles).

On the class of distributions with finite mean, Newey and Powell, (1987)
introduced the $\tau$-expectile of a distribution $\mathrm{P}$,
$e_{\tau}(\mathrm{P})$, $\tau\in(0,1)$, as the unique solution to the equation

$\tau\int_{(x,\infty)}(y-x)\,\mathrm{d}\mathrm{P}(y)=(1-\tau)\int_{(-\infty,x]}(x-y)\,\mathrm{d}\mathrm{P}(y)$

in $x$. On the class of square-integrable distributions, $e_{\tau}$ is
elicitable with the asymmetric piecewise quadratic loss
$\mathrm{L}_{\tau}(a,y)=|\mathds{1}\\{y\leq a\\}-\tau|(a-y)^{2}$ as a strictly
consistent loss. Hence, Theorem 4.1 establishes the measurability of
$e_{\tau}$ on this class. For the measurability on the class of integrable
distributions, one can consider the slightly modified loss function
$\widetilde{\mathrm{L}}_{\tau}(a,y)=\mathrm{L}_{\tau}(a,y)-\mathrm{L}_{\tau}(0,y)$.

Bellini et al., (2014) study _generalized_ $\tau$-quantiles which arise as
minimizers of the expected loss
$L_{\tau,\phi_{1},\phi_{2}}(a,y)=\mathds{1}\\{y\leq
a\\}(1-\tau)\phi_{1}(|a-y|)+\mathds{1}\\{y>a\\}\tau\phi_{2}(|a-y|)$ for two
convex and strictly increasing functions
$\phi_{1},\phi_{2}:[0,\infty)\to[0,\infty)$. Clearly, if $\phi_{1}$ and
$\phi_{2}$ are both the absolute function (the squared function) the
$\tau$-quantile ($\tau$-expectile) arises. Elliott et al., (2005) studied the
situation of general power functions. Theorem 4.1 then establishes the
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurability of these generalized
$\tau$-quantiles.

######  Example 4.3 (Variance and Expected Shortfall).

On the class of square-integrable distributions $\mathcal{P}$, the squared
loss $L(a,y)=(a-y)^{2}$ is a strictly $\mathcal{P}$-consistent loss for the
mean functional. The corresponding entropy functional $T_{\mathrm{ent}}$ is
the variance functional. Therefore, Theorem 4.1 establishes the
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurability of the variance as an
alternative argument to the direct application of Example 3.3.

For the lower or the upper Expected Shortfall at level $\alpha$ it is well
known (Embrechts and Wang,, 2015, Lemmas 2.3 and 3.3) that both versions can
be written as the entropy of a consistent loss function for $q_{\alpha}^{-}$.
Therefore, Theorem 4.1 establishes an alternative argument for the
measurability to that provided in Example 3.5.

An alternative approach to establish the measurability of a functional is to
exploit identifiability instead of elicitability. A functional is called
identifiable if there exists a moment or identification function
$\mathrm{V}\colon A\times O\to\mathbb{R}$ such that the functional is the
unique zero of the expected identification function. Indeed, under some
technical assumptions, elicitability and identifiability are equivalent for
one-dimensional functionals (Steinwart et al.,, 2014, Theorem 5), and
concerning measurability, one ends up with similar results and examples as
considered in this section.

######  Remark 4.4.

Proposition 3.7 in Bellini and Bignozzi, (2015) establishes $\psi$-weak
continuity of elicitable functionals on classes of probability measures with
compact support under additional regularity conditions on the corresponding
strictly consistent loss function. $\psi$-weak continuity has been discussed
in Krätschmer et al., (2014) and generalizes continuity in the Wasserstein
metric. It implies $\mathcal{A}(\mathcal{P})$-measurability in the sense
studied in this paper.

##  5 Measurability of scoring rules

In this section we briefly discuss measurability of scoring rules, which are
used to compare probabilistic forecasts. Consider maps

$\mathrm{S}:\mathcal{P}\times O\to(-\infty,\infty],$ (3)

such that for each $\mathrm{P}\in\mathcal{P}$, $\mathrm{S}(\mathrm{P};\cdot)$
is Borel-measurable and that for each $\mathrm{Q}\in\mathcal{P}$,

$\mathbb{E}_{\mathrm{Q}}\big{[}\mathrm{S}(\mathrm{P};Y)\big{]}=\int_{O}\mathrm{S}(\mathrm{P};y)\,{\mathrm{d}}\mathrm{Q}(y)$

exists. Note that we allow $\infty$ in the range of $\mathrm{S}$ in (3) to
treat the common logarithmic score, see Example 5.4. $\mathrm{S}$ is called a
strictly $\mathcal{P}$-proper scoring rule if

$\mathbb{E}_{\mathrm{Q}}\big{[}\mathrm{S}(\mathrm{Q};Y)\big{]}<\mathbb{E}_{\mathrm{Q}}\big{[}\mathrm{S}(\mathrm{P};Y)\big{]}\qquad\text{for
all }\ \mathrm{P},\mathrm{Q}\in\mathcal{P},\ \mathrm{P}\not=\mathrm{Q},$

meaning that forecasting the true distribution $\mathrm{Q}$ uniquely minimizes
the expected score.

Recall that the ideal probabilistic forecast based on an information set
$\mathcal{F}$ is the conditional distribution $\mathbb{P}_{Y|\mathcal{F}}$.
That means in a forecasting situation, with available information
$\mathcal{F}$ (e.g. given by regressors), it is natural that the forecasts
have the form of Markov kernels from $(\Omega,\mathcal{F})$ to
$(O,\mathcal{O})$. Inserting a Markov kernel $\kappa$ for $\mathrm{P}$ leads
to the map

$\omega\mapsto\mathrm{S}\big{(}\kappa(\omega,\cdot);Y(\omega)\big{)},$ (4)

and averages of such expressions, which estimate expected values, are used
when comparing different forecasts. But in order to form the expected value in
(4) we in particular require measurability, i.e. we need that (4) is an
$\mathcal{A}-\mathcal{B}$-measurable random variable. Since
$\omega\mapsto\big{(}\kappa(\omega,\cdot);Y(\omega)\big{)}$ is
$\mathcal{A}-\mathcal{A}(\mathcal{P})\otimes\mathcal{O}$-measurable, this will
be guaranteed if we assume that $\mathrm{S}$ in (3) is
$\mathcal{A}(\mathcal{P})\otimes\mathcal{O}-\mathcal{B}$-measurable.

######  Lemma 5.1.

Suppose that $O$ is a one-dimensional interval, and that

  1. (i)

for each $y\in O$, the map
$\mathrm{S}(\cdot;y):\mathcal{P}\to(-\infty,\infty]$ is
$\mathcal{A}(\mathcal{P})-\mathcal{B}$-measurable,

  2. (ii)

for each $\mathrm{P}\in\mathcal{P}$, the map $\mathrm{S}(\mathrm{P};\cdot)$ is
right-continuous (or left-continuous).

Then $\mathrm{S}$ in (3) is
$\mathcal{A}(\mathcal{P})\otimes\mathcal{O}-\mathcal{B}$-measurable.

###### Proof.

For a finite interval, say $O=(0,1]$, for $n\in\mathbb{N}$, by (i) the map

$\mathrm{S}_{n}(\mathrm{P},y)=\sum_{k=1}^{2^{n}}\,\mathrm{S}(\mathrm{P};k/2^{n})\,\mathds{1}_{((k-1)/2^{n},k/2^{n}]}(y)$

is $\mathcal{A}(\mathcal{P})\otimes\mathcal{O}-\mathcal{B}$-measurable and by
(ii), $\mathrm{S}_{n}(\mathrm{P},y)\to\mathrm{S}(\mathrm{P},y)$ for
$n\to\infty$. An infinite interval can be treated analogously by extending the
sum to an infinite one. ∎

######  Remark 5.2.

The argument in Lemma 5.1 can be used to obtain a similar result for
multivariate observations, where right-continuity is defined, as for
multivariate distribution functions, by taking limits from above in each
coordinate. However, the form of the observation domain $O$ is also an issue,
and to proceed as above we require that $O$ is a possibly infinite
hyperrectangle.

######  Example 5.3 (Continuous ranked probability score).

The continuous ranked probability score (CRPS) of a distribution $F$ with
finite first moment is

$\operatorname{CRPS}(F,y)=\int_{-\infty}^{\infty}\big{(}F(z)-\mathds{1}\\{y\leq
z\\}\big{)}^{2}\,\mathrm{d}z,\quad F\in\mathcal{P},$ (5)

see Gneiting and Raftery, (2007). It has an alternative representation in
terms of an integrated quantile loss, and also as

$\displaystyle\operatorname{CRPS}(F,y)$
$\displaystyle=\mathbb{E}_{F}\big{[}\big{|}y-X\big{|}\big{]}-\frac{1}{2}\,\mathbb{E}_{F}\big{[}\big{|}X^{\prime}-X\big{|}\big{]},$
(6)

where $X$ and $X^{\prime}$ are independent copies with distribution $F$. From
both (6) and (5) one can readily see that $\operatorname{CRPS}(F,y)$ is
continuous as a function of $y$, invoking the dominated convergence theorem.
Further, for fixed $y$, (6) implies measurability of
$\operatorname{CRPS}(F,y)$ as a map in $F$, while from (5) the measurability
can be obtained with a Riemann sum approximation similarly as in Example 3.5.
Similar arguments can be made and hence the measurability is obtained for
weighted versions of the CRPS as introduced in Gneiting and Ranjan, (2011) and
Holzmann and Klar, (2017).

######  Example 5.4 (Logarithmic score and local scoring rules).

Assume that $\mathcal{P}$ is a family of right-continuous Lebesgue densities
on the real line. We write $\mathrm{p}\in\mathcal{P}$ and denote the
distribution function associated with $\mathrm{p}$ by $F_{\mathrm{p}}$. The
logarithmic score is

$\operatorname{LOGS}(\mathrm{p},y)=-\log\mathrm{p}(y),\qquad\mathrm{p}\in\mathcal{P},$

see Good, (1952) and Gneiting and Raftery, (2007).

To apply Lemma 5.1, observe that $\operatorname{LOGS}(\mathrm{p},y)$ is right-
continuous as a function of $y$ by our assumption on $\mathrm{p}$ and
continuity of the logarithm. For the measurability in $\mathrm{p}$ for fixed
$y$, note that by right-continuity,

$\lim_{n\to\infty}\log\Big{(}n\,\big{(}F_{\mathrm{p}}(y+1/n)-F_{\mathrm{p}}(y)\big{)}\Big{)}=\log\mathrm{p}(y),$

showing measurability, since the evaluation of the distribution function is
measurable. Measurability extends to weighted versions of the logarithmic
score as in Diks et al., (2011) and in Holzmann and Klar, (2017), and under
appropriate assumptions on higher derivatives to higher-order local scoring
rules (Ehm and Gneiting,, 2012; Parry et al.,, 2012) such as the Hyvärinen
score (Hyvärinen,, 2005).

## Acknowledgements

We would like to thank Johannes Resin for valuable feedback on an earlier
version of the paper.

## References

  * Adrian and Brunnermeier, (2016) Adrian, T. and Brunnermeier, M. K. (2016).  CoVaR.  Am. Econ. Rev., 106(7):1705–1741.
  * Bellini and Bignozzi, (2015) Bellini, F. and Bignozzi, V. (2015).  On elicitable risk measures.  Quant. Finance, 15(5):725–733.
  * Bellini et al., (2014) Bellini, F., Klar, B., Müller, A., and Rosazza Gianin, E. (2014).  Generalized quantiles as risk measures.  Insurance Math. Econom., 54:41–48.
  * Borgonovo et al., (2021) Borgonovo, E., Hazen, G. B., Jose, V. R. R., and Plischke, E. (2021).  Probabilistic sensitivity measures as information value.  Eur. J. Oper. Res., 289(2):595–610.
  * Cont et al., (2010) Cont, R., Deguest, R., and Scandolo, G. (2010).  Robustness and sensitivity analysis of risk measurement procedures.  Quant. Finance, 10:593–606.
  * Dawid, (1984) Dawid, A. P. (1984).  Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach.  J. Roy. Statist. Soc. Ser. A, 147(2):278–292.
  * de Castro et al., (2021) de Castro, L. I., Costa, B. N., Galvao, A. F., and Zubelli, J. (2021).  Conditional quantiles: An operator-theoretical approach.  Preprint.  https://dx.doi.org/10.2139/ssrn.3924597.
  * Diebold and Mariano, (1995) Diebold, F. X. and Mariano, R. S. (1995).  Comparing predictive accuracy.  J. Bus. Econom. Statist., 13:253–263.
  * Diks et al., (2011) Diks, C., Panchenko, V., and van Dijk, D. (2011).  Likelihood-based scoring rules for comparing density forecasts in tails.  J. Econom., 163(2):215–230.
  * Dimitriadis and Bayer, (2019) Dimitriadis, T. and Bayer, S. (2019).  A joint quantile and expected shortfall regression framework.  Electron. J. Statist., 13(1):1823–1871.
  * Ehm and Gneiting, (2012) Ehm, W. and Gneiting, T. (2012).  Local proper scoring rules of order two.  Ann. Statist., 40(1):609 – 637.
  * Elliott et al., (2005) Elliott, G., Timmermann, A., and Komunjer, I. (2005).  Estimation and Testing of Forecast Rationality under Flexible Loss.  Rev. Econ. Stud., 72(4):1107–1125.
  * Embrechts and Wang, (2015) Embrechts, P. and Wang, R. (2015).  Seven proofs for the subadditivity of expected shortfall.  Depend. Model., 3:126–140.
  * Fissler et al., (2021) Fissler, T., Frongillo, R., Hlavinová, J., and Rudloff, B. (2021).  Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals.  Electron. J. Statist., 15(1):1034–1084.
  * Fissler and Hoga, (2021) Fissler, T. and Hoga, Y. (2021).  Backtesting systemic risk forecasts using multi-objective elicitability.  Preprint.  https://arxiv.org/abs/2104.10673.
  * Fissler and Pesenti, (2022) Fissler, T. and Pesenti, S. M. (2022).  Sensitivity measures based on scoring functions.  Preprint.  https://doi.org/10.48550/arXiv.2203.00460.
  * Fissler and Ziegel, (2019) Fissler, T. and Ziegel, J. F. (2019).  Order-sensitivity and equivariance of scoring functions.  Electron. J. Stat., 13(1):1166–1211.
  * Fissler and Ziegel, (2021) Fissler, T. and Ziegel, J. F. (2021).  On the elicitability of range value at risk.  Stat. Risk Model., 38(1-2):25–46.
  * Girardi and Tolga Ergün, (2013) Girardi, G. and Tolga Ergün, A. (2013).  Systemic risk measurement: Multivariate GARCH estimation of CoVaR.  J. Bank. Financ., 37(8):3169–3180.
  * Gneiting, (2011) Gneiting, T. (2011).  Making and Evaluating Point Forecasts.  J. Amer. Statist. Assoc., 106:746–762.
  * Gneiting et al., (2007) Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2007).  Probabilistic forecasts, calibration and sharpness.  J. R. Stat. Soc. Ser. B. Stat. Methodol., 69:243–268.
  * Gneiting and Katzfuss, (2014) Gneiting, T. and Katzfuss, M. (2014).  Probabilistic forecasting.  Annu. Rev. Stat. Appl., 1(1):125–151.
  * Gneiting and Raftery, (2007) Gneiting, T. and Raftery, A. (2007).  Strictly Proper Scoring Rules, Prediction, and Estimation.  J. Amer. Statist. Assoc., 102:359–378.
  * Gneiting and Ranjan, (2011) Gneiting, T. and Ranjan, R. (2011).  Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules.  J. Bus. Econom. Statist., 29(3):411–422.
  * Gneiting and Ranjan, (2013) Gneiting, T. and Ranjan, R. (2013).  Combining predictive distributions.  Electron. J. Statist., pages 1747–1782.
  * Gneiting and Resin, (2021) Gneiting, T. and Resin, J. (2021).  Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination.  Preprint.  https://doi.org/10.48550/arXiv.2108.03210.
  * Good, (1952) Good, I. J. (1952).  Rational decisions.  J. R. Stat. Soc. Ser. B. Stat. Methodol., 14(1):107–114.
  * Hoga and Dimitriadis, (2021) Hoga, Y. and Dimitriadis, T. (2021).  On testing equal conditional predictive ability under measurement error.  Preprint.  https://arxiv.org/abs/2106.11104.
  * Holzmann and Eulert, (2014) Holzmann, H. and Eulert, M. (2014).  The role of the information set for forecasting – with applications to risk management.  Ann. Appl. Stat., 8:79–83.
  * Holzmann and Klar, (2017) Holzmann, H. and Klar, B. (2017).  Focusing on regions of interest in forecast evaluation.  Ann. Appl. Stat., 11(4):2404–2431.
  * Huber and Ronchetti, (2009) Huber, P. J. and Ronchetti, E. M. (2009).  Robust Statistics.  John Wiley & Sons, Inc., Hoboken, New Jersey, second edition.
  * Hyvärinen, (2005) Hyvärinen, A. (2005).  Estimation of non-normalized statistical models by score matching.  J. Mach. Learn. Res., 6(24):695–709.
  * Kiesel et al., (2016) Kiesel, R., Rühlicke, R., Stahl, G., and Zheng, J. (2016).  The wasserstein metric and robustness in risk management.  Risks, 4(32).
  * Koenker, (2005) Koenker, R. (2005).  Quantile Regression.  Cambridge University Press, Cambridge.
  * Krätschmer et al., (2014) Krätschmer, V., Schied, A., and Zähle, H. (2014).  Comparative and qualitative robustness for law-invariant risk measures.  Finance Stoch., 18:271–295.
  * Lambert, (2019) Lambert, N. (2019).  Elicitation and Evaluation of Statistical Functionals.  Preprint.  http://ai.stanford.edu/~nlambert/papers/elicitability.pdf.
  * Lambert et al., (2008) Lambert, N., Pennock, D. M., and Shoham, Y. (2008).  Eliciting properties of probability distributions.  In Proceedings of the 9th ACM Conference on Electronic Commerce, pages 129–138, Chicago, Il, USA. ACM.
  * Molchanov, (2017) Molchanov, I. (2017).  Theory of Random Sets.  Probability Theory and Stochastic Modelling. Springer-Verlag London, London, 2 edition.
  * Nau, (1985) Nau, R. F. (1985).  Should Scoring Rules Be ‘Effective’?  Manag. Sci., 31(5):527–535.
  * Newey and Powell, (1987) Newey, W. K. and Powell, J. L. (1987).  Asymmetric Least Squares Estimation and Testing.  Econometrica, 55:819–847.
  * Nolde and Ziegel, (2017) Nolde, N. and Ziegel, J. F. (2017).  Elicitability and backtesting: Perspectives for banking regulation.  Ann. Appl. Stat., 11(4):1833–1874.
  * Parry et al., (2012) Parry, M., Dawid, A. P., and Lauritzen, S. (2012).  Proper local scoring rules.  Ann. Statist., 40(1):561–592.
  * Pohle, (2020) Pohle, M.-O. (2020).  The Murphy decomposition and the calibration-resolution principle: A new perspective on forecast evaluation.  Preprint.  https://doi.org/10.48550/arXiv.2005.01835.
  * Steinwart et al., (2014) Steinwart, I., Pasin, C., Williamson, R., and Zhang, S. (2014).  Elicitation and Identification of Properties.  JMLR Workshop Conf. Proc., 35:1–45.
  * van der Vaart, (1998) van der Vaart, A. W. (1998).  Asymptotic statistics.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
  * Weber, (2006) Weber, S. (2006).  Distribution-Invariant Risk Measures, Information, and Dynamic Consistency.  Math. Finance, 16:419–441.