The Bayes Estimator of a Conditional Density: Consistency
A.G. Nogales

Dpto. de Matemáticas, Universidad de Extremadura

Avda. de Elvas, s/n, 06006–Badajoz, SPAIN.

e-mail<EMAIL_ADDRESS>
> Abstract. In a Bayesian framework we prove that the optimal estimator of a
> conditional density is consistent.

  * AMS Subject Class. (2020): 62F15, 62G07

  * Key words and phrases:  Bayesian estimation of a conditional density, posterior predictive distribution.

##  1 Introduction.

In the context of Functional Data Analysis, the problems of estimating a
density, conditional or not, or a regression curve, both from the frequentist
point of view and from the Bayesian point of view, are of particular interest.
Nogales (2022a) addresses the density estimation problem from a Bayesian point
of view and obtains the Bayes estimator for certain loss functions that are
derived naturally from the quadratic error loss function when trying to
estimate real functions of the parameter; under mild regularity conditions,
that Bayesian estimator is concretely proven to be the posterior predictive
density, and in Nogales (2022c) this optimal estimator is proven to be
consistent. Nogales (2022b) obtains, among other results, the Bayes estimator
of the conditional density, and it is the main aim of this paper to show its
consistency.

The works mentioned above include references to the posterior predictive
distribution, its usefulness in Predictive Inference and other areas of
Statistical Inference, and its computation. Among those references we want to
highlight here Geisser (1993) and Gelman et al. (2014). Ghosal et al. (2017)
includes powerful tools to achieve this end.

Some examples are included to illustrate the main result of the paper. For
ease of reading we reproduce here an appendix from Nogales (2022a) to recall
the basic concepts of Bayesian inference but, mainly, to explain the (somehow
unusual) notation and terminology used in this work.

##  2 The framework.

Let
$(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
be a Bayesian statistical experiment and
$X_{i}:(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})\rightarrow(\Omega_{i},\mathcal{A}_{i})$,
$i=1,2$, two statistics. Consider the Bayesian experiment image of
$(X_{1},X_{2})$

$(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2},\\{P_{\theta}^{(X_{1},X_{2})}\colon\theta\in(\Theta,\mathcal{T},Q)\\}).$

In the next, we shall suppose that
$P^{(X_{1},X_{2})}(\theta,A_{12}):=P_{\theta}^{(X_{1},X_{2})}(A_{12})$,
$\theta\in\Theta,\ A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}$, is a Markov
kernel. Let us write $R_{\theta}=P_{\theta}^{(X_{1},X_{2})}$ and
$p_{j}(x):=x_{j}$ for $j=1,2$,
$x:=(x_{1},x_{2})\in\Omega_{1}\times\Omega_{2}$. Hence

$\displaystyle P_{\theta}^{X_{1}}=R_{\theta}^{p_{1}},\quad
P_{\theta}^{X_{2}|X_{1}=x_{1}}=R_{\theta}^{p_{2}|p_{1}=x_{1}}\quad\hbox{and}\quad
E_{P_{\theta}}(X_{2}|X_{1}=x_{1})=E_{R_{\theta}}(p_{2}|p_{1}=x_{1}).$

Given an integer $n$, for $m=n$ (resp. $m=\mathbb{N}$), the Bayesian
experiment corresponding to a $n$-sized sample (resp. an infinite sample) of
the joint distribution of $(X_{1},X_{2})$ is

$\big{(}(\Omega_{1}\times\Omega_{2})^{m},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m},\big{\\{}R_{\theta}^{m}\colon\theta\in(\Theta,\mathcal{T},Q)\big{\\}}\big{)}\qquad\hbox{(1)}$

We write $R^{m}(\theta,A^{\prime}_{12,m})=R_{\theta}^{m}(A^{\prime}_{12,m})$
for $A^{\prime}_{12,m}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}$ and

$\Pi_{12,{m}}:=Q\otimes R^{m},$

for the joint distribution of the parameter and the sample, i.e.

$\Pi_{12,{m}}(A^{\prime}_{12,m}\times
T)=\int_{T}R_{\theta}^{m}(A^{\prime}_{12,m})dQ(\theta),\quad
A^{\prime}_{12,m}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m},T\in\mathcal{T}.$

The corresponding prior predictive distribution $\beta_{12,{m},Q}^{*}$ on
$(\Omega_{1}\times\Omega_{2})^{m}$ is

$\beta_{12,{m},Q}^{*}(A^{\prime}_{12,m})=\int_{\Theta}R_{\theta}^{m}(A^{\prime}_{12,m})dQ(\theta),\quad
A^{\prime}_{12,m}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}.$

The posterior distribution is a Markov kernel

$R_{m}^{*}:((\Omega_{1}\times\Omega_{2})^{m},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m})\mbox{$\succ$$\longrightarrow$}(\Theta,\mathcal{T})$

such that, for all
$A^{\prime}_{12,m}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}$ and
$T\in\mathcal{T}$,

$\Pi_{12,{m}}(A^{\prime}_{12,m}\times
T)=\int_{T}R_{\theta}^{m}(A^{\prime}_{12,m})dQ(\theta)=\int_{A^{\prime}_{12,m}}R_{m}^{*}(x^{\prime},T)d\beta_{12,{m},Q}^{*}(x^{\prime}).$

Let us write $R_{{m},x^{\prime}}^{*}(T):=R_{m}^{*}(x^{\prime},T)$.

The posterior predictive distribution on
$\mathcal{A}_{1}\times\mathcal{A}_{2}$ is the Markov kernel

${R_{{m}}^{*}}^{R}:((\Omega_{1}\times\Omega_{2})^{m},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m})\mbox{$\succ$$\longrightarrow$}(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})$

defined, for $x^{\prime}\in(\Omega_{1}\times\Omega_{2})^{m}$, by

${R_{{m}}^{*}}^{R}(x^{\prime},A_{12}):=\int_{\Theta}R_{\theta}(A_{12})dR_{{m},x^{\prime}}^{*}(\theta)$

It follows that, with obvious notations,

$\int_{\Omega_{1}\times\Omega_{2}}f(x)d{R_{{m},x^{\prime}}^{*}}^{\\!\\!\\!\\!R}(x)=\int_{\Theta}\int_{\Omega_{1}\times\Omega_{2}}f(x)dR_{\theta}(x)dR_{{m},x^{\prime}}^{*}(\theta)$

for any real random variable (r.r.v. for short) $f$ whose integral exists.

We can also consider the posterior predictive distribution on
$(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}$ defined as the Markov kernel

${R_{{m}}^{*}}^{R^{m}}:((\Omega_{1}\times\Omega_{2})^{m},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m})\mbox{$\succ$$\longrightarrow$}((\Omega_{1}\times\Omega_{2})^{m},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m})$

such that

${R_{{m}}^{*}}^{R^{m}}(x^{\prime},A^{\prime}_{12,m}):=\int_{\Theta}R_{\theta}^{m}(A^{\prime}_{12,m})dR_{{m},x^{\prime}}^{*}(\theta)$

We introduce some notations for
$(x^{\prime},x,\theta)\in(\Omega_{1}\times\Omega_{2})^{m}\times(\Omega_{1}\times\Omega_{2})\times\Theta$:

$\displaystyle\pi^{\prime}_{m}(x^{\prime},x,\theta):=x^{\prime},\quad\pi_{m}(x^{\prime},x,\theta):=x,\quad\pi_{j,m}(x^{\prime},x,\theta):=x_{j},\;\;j=1,2,\quad
q_{m}(x^{\prime},x,\theta):=\theta$
$\displaystyle\pi^{\prime}_{i,m}(x^{\prime},x,\theta):=x^{\prime}_{i}:=(x^{\prime}_{i1},x^{\prime}_{i2}),\quad\pi^{\prime}_{(i),m}(x^{\prime},x,\theta):=(x^{\prime}_{1},\dots,x^{\prime}_{i}),$

for $1\leq i\leq m$ (read $i\in\mathbb{N}$ if $m=\mathbb{N}$).

Let us consider the probability space

$\displaystyle((\Omega_{1}\times\Omega_{2})^{m}\times(\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T},\Pi_{m}),\qquad\hbox{(2)}$

where

$\Pi_{m}(A^{\prime}_{12,m}\times A_{12}\times
T)=\int_{T}R_{\theta}(A_{12})R_{\theta}^{m}(A^{\prime}_{12,m})dQ(\theta),$

when $A^{\prime}_{12,m}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}$,
$A_{12}\in\mathcal{A}_{1}\times\mathcal{A}_{2}$ and $T\in\mathcal{T}$.

So, for a r.r.v. $f$ on
$((\Omega_{1}\times\Omega_{2})^{m}\times(\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})^{m}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T})$,

$\int
fd\Pi_{m}=\int_{\Theta}\int_{(\Omega_{1}\times\Omega_{2})^{m}}\int_{\Omega_{1}\times\Omega_{2}}f(x^{\prime},x,\theta)dR_{\theta}(x)dR_{\theta}^{m}(x^{\prime})dQ(\theta)\qquad{\rm(3)}$

provided that the integral exists. Moreover, for a r.r.v. $h$ on
$((\Omega_{1}\times\Omega_{2})\times\Theta,(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T})$,

$\int
hd\Pi_{m}=\int_{\Theta}\int_{\Omega_{1}\times\Omega_{2}}h(x,\theta)dR_{\theta}(x)dQ(\theta)=\int_{\Omega_{1}\times\Omega_{2}}\int_{\Theta}h(x,\theta)dR^{*}_{1,x}(\theta)d\beta^{*}_{12,1,Q}(x).$

The following proposition is straightforward.

######  Proposition 1.

For $n\in\mathbb{N}$, we have that

$\displaystyle\Pi_{\mathbb{N}}^{(\pi^{\prime}_{(n),\mathbb{N}},\pi_{1,\mathbb{N}},q_{\mathbb{N}})}=\Pi_{n},\qquad\Pi_{\mathbb{N}}^{(\pi^{\prime}_{(n),\mathbb{N}},\pi_{1,\mathbb{N}})}=\Pi_{n}^{(\pi^{\prime}_{(n),n},\pi_{1,n})},$
$\displaystyle\Pi_{m}^{q_{m}}=Q,\quad\Pi_{m}^{(\pi^{\prime}_{m},q_{m})}=\Pi_{12,m},\quad\Pi_{m}^{\pi^{\prime}_{m}}=\beta_{12,m,Q}^{*},\quad\Pi_{m}^{(\pi_{m},q_{m})}=\Pi_{12,1},\quad\Pi_{m}^{\pi_{m}}=\beta_{12,1,Q}^{*}$
$\displaystyle\Pi_{m}^{\pi^{\prime}_{m}|q_{m}=\theta}=R_{\theta}^{m},\quad\Pi_{m}^{\pi_{m}|q_{m}=\theta}=R_{\theta},\quad\Pi_{m}^{q_{m}|\pi^{\prime}_{m}=x^{\prime}}=R^{*}_{m,x^{\prime}},\quad\Pi_{m}^{q_{m}|\pi_{m}=x}=R^{*}_{1,x^{\prime}}$

In particular, the probability space (2) contains all the basic ingredients of
the Bayesian experiment (1), i.e., the prior distribution, the sampling
probabilities, the posterior distributions and the prior predictive
distribution. When $m=\mathbb{N}$ it becomes the natural framework to address
the problem considered in this paper, as we can see in the next.

##  3 Consistency of the Bayes estimator of the conditional density

In the finite case ($m=n$), according to Theorem 1 of Nogales (2022b), when
$\mathcal{A}_{2}$ is separable, the conditional distribution of $p_{2}$ given
$p_{1}=x_{1}$ with respect to the posterior predictive distribution
${R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}$ is the Bayes estimator of the
conditional distribution $R^{p_{2}|p_{1}}$ for the squared total variation
loss function, i.e.,

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\left|\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}-R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})\right|^{2}d\Pi_{n}(x^{\prime},x,\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\big{|}M(x^{\prime},x_{1},A_{2})-R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})\big{|}^{2}d\Pi_{n}(x^{\prime},x,\theta)$

for any estimator $M$ of the conditional distribution $R^{p_{2}|p_{1}}$.
Recall from Nogales (2022b) that an estimator of the conditional distribution
$P_{\theta}^{X_{2}|X_{1}}$ from a $n$-sized sample of the joint distribution
of $(X_{1},X_{2})$ is a Markov kernel

$M:((\Omega_{1}\times\Omega_{2})^{n}\times\Omega_{1},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$

so that, being observed
$x^{\prime}=((x^{\prime}_{11},x^{\prime}_{21}),\dots,(x^{\prime}_{1n},x^{\prime}_{2n}))\in(\Omega_{1}\times\Omega_{2})^{n}$,
$M(x^{\prime},x_{1},\cdot)$ is a probability measure on $\mathcal{A}_{2}$ that
is considered as an estimation of the conditional distribution
$P_{\theta}^{X_{2}|X_{1}=x_{1}}$ for a given $x_{1}\in\Omega_{1}$.

We wonder if the Bayes risk of this estimator goes to 0 when $n$ grows to
$\infty$, i.e. if

$\lim_{n}\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\left|\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}(A_{2})-R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})\right|^{2}d\Pi_{n}(x^{\prime},x,\theta)=0.\qquad(4)$

When the joint distribution $R_{\theta}=P_{\theta}^{(X_{1},X_{2})}$ has a
density $f_{\theta}$ respect to the product of two $\sigma$-finite measures
$\mu_{1}$ and $\mu_{2}$ on $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$, resp., the
conditional density is

$f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f_{\theta}(x_{1},x_{2})}{f_{\theta,X_{1}}(x_{1})}$

for almost every $x_{1}$, where $f_{\theta,X_{1}}(x_{1})$ stands for the
marginal density of $X_{1}$.

According to Theorem 2 of Nogales (2022b), when $\mathcal{A}_{2}$ is
separable, the Bayes estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}}$ (from an $n$-sized sample) for the $L^{1}$-squared
loss function is the $\mu_{2}$-density

${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f^{*}_{n,x^{\prime}}(x_{1},x_{2})}{f^{*}_{n,x^{\prime},1}(x_{1})}=\frac{\int_{\Theta}f_{\theta}(x_{1},x_{2})r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)}{\int_{\Omega_{2}}\int_{\Theta}f_{\theta}(x_{1},t)r^{*}_{n,x^{\prime}}(\theta)dQ(\theta)d\mu_{2}(t)}$

of the conditional distribution
$\left[{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}}$ of $p_{2}$
given $p_{1}$ with respect to the posterior predictive distribution
${R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}$, i.e.

$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\left(\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)\right)^{2}d\Pi_{n}(x^{\prime},x,\theta)\leq$
$\displaystyle\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\left(\int_{\Omega_{2}}\left|d(x^{\prime},x_{1},t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)\right)^{2}d\Pi_{n}(x^{\prime},x,\theta),$

for any estimator $d$ of the conditional density.

Recall from Nogales (2022b) that an estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}}$ from a $n$-sized sample of the joint distribution
of $(X_{1},X_{2})$ is a map

$d:((\Omega_{1}\times\Omega_{2})^{n}\times\Omega_{1}\times\Omega_{2},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1}\times\mathcal{A}_{2})\rightarrow(\mathbb{R},\mathcal{R})$

so that, being observed
$x^{\prime}=((x^{\prime}_{11},x^{\prime}_{21}),\dots,(x^{\prime}_{1n},x^{\prime}_{2n}))\in(\Omega_{1}\times\Omega_{2})^{n}$,
$d(x^{\prime},x_{1},\cdot)$ is considered as an estimation of the conditional
density $f_{\theta}^{X_{2}|X_{1}=x_{1}}$ of $X_{2}$ given $X_{1}=x_{1}$.

In terms of densities, question (4) reads as follows:

$\lim_{n}\int_{(\Omega_{1}\times\Omega_{2})^{n+1}\times\Theta}\left(\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)\right)^{2}d\Pi_{n}(x^{\prime},x,\theta)=0.\qquad(5)$

Without loss of generality we can suppose that $\mu_{2}$ is a probability
measure.

In the next we will assume the following additional regularity conditions:

  * (i)

$(\Omega_{i},\mathcal{A}_{i})$, $i=1,2$, are standard Borel spaces,

  * (ii)

$\Theta$ is a Borel subset of a Polish space and $\mathcal{T}$ is its Borel
$\sigma$-field, and

  * (iii)

$\\{R_{\theta}\colon\theta\in\Theta\\}$ is identifiable.

Let us consider the auxiliary Bayesian experiment

$(\Omega_{2}\times(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(\Omega_{1}\times\Omega_{2})\times\Theta,\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})\times\mathcal{T},\mu_{2}\times\Pi_{\mathbb{N}}).$

For $t\in\Omega_{2}$,
$x^{\prime}\in(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}$,
$x\in\Omega_{1}\times\Omega_{2}$, $\theta\in\Theta$ and $n\in\mathbb{N}$, we
write

$\displaystyle I^{\prime}(t,x^{\prime},x,\theta)=x^{\prime},\quad
I^{\prime}_{n}(t,x^{\prime},x,\theta):=x^{\prime}_{n}:=(x^{\prime}_{n1},x^{\prime}_{n2}),\quad
I^{\prime}_{(n)}(t,x^{\prime},x,\theta):=(x^{\prime}_{1},\dots,x^{\prime}_{n}),$
$\displaystyle I(t,x^{\prime},x,\theta)=x,\quad
I_{j}(t,x^{\prime},x,\theta)=x_{j},\;j=1,2,\quad
K(t,x^{\prime},x,\theta)=t,\quad J(t,x^{\prime},x,\theta)=\theta.$

A similar result to Proposition 1 holds for these projections. The following
lemma will be useful to solve our problem.

######  Lemma 1.

Let $Y(t,x^{\prime},x,\theta):=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)$.

(i) For $n\in\mathbb{N}$ we have that

$\displaystyle\begin{split}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)&=E_{\mu_{2}\times\Pi_{\mathbb{N}}}(Y|(K,I^{\prime}_{(n)},I_{1})=(t,x^{\prime}_{(n)},x_{1}))\\\
&=\int_{\Theta}Y(t,x^{\prime},x,\theta)d(\mu_{2}\times\Pi_{\mathbb{N}})^{J|(K,I^{\prime}_{(n)},I_{1})=(t,x^{\prime}_{(n)},x_{1})}(\theta).\end{split}$

(ii)

${f^{*}_{\mathbb{N},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)=E_{\mu_{2}\times\Pi_{\mathbb{N}}}(Y|(K,I^{\prime},I_{1})=(t,x^{\prime},x_{1})).$

When
$\mathcal{A}^{\prime}_{(n)}:=(K,I^{\prime}_{(n)},I_{1})^{-1}(\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}\times\mathcal{A}_{1})$,
we have that $(\mathcal{A}^{\prime}_{(n)})_{n}$ is an increasing sequence of
sub-$\sigma$-fields of
$\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}\times\mathcal{A}_{1}$
such that
$\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}\times\mathcal{A}_{1}=\sigma(\cup_{n}\mathcal{A}^{\prime}_{(n)})$.
According to the martingale convergence theorem of Lévy, if $Y$ is
$(\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}\times\mathcal{A}_{1}\times\mathcal{T})$-measurable
and $\mu_{2}\times\Pi_{\mathbb{N}}$-integrable, then

$E_{\mu_{2}\times\Pi_{\mathbb{N}}}(Y|\mathcal{A}^{\prime}_{(n)})$

converges $(\mu_{2}\times\Pi_{\mathbb{N}})$-a.e. and in
$L^{1}(\mu_{2}\times\Pi_{\mathbb{N}})$ to
$E_{\mu_{2}\times\Pi_{\mathbb{N}}}(Y|\mathcal{A}_{2}\times(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}\times\mathcal{A}_{1})$.

Let us consider the $\mu_{2}\times\Pi_{\mathbb{N}}$-integrable function

$Y(t,x^{\prime},x,\theta):=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t).$

Hence, it follows from the aforementioned theorem of Lévy that

$\lim_{n}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)={f^{*}_{\mathbb{N},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t),\quad\mu_{2}\times\Pi_{\mathbb{N}}-\hbox{a.e.}\qquad(10)$

and

$\lim_{n}\int_{\Omega_{2}\times(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(\Omega_{1}\times\Omega_{2})\times\Theta}\left|{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-{f^{*}_{\mathbb{N},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)\right|d(\mu_{2}\times\Pi_{\mathbb{N}})(t,x^{\prime},x,\theta)=0,$

i.e.,

$\lim_{n}\int_{(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(\Omega_{1}\times\Omega_{2})\times\Theta}\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-{f^{*}_{\mathbb{N},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=0.\qquad(11)$

On the other hand, as a consequence of a known theorem of Doob (see Theorem
6.9 and Proposition 6.10 from Ghosal et al. (2017, p. 129, 130)) we have that,
for every $t\in\Omega_{2}$ and $x_{1}\in\Omega_{1}$,

$\lim_{n}\int_{\Theta}f_{\theta^{\prime}}^{X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})^{J|(K,I^{\prime}_{(n)},I_{1})=(t,x^{\prime}_{(n)},x_{1})}(\theta^{\prime})=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t),\quad
R_{\theta}^{\mathbb{N}}-\hbox{a.e.}$

for $Q$-almost every $\theta$. Hence, according to Lemma 1 (i),

$\lim_{n}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t),\quad
R_{\theta}^{\mathbb{N}}-\hbox{a.e.}$

for $Q$-almost every $\theta$. In particular,

$\lim_{n}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t),\quad(\mu_{2}\times\Pi_{\mathbb{N}})-\hbox{a.e.}$

In this sense we can say that the posterior predictive conditional density
${f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)$ is a
strongly consistent estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)$.

From this and (10) we obtain that

${f^{*}_{\mathbb{N},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t),\quad(\mu_{2}\times\Pi_{\mathbb{N}})-\hbox{a.e.}$

According to (11) we obtain that

$\lim_{n}\int_{(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(\Omega_{1}\times\Omega_{2})\times\Theta}\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=0,\qquad(12)$

which proves that the risk function of the Bayes estimator
${f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)$ of the
conditional density $f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)$ converges to 0 for the
$L^{1}$-loss function. Since

$\sup_{A_{2}\in\mathcal{A}_{2}}\left|\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}(A_{2})-R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})\right|=\frac{1}{2}\int_{\Omega_{2}}\left|{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)-f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)\right|d\mu_{2}(t)$

we have that

$\lim_{n}\int_{(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(\Omega_{1}\times\Omega_{2})\times\Theta}\sup_{A_{2}\in\mathcal{A}_{2}}\left|\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}(A_{2})-R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})\right|d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=0,\qquad(13)$

which proves that that the risk function of the Bayes estimator
$\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}$ of
the conditional distribution $R_{\theta}^{p_{2}|p_{1}=x_{1}}$ converges to 0
for the total variation loss function.

We wonder if these results remain true for the squared versions of the loss
functions. The answer is yes because bounded $L^{1}$-convergence on a
probability space implies $L^{2}$-convergence.

So, we have proved the following result.

######  Theorem 1.

Let
$(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
be a Bayesian statistical experiment and
$X_{i}:(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})\rightarrow(\Omega_{i},\mathcal{A}_{i})$,
$i=1,2$, two statistics. Consider the Bayesian experiment image of
$(X_{1},X_{2})$

$(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2},\\{R_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\}).$

where $R_{\theta}:=P_{\theta}^{(X_{1},X_{2})}$ is supposed dominated by the
product $\mu_{1}\times\mu_{2}$ of two $sigma$-finite measures $\mu_{1}$ and
$\mu_{2}$ on $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$, resppectively. Let us
suppose that
$(\Omega_{1}\times\Omega_{2},\mathcal{A}_{1}\times\mathcal{A}_{2})$ is a Borel
standar space, that $\Theta$ is a Borel subset of a Polish space and
$\mathcal{T}$ is its Borel $\sigma$-field. Suppose also that the likelihood
function
$\mathcal{L}(x_{1},x_{2};\theta):=f_{\theta}(x_{1},x_{2})=\frac{dR_{\theta}}{d(\mu_{1}\times\mu_{2})}(x_{1},x_{2})$
is $\mathcal{A}_{1}\times\mathcal{A}_{2}\times\mathcal{T}$-measurable and the
family $\\{R_{\theta}\colon\theta\in\Theta\\}$ is identifiable. Then:

  * (a)

The posterior predictive conditional density
${f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}}$ is the Bayes
estimator of the conditional density $f_{\theta}^{X_{2}|X_{1}}$ in the product
experiment
$((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n},\\{R_{\theta}^{n}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
for the squared $L^{1}$ loss function. Moreover the risk function converges to
0 both for the $L^{1}$ loss function and the squared $L^{1}$ loss function.

  * (b)

The posterior predictive conditional distribution
$\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}$ is
the Bayes estimator of the sampling conditional distribution
$P_{\theta}^{X_{2}|X_{1}}$ in the product experiment
$((\Omega_{1}\times\Omega_{2})^{n},(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n},\\{R_{\theta}^{n}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
for the squared variation total loss function. Moreover the risk function
converges to 0 both for the total variation loss function and the squared
total variation loss function.

  * (c)

The posterior predictive density is a strongly consistent estimator of the
conditional density $f_{\theta}^{X_{2}|X_{1}}$, in the sense that

$\lim_{n}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)=f_{\theta}^{X_{2}|X_{1}=x_{1}}(t),\quad(\mu_{2}\times\Pi_{\mathbb{N}})-\hbox{a.e.}$

for $Q$-almost every $\theta\in\Theta$.

##  4 Proof of the Lemma

######  Proof.

(i) According to Lemma 1 of Nogales (2022b), we have that, for all
$A^{\prime}_{12,n}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{n}$ and all
$A_{1}\in\mathcal{A}_{1}$,

$\displaystyle\int_{A^{\prime}_{12,n}\times
A_{1}\times\Omega_{2}\times\Theta}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})d\Pi_{n}(x^{\prime},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,n}\times
A_{1}}\left[{R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}=x_{1}}\\!\\!(A_{2})d{\Pi_{n}}^{(\pi^{\prime},p_{1})}(x^{\prime},x_{1}).$

From this and Proposition 1 we obtain that, given $A_{2}\in\mathcal{A}_{2}$,

$\displaystyle\int_{A_{2}\times(A^{\prime}_{12,n}\times(\Omega_{1}\times\Omega_{2})^{\mathbb{N}})\times(A_{1}\times\Omega)\times\Theta}f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})(t,x^{\prime},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,n}\times
A_{1}\times\Omega_{2}\times\Theta}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})d\Pi_{n}(x^{\prime}_{(n)},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,n}\times
A_{1}}\left[{R^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}=x_{1}}\\!\\!(A_{2})d{\Pi_{n}}^{(\pi^{\prime}_{(n),n},\pi_{1,n})}(x^{\prime}_{(n)},x_{1})=$
$\displaystyle\int_{A_{2}\times A^{\prime}_{12,n}\times
A_{1}}{f^{*}_{n,x^{\prime}_{(n)}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})^{(K,I^{\prime}_{(n)},I_{1})}(t,x^{\prime}_{(n)},x_{1}),$

which finishes the proof of the first part of (i). Finally, desintegrating

$\mu_{2}\times\Pi_{\mathbb{N}}=(\mu_{2}\times\Pi_{\mathbb{N}})^{(J|(K,I^{\prime}_{(n)},I_{1}))}\otimes(\mu_{2}\times\Pi_{\mathbb{N}})^{(K,I^{\prime}_{(n)},I_{1})}$

we obtain

$\displaystyle\int_{A_{2}\times(A^{\prime}_{12,n}\times(\Omega_{1}\times\Omega_{2})^{\mathbb{N}})\times(A_{1}\times\Omega)\times\Theta}f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})(t,x^{\prime},x,\theta)=$
$\displaystyle\int_{A_{2}\times A^{\prime}_{12,n}\times
A_{1}}\int_{\Theta}f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})^{J|(K,I^{\prime}_{(n)},I_{1})=(t,x^{\prime}_{(n)},x_{1})}(\theta)d(\mu_{2}\times\Pi_{\mathbb{N}})^{(K,I^{\prime}_{(n)},I_{1})}(t,x^{\prime}_{(n)},x_{1})$

which proves the second part of (i).

(ii) An analogous reasoning would work in this case if we can prove that, for
all
$A^{\prime}_{12,\mathbb{N}}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}$
and all $A_{1}\in\mathcal{A}_{1}$,

$\displaystyle\int_{A^{\prime}_{12,\mathbb{N}}\times
A_{1}\times\Omega_{2}\times\Theta}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times
A_{1}}\left[{R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}=x_{1}}\\!\\!(A_{2})d{\Pi_{\mathbb{N}}}^{(\pi^{\prime},p_{1})}(x^{\prime},x_{1}).\qquad\hbox{(6)}\
$

According to (3), we have that, for all
$A^{\prime}_{12,{\mathbb{N}}}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}$
and all $A_{1}\in\mathcal{A}_{1}$,

$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times
A_{1}\times\Omega_{2}\times\Theta}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=$
$\displaystyle\int_{\Theta}\int_{A^{\prime}_{12,{\mathbb{N}}}}\int_{A_{1}}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})dR_{\theta}^{p_{1}}(x_{1})dR_{\theta}^{\mathbb{N}}(x^{\prime})dQ(\theta)=$
$\displaystyle\int_{\Theta}R_{\theta}^{\mathbb{N}}(A^{\prime}_{12,{\mathbb{N}}})R_{\theta}(A_{1}\times
A_{2})dQ(\theta).\quad\hbox{(7)}$

Note that, by definition of conditional distribution,

${R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}(A_{1}\times
A_{2})=\int_{A_{1}}\left[{R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}=x_{1}}\\!\\!(A_{2})d\\!\left[{R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{1}}\\!\\!(x_{1}),\qquad\hbox{(8)}$

and, by definition of posterior predictive distribution,

${R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}(A_{1}\times
A_{2})=\int_{\Theta}R_{\theta}(A_{1}\times
A_{2})dR^{*}_{{\mathbb{N}},x^{\prime}}(\theta).$

Notice that, for any
$A^{\prime}_{12,{\mathbb{N}}}\in(\mathcal{A}_{1}\times\mathcal{A}_{2})^{\mathbb{N}}$,
being
$R^{*}_{{\mathbb{N}},x^{\prime}}=\Pi_{\mathbb{N}}^{q_{\mathbb{N}}|\pi^{\prime}_{\mathbb{N}}=x^{\prime}}$,

$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}}\int_{\Theta}R_{\theta}(A_{1}\times
A_{2})dR^{*}_{{\mathbb{N}},x^{\prime}}(\theta)d\Pi_{\mathbb{N}}^{\pi^{\prime}_{\mathbb{N}}}(x^{\prime})=\int_{A^{\prime}_{12,{\mathbb{N}}}}\int_{\Theta}R_{\theta}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}^{q_{\mathbb{N}}|\pi^{\prime}_{\mathbb{N}}=x^{\prime}}(\theta)d\Pi_{\mathbb{N}}^{\pi^{\prime}_{\mathbb{N}}}(x^{\prime})=$
$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times\Theta}R_{\theta}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}^{(\pi^{\prime}_{\mathbb{N}},q_{\mathbb{N}})}(x^{\prime},\theta)=\int_{{\pi^{\prime}_{\mathbb{N}}}^{-1}(A^{\prime}_{12,{\mathbb{N}}})}R_{\theta}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta),$

proving that

$\int_{\Theta}R_{\theta}(A_{1}\times
A_{2})dR^{*}_{{\mathbb{N}},x^{\prime}}(\theta)=E_{\Pi_{\mathbb{N}}}(r_{A_{1}\times
A_{2}}|\pi^{\prime}_{\mathbb{N}}=x^{\prime})$

where $r_{A_{1}\times A_{2}}(\theta):=R_{\theta}(A_{1}\times A_{2})$ and,
hence

${R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}(A_{1}\times
A_{2})=E_{\Pi_{\mathbb{N}}}(r_{A_{1}\times
A_{2}}|\pi^{\prime}_{\mathbb{N}}=x^{\prime}).$

So, by definition of conditional expectation,

$\int_{A^{\prime}_{12,{\mathbb{N}}}}{R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}^{\pi^{\prime}_{\mathbb{N}}}(x^{\prime})=\int_{A^{\prime}_{12,{\mathbb{N}}}\\!\times\Omega_{1}\times\Omega_{2}\times\Theta}R_{\theta}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta).\qquad\hbox{(9)}$

But the second term of this equation is

$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times\Omega_{1}\times\Omega_{2}\times\Theta}R_{\theta}(A_{1}\times
A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=$
$\displaystyle\int_{\Theta}\int_{A^{\prime}_{12,{\mathbb{N}}}}R_{\theta}(A_{1}\times
A_{2})dR_{\theta}^{\mathbb{N}}(x^{\prime})dQ(\theta)=\int_{\Theta}R_{\theta}^{\mathbb{N}}(A^{\prime}_{12,{\mathbb{N}}})R_{\theta}(A_{1}\times
A_{2})dQ(\theta)$

which coincides with (7).

From (6) and Proposition 1 we obtain that, given $A_{2}\in\mathcal{A}_{2}$,

$\displaystyle\int_{A_{2}\times(A^{\prime}_{12,{\mathbb{N}}}\times(\Omega_{1}\times\Omega_{2})^{\mathbb{N}}\times(A_{1}\times\Omega)\times\Theta}f_{\theta}^{X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})(t,x^{\prime},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times
A_{1}\times\Omega_{2}\times\Theta}R_{\theta}^{p_{2}|p_{1}=x_{1}}(A_{2})d\Pi_{\mathbb{N}}(x^{\prime},x,\theta)=$
$\displaystyle\int_{A^{\prime}_{12,{\mathbb{N}}}\times
A_{1}}\left[{R^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!R}\right]^{p_{2}|p_{1}=x_{1}}\\!\\!(A_{2})d{\Pi_{\mathbb{N}}}^{(\pi^{\prime}_{\mathbb{N}},\pi_{1,{\mathbb{N}}})}(x^{\prime},x_{1})=$
$\displaystyle\int_{A_{2}\times A^{\prime}_{12,{\mathbb{N}}}\times
A_{1}}{f^{*}_{{\mathbb{N}},x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(t)d(\mu_{2}\times\Pi_{\mathbb{N}})^{(K,I^{\prime},I_{1})}(t,x^{\prime},x_{1}),$

which finishes the proof of (ii). $\Box$

##  5 Examples.

######  Example 1.

Let us suppose that, for $\theta,\lambda,x_{1}>0$,
$P_{\theta}^{X_{1}}=G(1,\theta^{-1})$,
$P_{\theta}^{X_{2}|X_{1}=x_{1}}=G(1,(\theta x_{1})^{-1})$ and
$Q=G(1,\lambda^{-1})$, where $G(\alpha,\beta)$ denotes the gamma distribution
of parameters $\alpha,\beta>0$. Hence the joint density of $X_{1}$ and $X_{2}$
is

$f_{\theta}(x_{1},x_{2})=\theta^{2}x_{1}\exp\\{-\theta
x_{1}(1+x_{2})\\}I_{]0,\infty[^{2}}(x_{1},x_{2}).$

It is shown in Nogales (2022b), Example 1, that the Bayes estimator of the
conditional density $f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2})=\theta
x_{1}\exp\\{-\theta x_{1}x_{2}\\}I_{]0,\infty[}(x_{2})$ is, for
$x_{1},x_{2}>0$,

$\displaystyle{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=x_{1}}(x_{2}):=\frac{f^{*}_{n,x^{\prime}}(x_{1},x_{2})}{f^{*}_{n,x^{\prime},1}(x_{1})}=$
$\displaystyle(2n+2)x_{1}\frac{[\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+2}}{[\lambda+x_{1}(1+x_{2})+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2})]^{2n+3}}=$
$\displaystyle\frac{(2n+2)x_{1}a_{n}(x^{\prime},x_{1})^{2n+2}}{(x_{1}x_{2}+a_{n}(x^{\prime},x_{1}))^{2n+3}}$

where

$a_{n}(x^{\prime},x_{1})=\lambda+x_{1}+\sum_{i=1}^{n}x^{\prime}_{i1}(1+x^{\prime}_{i2}).$

Theorem 1 shows that this a strongly consistent estimator of de conditional
density $f_{\theta}^{X_{2}|X_{1}=x_{1}}$.

######  Example 2.

Let us suppose that $X_{1}$ has a Bernoulli distribution of unknown parameter
$\theta\in]0,1[$ (i.e. $P_{\theta}^{X_{1}}=Bi(1,\theta)$) and, given
$X_{1}=k_{1}\in\\{0,1\\}$, $X_{2}$ has distribution $Bi(1,1-\theta)$ when
$k_{1}=0$ and $Bi(1,\theta)$ when $k_{1}=1$, i.e.
$P_{\theta}^{X_{2}|X_{1}=k_{1}}=Bi(1,k_{1}+(1-2k_{1})(1-\theta))$. We can
think of tossing a coin with probability $\theta$ of getting heads ($=1$) and
making a second toss of this coin if it comes up heads on the first toss, or
tossing a second coin with probability $1-\theta$ of making heads if the first
toss is tails ($=0$). Consider the uniform distribution on $]0,1[$ as the
prior distribution $Q$.

So, the joint probability function of $X_{1}$ and $X_{2}$ is

$\displaystyle\begin{split}f_{\theta}(k_{1},k_{2})&=\theta^{k_{1}}(1-\theta)^{1-k_{1}}[k_{1}+(1-2k_{1})(1-\theta)]^{k_{2}}[1-k_{1}-(1-2k_{1})(1-\theta)]^{1-k_{2}}\\\
&=\begin{cases}\theta(1-\theta)\hbox{ if }k_{2}=0,\\\ (1-\theta)^{2}\hbox{ if
}k_{1}=0,k_{2}=1,\\\ \theta^{2}\hbox{ if
}k_{1}=1,k_{2}=1.\end{cases}\end{split}$

It is shown in Nogales(2022b), Example 2, that the Bayes estimator of the
conditional probability function

$f_{\theta}^{X_{2}|X_{1}=k_{1}}(k_{2})=[k_{1}+(1-2k_{1})(1-\theta)]^{k_{2}}[1-k_{1}-(1-2k_{1})(1-\theta)]^{1-k_{2}}$

is

$\displaystyle{f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!X_{2}|X_{1}=k_{1}}(k_{2}):=\frac{f^{*}_{n,k^{\prime}}(k_{1},k_{2})}{f^{*}_{n,k^{\prime},1}(k_{1})}=\begin{cases}\frac{2n+2}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+3}&\hbox{
if }k_{1}=k_{2}=0,\vspace{1ex}\\\
\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+3}&\hbox{
if }k_{1}=0,k_{2}=1,\vspace{1ex}\\\
\frac{2n+3}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+4}&\hbox{ if
}k_{1}=1,k_{2}=0,\vspace{1ex}\\\
\frac{n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+1}{2n+n_{+0}(k^{\prime})+2n_{01}(k^{\prime})+4}&\hbox{
if }k_{1}=k_{2}=1.\end{cases}$

Theorem 1 shows that this is a strongly consistent estimator of the
conditional probability function $f_{\theta}^{X_{2}|X_{1}=k_{1}}$.

######  Example 3.

Let $(X_{1},X_{2})$ have bivariate normal distribution

$N_{2}\left(\left(\begin{array}[]{c}\theta\\\
\theta\end{array}\right),\sigma^{2}\left(\begin{array}[]{cc}1&\rho\\\
\rho&1\end{array}\right)\right),$

and consider the prior distribution $Q=N(\mu,\tau^{2})$. It is shown in
Nogales (2022b), Example 3, that that the conditional distribution

$\left({R^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!R}\right)^{p_{2}|p_{1}=x_{1}}$

is

$N\big{(}(1-\rho_{1})m_{1}(x^{\prime})+\rho_{1}x_{1},\sigma_{1}^{2}(1-\rho_{1}^{2})\big{)}$

where

$\displaystyle\rho_{1}=-\frac{a_{n}(\rho,\sigma,\tau)+\frac{1-\rho}{1+\rho}}{a_{n}(\rho,\sigma,\tau)-\frac{1-\rho}{1+\rho}}\cdot\rho,\quad\sigma_{1}^{2}=\frac{a_{n}(\rho,\sigma,\tau)}{a_{n}(\rho,\sigma,\tau)-\frac{1-\rho}{1+\rho}}\cdot\sigma^{2},$
$\displaystyle
m_{1}(x^{\prime})=m_{2}(x^{\prime})=\frac{s_{1}(x^{\prime})+(1+\rho)\frac{\sigma^{2}}{\tau^{2}}\mu}{2(1-\rho_{1})(1+\rho)^{2}\sigma^{2}a_{n}(\rho,\sigma,\tau)},$
$\displaystyle
a_{n}(\rho,\sigma,\tau):=2(n+1)(1+\rho)+\frac{\sigma^{2}}{\tau^{2}},\quad
s_{1}(x^{\prime}):=\sum_{i}(x^{\prime}_{i1}+x^{\prime}_{i2}).$

and that its density
${f^{*}_{n,x^{\prime}}}^{\\!\\!\\!\\!\\!p_{2}|p_{1}=x_{1}}$ is the Bayes
estimator of the conditional density

$f_{\theta}^{X_{2}|X_{1}=x_{1}}(x_{2})=\frac{1}{\sigma\sqrt{2\pi(1-\rho^{2})}}\exp\left\\{-\frac{1}{2\sigma^{2}(1-\rho^{2})}[x_{2}-(1-\rho)\theta-\rho
x_{1}]^{2}\right\\}$

for the $L^{1}$-squared loss function. Theorem 1 proves that it is a strongly
consistent estimator of the conditional density
$f_{\theta}^{X_{2}|X_{1}=x_{1}}$.

##  6 Appendix.

Let us briefly recall some basic concepts about Markov kernels, mainly to fix
the notations. In the next, $(\Omega,\mathcal{A})$,
$(\Omega_{1},\mathcal{A}_{1})$ and so on will denote measurable spaces.

######  Definition 1.

1) (Markov kernel) A Markov kernel
$M_{1}:(\Omega,\mathcal{A})\mbox{$\succ$$\longrightarrow$}(\Omega_{1},\mathcal{A}_{1})$
is a map $M_{1}:\Omega\times\mathcal{A}_{1}\rightarrow[0,1]$ such that: (i)
$\forall\omega\in\Omega$, $M_{1}(\omega,\cdot)$ is a probability measure on
$\mathcal{A}_{1}$, (ii) $\forall A_{1}\in\mathcal{A}_{1}$,
$M_{1}(\cdot,A_{1})$ is $\mathcal{A}$-measurable.

2) (Image of a Markov kernel) The image (or probability distribution) of a
Markov kernel
$M_{1}:(\Omega,\mathcal{A},P)\mbox{$\succ$$\longrightarrow$}(\Omega_{1},\mathcal{A}_{1})$
on a probability space is the probability measure $P^{M_{1}}$ on
$\mathcal{A}_{1}$ defined by
$P^{M_{1}}(A_{1}):=\int_{\Omega}M_{1}(\omega,A_{1})\,dP(\omega)$.

3) (Composition of Markov kernels) Given two Markov kernels
$M_{1}:(\Omega_{1},\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$
and
$M_{2}:(\Omega_{2},\mathcal{A}_{2})\mbox{$\succ$$\longrightarrow$}(\Omega_{3},\mathcal{A}_{3})$,
its composition is defined as the Markov kernel
$M_{2}M_{1}:(\Omega_{1},\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{3},\mathcal{A}_{3})$
given by

$M_{2}M_{1}(\omega_{1},A_{3})=\int_{\Omega_{2}}M_{2}(\omega_{2},A_{3})M_{1}(\omega_{1},d\omega_{2}).$

######  Remarks.

1) (Markov kernels as extensions of the concept of random variable) The
concept of Markov kernel extends the concept of random variable (or measurable
map). A random variable
$T_{1}:(\Omega,\mathcal{A},P)\rightarrow(\Omega_{1},\mathcal{A}_{1})$ will be
identified with the Markov kernel
$M_{T_{1}}:(\Omega,\mathcal{A};P)\mbox{$\succ$$\longrightarrow$}(\Omega_{1},\mathcal{A}_{1})$
defined by
$M_{T_{1}}(\omega,A_{1})=\delta_{T_{1}(\omega)}(A_{1})=I_{A_{1}}(T_{1}(\omega))$,
where $\delta_{T_{1}(\omega)}$ denotes the Dirac measure -the degenerate
distribution- at the point $T_{1}(\omega)$, and $I_{A_{1}}$ is the indicator
function of the event $A_{1}$. In particular, the probability distribution
$P^{M_{T_{1}}}$ of $M_{T_{1}}$ coincides with the probability distribution
$P^{T_{1}}$ of $T_{1}$ defined as $P^{T_{1}}(A_{1}):=P(T_{1}\in A_{1})$

2) Given a Markov kernel
$M_{1}:(\Omega_{1},\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$
and a random variable
$X_{2}:(\Omega_{2},\mathcal{A}_{2})\rightarrow(\Omega_{3},\mathcal{A}_{3})$,
we have that
$M_{X_{2}}M_{1}(\omega_{1},A_{3})=M_{1}(\omega_{1},X_{2}^{-1}(A_{3}))=M_{1}(\omega_{1},\cdot)^{X_{2}}(A_{3}).$
We write $X_{2}M_{1}:=M_{X_{2}}M_{1}$.

3) Given a Markov kernel
$M_{1}:(\Omega_{1},\mathcal{A}_{1},P_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$
we write $P_{1}\otimes M_{1}$ for the only probability measure on the product
$\sigma$-field $\mathcal{A}_{1}\times\mathcal{A}_{2}$ such that

$(P_{1}\otimes M_{1})(A_{1}\times
A_{2})=\int_{A_{1}}M_{1}(\omega_{1},A_{2})dP_{1}(\omega_{1}),\quad
A_{i}\in\mathcal{A}_{i},\,i=1,2.$

4) Given two r.v.
$X_{i}:(\Omega,\mathcal{A},P)\rightarrow(\Omega_{i},\mathcal{A}_{i})$,
$i=1,2$, we write $P^{X_{2}|X_{1}}$ for the conditional distribution of
$X_{2}$ given $X_{1}$, i.e. for the Markov kernel
$P^{X_{2}|X_{1}}:(\Omega_{1},\mathcal{A}_{1})\mbox{$\succ$$\longrightarrow$}(\Omega_{2},\mathcal{A}_{2})$
such that

$P^{(X_{1},X_{2})}(A_{1}\times
A_{2})=\int_{A_{1}}P^{X_{2}|X_{1}=x_{1}}(A_{2})dP^{X_{1}}(x_{1}),\quad
A_{i}\in\mathcal{A}_{i},\,i=1,2.$

So $P^{(X_{1},X_{2})}=P^{X_{1}}\otimes P^{X_{2}|X_{1}}$. $\Box$

Let
$(\Omega,\mathcal{A},\\{P_{\theta}\colon\theta\in(\Theta,\mathcal{T},Q)\\})$
be a Bayesian statistical experiment, where $Q$ denotes the prior distribution
on the parameter space $(\Theta,\mathcal{T})$. We suppose that
$P(\theta,A):=P_{\theta}(A)$ is a Markov kernel
$P:(\Theta,\mathcal{T})\mbox{$\succ$$\longrightarrow$}(\Omega,\mathcal{A})$.
When needed we shall suppose that $P_{\theta}$ has a density (Radon-Nikodym
derivative) $p_{\theta}$ with respect to a $\sigma$-finite measure $\mu$ on
$\mathcal{A}$ and that the likelihood function
$\mathcal{L}(\omega,\theta):=p_{\theta}(\omega)$ is
$\mathcal{A}\times\mathcal{T}$-measurable (this is sufficient to prove that
$P$ is a Markov kernel).

Let $\Pi:=Q\otimes P$, i.e.

$\Pi(A\times T)=\int_{T}P_{\theta}(A)dQ(\theta),\quad
A\in\mathcal{A},T\in\mathcal{T}.$

The prior predictive distribution is $\beta_{Q}^{*}:=\Pi^{I}$ (the
distribution of $I$ with respect to $\Pi$), where $I(\omega,\theta):=\omega$.
So

$\beta_{Q}^{*}(A)=\int_{\Theta}P_{\theta}(A)dQ(\theta).$

The posterior distribution is a Markov kernel
$P^{*}:(\Omega,\mathcal{A})\mbox{$\succ$$\longrightarrow$}(\Theta,\mathcal{T})$
such that

$\Pi(A\times
T)=\int_{T}P_{\theta}(A)dQ(\theta)=\int_{A}P^{*}_{\omega}(T)d\beta_{Q}^{*}(\omega),\quad
A\in\mathcal{A},T\in\mathcal{T},$

i.e. such that $\Pi=Q\otimes P=\beta_{Q}^{*}\otimes P^{*}$. This way the
Bayesian statistical experiment can be identified with the probability space
$(\Omega\times\Theta,\mathcal{A}\times\mathcal{T},\Pi)$, as proposed, for
instance, in Florens et al. (1990).

It is well known that, for $\omega\in\Omega$, the posterior $Q$-density is
proportional to the likelihood, i.e.

$p^{*}_{\omega}(\theta):=\frac{dP^{*}_{\omega}}{dQ}(\theta)=C(\omega)p_{\theta}(\omega)$

where $C(\omega)=[\int_{\Theta}p_{\theta}(\omega)dQ(\theta)]^{-1}$.

The posterior predictive distribution on $\mathcal{A}$ given $\omega$ is

${P_{\omega}^{*}}^{P}(A)=\int_{\Theta}P_{\theta}(A)dP_{\omega}^{*}(\theta),\quad
A\in\mathcal{A}.$

This is a Markov kernel

$PP^{*}(\omega,A):={P_{\omega}^{*}}^{P}(A).$

It is readily shown that the posterior predictive density is

$\frac{d{P_{\omega}^{*}}^{P}}{d\mu}(\omega^{\prime})=\int_{\Theta}p_{\theta}(\omega^{\prime})p^{*}_{\omega}(\theta)dQ(\theta).$

We know from Nogales (2022a) that

$\int_{\Omega\times\Theta}\sup_{A\in\mathcal{A}}|{P_{\omega}^{*}}^{P}(A)-P_{\theta}(A)|^{2}d\Pi(\omega,\theta)\leq\int_{\Omega\times\Theta}\sup_{A\in\mathcal{A}}|M(\omega,A)-P_{\theta}(A)|^{2}d\Pi(\omega,\theta),$

for every Markov kernel
$M:(\Omega,\mathcal{A})\mbox{$\succ$$\longrightarrow$}(\Omega,\mathcal{A})$
provided that $\mathcal{A}$ is separable (recall that a $\sigma$-field is said
to be separable, or countably generated, if it contains a countable subfamily
which generates it). We also have that, for a real statistic $X$ with finite
mean, the posterior predictive mean

$E_{(P_{\omega}^{*})^{P}}(X)=\int_{\Theta}\int_{\Omega}X(\omega^{\prime})dP_{\theta}(\omega^{\prime})dP_{\omega}^{*}(\theta)$

is the Bayes estimator of $f(\theta):=E_{\theta}(X)$, as
$E_{(P_{\omega}^{*})^{P}}(X)=E_{P_{\omega}^{*}}(E_{\theta}(X))$.

##  7 Acknowledgements.

This paper has been supported by the Junta de Extremadura (Spain) under the
grant Gr21044.

##  8 References.

  * •

Geisser, S. (1993) Predictive Inference: An Introduction, Springer Science+
Business Media, Dordrecht.

  * •

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.
(2014) Bayesian Data Analysis, 3rd ed., CRC Press.

  * •

Ghosal, S., Vaart, A.v.d. (2017) Fundamentals of Noparametric Bayesian
Inference, Cambridge University Press, Cambridge UK.

  * •

Nogales, A.G. (2022a), On Bayesian Estimation of Densities and Sampling
Distributions: the Posterior Predictive Distribution as the Bayes Estimator,
Statistica Neerlandica 76(2), 236-250.

  * •

Nogales, A.G. (2022b), Optimal Bayesian Estimation of a Regression Curve, a
Conditional Density, and a Conditional Distribution, Mathematics 10(8), 1213.

  * •

Nogales, A.G. (2022c), On consistency of the Bayes Estimator of the Density,
Mathematics, 10(4), 636.