the true values as more samples are used.

Let $X_n$ and $X$ be random variables in $\mathbb{R}^d$. If $$\lim_{n\rightarrow \infty} \operatorname{\mathbb{E}}\left[ ||X_n - X ||^2\right] = 0,$$ then, for any $\epsilon >0$, $$\lim_{n \rightarrow \infty} \mathbb{P} \left( ||X_n - X|| > \epsilon \right) = 0.$$ That is, $L_2$ convergence implies convergence in probability.

By Chebyshev’s inequality, for any $\epsilon>0$ $$\mathbb{P} \left( ||X_n - X||>\epsilon \right) \le \frac{\operatorname{\mathbb{E}}\left[ ||X_n -X||^2 \right]}{\epsilon^2}.$$

Assume $E$ is finite, $|E| = \chi$. Assume the conditions of Lemma 1 hold for distributions $p$ and $q^{(\mathbf{w})}$. Assume that a unique $\mathbf{w}^*$ maximises $D_h\left(p || q^{(\mathbf{w})} \right)$ and arrange parameter values $\mathbf{w}^j$ for $j \in \{1, \dots , \chi\}$ in order such that they are descending in $D_h\left(p || q^{(\mathbf{w})} \right)$. That is $\mathbf{w}^1 = \mathbf{w}^*$, using $\mathbf{w}^2$ gives the next biggest value and so on. Then $$\lim_{n \rightarrow \infty} \mathbb{P} \left( \operatorname*{\arg\!\max}_{\mathbf{w} \in S} \hat{D}_h \left( X_{1:n} || Y_{1:n}^{(\mathbf{w})} \right) = \mathbf{w}^* \right) = 1.$$

Let $\epsilon >0$. Take $\delta < D_h\left(p || q^{(\mathbf{w}^*)} \right) - D_h\left(p || q^{(\mathbf{w}^2)} \right)$. Using Lemma 1, we have $L_2$ convergence for the estimator $\hat{D}_h$ and, via Lemma 2, this implies convergence in probability. Therefore $\exists M \in \mathbb{N}$ such that 