$\forall n \ge M$ $$\mathbb{P}\left( \left|\hat{D}_h \left(X_{1:n} || Y_{1:n}^{(\mathbf{w})} \right) - D_h \left(p || q^{(\mathbf{w})} \right) \right| > \delta \right) < \epsilon.$$ Therefore $\forall n \ge M$ $$\mathbb{P} \left( \operatorname*{\arg\!\max}_{\mathbf{w} \in E} \hat{D}_h \left( X_{1:n} || Y_{1:n}^{(\mathbf{w})} \right) = \mathbf{w}^* \right) > 1-\epsilon .$$

If we make the (generally unrealistic) assumption that the space of possible parameters, $S$, is finite, then we are able to show that, in the limit of a large number of samples, we can recover the optimum parameters with probability 1. We next explore how this can be extended to a compact, continuous space of parameters, $S$, provided we make assumptions about the structure of dependence of the distance estimator on parameters $\mathbf{w} \in E$. We show that assumptions about this dependence structure of the estimator on the parameters can be justified by considering some of the details of Algorithm 2.

\[lemma:piecewise\_const\] Consisder the estimator of the Hellinger distance as a function of parameters, $\mathbf{w}$, such that $$L(\mathbf{w}) = \hat{D}_h \left(X_{1:n} || Y_{1:n}^{(\mathbf{w})}\right).$$ Then $L(\mathbf{w})$ is piecewise constant with respect to parameters $\mathbf{w} \in E$, with finitely many discontinuities.

Suppose that, according to Algorithm 2, at generation $t$, we have generated pseudo data $\{\mathbf{x}^{i*}\}_{i=1}^N$, which we 