density on $\Theta$. Then, $${\mathbb{P}}\big(\mathrm{as} \mbox{ $\max\{h_s,\,s<t\}\rightarrow 0$},\  \tilde{\pi}_{t}\Rightarrow \pi_t\big)=1,\quad\forall t\geq 1.$$

Establishing precisely the link between $\max\{h_s,\,s<t\}$ and the distance between $\tilde{\pi}_{t}$ and $\pi_t$ is beyond the scope of the paper. Nevertheless, Proposition \[prop:continuity\] has the merit to show that we can make $\tilde{\pi}_t$ arbitrary close to $\pi_t$ by reducing the size of the $h_s$’s, as illustrated in Section \[subsec:example\].

The sequence Lg\[sub:averaging\].
---------------------------------

Averaging is a standard technique used to improve the statistical and/or predictive properties of online procedures, see for instance @polyak1992acceleration and @shalev2011online for its use within stochastic approximations (SA) and online learning algorithms, respectively.

Following this idea, and as announced in the introductory section, we introduce the sequence $(\bar{\pi}_t)_{t\geq 0}$ defined by $$\begin{aligned}
\label{eq:pi_bar}
\bar{\pi}_t({\mathrm{d}}\theta)=\frac{1}{t+1}\sum_{s=0}^t\tilde{\pi}_s({\mathrm{d}}\theta),\quad t\geq 0.\end{aligned}$$ Recalling that $\tilde{\theta}_t={\mathbb{E}}_{\tilde{\pi}_t}[\theta]$ and that $\bar{\theta}_t={\mathbb{E}}_{\bar{\pi}_t}[\theta]$, it is worth noting that, for all $t\geq 0$, we have $\bar{\theta}_t=(t+1)^{-1}\sum_{s=0}^t\tilde{\theta}_s$.

Using Proposition \[prop:conv\_proba\] and Cesàro’s lemma, it can be easily checked that convergence in probability of $\bar{\pi}_t$ to $\delta_{\{\theta_\star\}}$ is implied by that of $\tilde{\pi}_t$. This is stated in the following proposition for future reference.

\[prop:bar\] Assume that $\tilde{\pi}_t\Rightarrow\delta_{\{\theta_\star\}}$ in ${\mathbb{P}}$-probability. Then, $\bar{\pi}_t\Rightarrow\delta_{\{\theta_\star\}}$ in ${\mathbb{P}}$-probability.

The main advantage of $\bar{\pi}_t$ over $\tilde{\pi}_t$ is that, empirically, the estimator $\bar{\theta}_t$ typically converges 