#  Local Limit Theorems and Strong Approximations for Robbins-Monro Procedures

V. Konakov 111National Research University Higher School of Economics, Russian
Federation<EMAIL_ADDRESS>E. Mammen 222Institute for Applied Mathematics,
Heidelberg University, Germany<EMAIL_ADDRESS>
###### Abstract

The Robbins-Monro algorithm is a recursive, simulation-based stochastic
procedure to approximate the zeros of a function that can be written as an
expectation. It is known that under some technical assumptions, Gaussian limit
distributions approximate the stochastic performance of the algorithm. Here,
we are interested in strong approximations for Robbins-Monro procedures. The
main tool for getting them are local limit theorems, that is, studying the
convergence of the density of the algorithm. The analysis relies on a version
of parametrix techniques for Markov chains converging to diffusions. The main
difficulty that arises here is the fact that the drift is unbounded.

##  1 Introduction

This paper is devoted to strong approximations for Robbins-Monro procedures.
The approximations are based on the study of a local limit theorem for
Robbins-Monro procedures. These algorithms have first been introduced in [26]
to approximate the solution of an equation $h(\theta)=0$, where randomly
disturbed values of $h(\theta)$ are observed at updated points $\theta$. Since
then, extensive literature have been published on the subject, but to the best
of our knowledge, a local limit theorem has never been obtained. We refer to
the monograph [1] and [24] for a general mathematical discussion of these
algorithms and a review of the literature. An important class of Robbins-Monro
procedures are optimisation methods based on stochastic gradient decent. There
is an increasing literature of their applications in the implementation of
artificial neural networks and in reinforcement learning. We refer to [5, 12,
14, 21, 22] and the references therein for some recent developments and
overviews. One possible application of the results obtained in this paper are
proofs of local invariance principles, that is, of the convergence in total
variation for a wide class of stochastic functionals of the Robbins-Monro
procedure to the functionals of a limiting diffusion process. This application
will be discussed in another publication. The proof of such a result would be
based on the results of this paper and the stratification method developed in
the papers of Y. Davydov, see [7, 10, 8, 9].

We fix a probability space $(\Omega,\mathcal{F},{\mathbb{P}})$ on which all
random variables we consider below are defined. Let $(\gamma_{k})_{k\geq 0}$
be a decreasing time step that will be specified later, and let
$(\eta_{k})_{k\geq 0}$ be a collection of independent and identically
distributed random variables. We define the following recursive procedure:

$\theta_{n+1}=\theta_{n}-\gamma_{n+1}H(\theta_{n},\eta_{n+1}),\
\theta_{0}\in{\mathbb{R}}^{d},$ (1)

where $H$ is a function from ${\mathbb{R}}^{d}\times\mathcal{X}$ to
${\mathbb{R}}^{d}$ with $\mathcal{X}$ equal to the support of $\eta_{i}$.
Without loss of generality we can assume that $\mathcal{X}$ is a subset of
${\mathbb{R}}$. Generally, the Robbins-Monro procedure is used to approximate
the zeros of the function: $h(\theta)={\mathbb{E}}[H(\theta,\eta)]$, where
$\eta$ has the same distribution as $\eta_{k}$.

Even though the general theory extends to the case of multiple zeros, in this
paper, we assume that $h$ has only one zero, $\theta^{*}$ (i.e. $h(\theta)=0$
iff $\theta=\theta^{*}$ ). We assume that the sequence $(\gamma_{k})_{k\geq
1}$ is chosen as

$\gamma_{k}=\frac{A}{k^{\beta}+B}$ (2)

with constants $A>0$, $B\geq 0$ and $1/2<\beta\leq 1$. For this choice we get
that

$\sum_{k\geq 1}\gamma_{k}=+\infty,\ \ \sum_{k\geq 1}\gamma_{k}^{2}<+\infty,$
(3)

which is usually assumed for the step sequence $(\gamma_{k})_{k\geq 1}$. Our
theory can be generalized to other monotonically decreasing choices of the
step sequence $\gamma_{k}$ as long as we have that (3) holds, and that

$\frac{\sqrt{\gamma_{k}}-\sqrt{\gamma_{k+1}}}{(\gamma_{k})^{3/2}}\to\bar{\alpha}$
(4)

for some constant $\bar{\alpha}$. Note that for the choice (2) we have that
$\bar{\alpha}=0$, if $1/2<\beta<1$ and $\bar{\alpha}=(2A(B+1))^{-1}$, if
$\beta=1$.

Under appropriate assumptions, it can be shown that the convergence:

$\theta_{n}\underset{n\rightarrow+\infty}{\longrightarrow}\theta^{*},$ (5)

holds almost surely, see [1]. Furthermore, Gaussian limit theorems have been
stated. For a formulation of such a result we remark first that after a
renormalisation the procedure (1) stabilizes around the solution of the
following Ordinary Differential Equation (ODE):

$\frac{d}{dt}\bar{\theta}_{t}=-h(\bar{\theta}_{t}),\
\bar{\theta}_{0}=\theta_{0},\mbox{ where
}h(\theta)={\mathbb{E}}[H(\theta,\eta)].$ (6)

Thus, fluctuations of the algorithm should be considered with respect to the
solution $(\bar{\theta}_{t})_{t\geq 0}$ of the ODE (6). For the defintion of
the renormalisation, we consider a shift in the indexation of the procedure
that will allow us to consider $(\theta_{n})_{n\geq 0}$ in a region that is
close to stationarity. Let $N\in{\mathbb{N}}$, and consider a sequence
$\left(\theta_{n}^{N}\right)_{n\geq 0}=\left(\theta_{N+n}\right)_{n\geq 0}$,
of shifted Robbins-Monro algorithms. These algorithms satisfy the following
recurrence equation:

$\theta_{n+1}^{N}=\theta_{n}^{N}-\gamma_{n+1}^{N}H(\theta_{n}^{N},\eta_{n+1}^{N})\\\
$ (7)

with starting value $\theta_{0}^{N}\in{\mathbb{R}}^{d}$, where
$\eta_{n+1}^{N}=\eta_{N+n+1}$, and $\gamma_{n+1}^{N}=\gamma_{N+n+1}$. Set now:

$\displaystyle t_{0}^{N}=0,\ t_{1}^{N}=\gamma_{1}^{N},\
t_{2}^{N}=\gamma_{1}^{N}+\gamma_{2}^{N},\ \dots,\
t_{k}^{N}=\gamma_{1}^{N}+\cdots+\gamma_{k}^{N}$

and define for an arbitrary fixed terminal time $T>0$:

$M(N)=\inf\\{k\in{\mathbb{N}}\ ;\ t_{k}^{N}\geq T\\}.$

Closeness of $\theta_{n}^{N}$ and $\bar{\theta}_{t}$ on the grid $t_{0}^{N},\
\dots,\ t_{M(N)}^{N}$ becomes intuitively clear because $\bar{\theta}_{t}$ is
for $t=t_{n+1}^{N}$ close to its Euler approximation

$\bar{\theta}^{N}_{n+1}=\bar{\theta}^{N}_{n}-\gamma_{n+1}^{N}h(\bar{\theta}^{N}_{n})\text{
with }\bar{\theta}^{N}_{0}=\theta_{0}.$

Note that the Robbins -Monro procedure can be rewritten as perturbed Euler
scheme

$\theta^{N}_{n+1}=\theta^{N}_{n}-\gamma_{n+1}^{N}h(\theta^{N}_{n})+\varepsilon_{n}^{N},$

where

$\varepsilon_{n}^{N}=\theta^{N}_{n+1}-\theta^{N}_{n}+\gamma_{n+1}^{N}h(\theta^{N}_{n})=-\gamma_{n+1}^{N}(H(\theta^{N}_{n},\eta^{N}_{n+1})-h(\theta^{N}_{n})).$

The centered innovations $\varepsilon_{n}^{N}$ may be considered as "small
fluctuations".

On the interval $[0,T]$ we consider the re-normalized process $U_{t}^{N}$ that
is equal to :

$U_{t}^{N}=\frac{\theta_{k}^{N}-\bar{\theta}_{t_{k}^{N}}}{\sqrt{\gamma_{k}^{N}}}$
(8)

as long as $t\in[t_{k}^{N},t_{k+1}^{N})$. Under our assumptions, stated in
Subsection 2.1, it can be shown that the convergence (5) holds and that the
process $U_{t}^{N}$ converges weakly to the solution $(X_{t})_{0\leq t\leq T}$
of the $d$-dimensional SDE:

$\mathrm{d}X^{i}_{t}=\bar{\alpha}X^{i}_{t}\mathrm{d}t-\sum_{j=1}^{d}\frac{\partial
h_{i}}{\partial x_{j}}(\bar{\theta}_{t})\cdot
X^{j}_{t}\mathrm{d}t+\sum_{j=1}^{d}R_{ij}^{1/2}(\bar{\theta}_{t})\mathrm{d}W^{j}_{t},\
i=1,...,d,$ (9)

or in matrix notation

$\mathrm{d}X_{t}=\left(\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{t})\right)X_{t}\mathrm{d}t+R^{1/2}(\bar{\theta}_{t})\mathrm{d}W_{t},$
(10)

where $W_{t}$ is the $d$-dimensional Brownian motion, where for
$\theta\in{\mathbb{R}}^{d}$ the matrix $R(\theta)$ is the covariance of
$H(\theta,\eta)$, and where we write $\mathcal{D}h(x)=({{\mathrm{grad}\
}}h_{1},...,{\mathrm{grad}\ }h_{d})^{\intercal}(x)=\left(\frac{\partial
h_{i}}{\partial x_{j}}(x)\right)_{1\leq i,j\leq d}$ for the $d\times d$ valued
derivative of $h$ at the point $x\in{\mathbb{R}}^{d}$. This convergence can be
shown by application of results discussed in [1] or in [18]. For a motivation
of drift and diffusion factor we remark that (4) holds and $\gamma_{k}^{N}\to
0$ and $\gamma_{k+1}^{N}/\gamma_{k}^{N}\to 1$ for $N\to\infty$.

Under additional assumptions, see Theorem 13 in [1], one can show that
$U_{t}^{N}$ has a stationary limit

$\mathrm{d}X^{*}_{t}=\left(\bar{\alpha}I-\mathcal{D}h(\theta^{*})\right)X_{t}\mathrm{d}t+R^{1/2}(\theta^{*})\mathrm{d}W_{t}.$
(11)

In this article, we are interested in a local limit theorem for quantifying
the convergence of the densities of a truncated modification of
$(U_{t}^{N})_{t\geq 0}$ towards the densities of $(X_{t})_{t\geq 0}$. This
will be done for the general case where a nonstationary limit (10) applies. We
will come back to a discussion of the stationary limit $X^{*}_{t}$ in
Subsection 2.7. Our main results are stated in the next section.

Throughout the paper $C$ denotes a positive constant that is chosen large
enough. At each appearance $C$ may denote another constant. This may be the
case even in the same formula.

##  2 Main Results

###  2.1 Assumptions and outline of the paper

Throughout the paper we will make the following assumptions.

  * (A1)

The innovations $\eta_{1},\eta_{2},...$ are i.i.d. with some distribution
$\mu$ and the step $\gamma_{k}$ follows (2).

  * (A2)

For any compact subset $Q$ of ${\mathbb{R}}^{d}$ there exist constants $C_{Q}$
and $q_{Q}$, possibly depending on $Q$ such that for all $\theta\in Q$

$|H(\theta,x)|\leq C_{Q}(1+|x|^{q_{Q}}).$

  * (A3)

The function $h(\theta)=\int H(\theta,x)\mu(\mathrm{d}x)$ has two bounded
derivatives and the function $H(\theta,x)$ is Lipschitz w.r.t. its first
argument with a constant not depending on $x$ in a tubular neighborhood
$\\{\theta\in{\mathbb{R}}^{d}:\|\theta-\bar{\theta}_{t}\|\leq\delta$ for
$t\in[0,T]$ with $\delta>0$ small enough.

For the next assumption we need some notation that will be used again when we
define the truncated processes. We define $a_{N}=\ln(1/\gamma_{1}^{N})$

$\displaystyle\chi_{N}(x)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{cc}x&\text{for }\|x\|\leq a_{N},\\\
a_{N}\frac{x}{\|x\|}\phi_{N}(\|x\|)&\text{for }\|x\|>a_{N}.\end{array}\right.$
$\displaystyle\phi_{N}(\|x\|)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{cl}k_{N}\int_{a_{N}}^{2a_{N}+1-\|x\|}\exp\left(-\frac{1}{(t-a_{N})(a_{N}+1-t)}\right)\mathrm{d}t&\text{for
}a_{N}<\|x\|\leq a_{N}+1,\\\ 1&\text{for }\|x\|\leq a_{N},\\\ 0&\text{for
}\|x\|>a_{N}+1,\end{array}\right.$ $\displaystyle\alpha_{t}^{N}$
$\displaystyle=$ $\displaystyle\alpha_{t_{k}^{N}}^{N}\text{ for }t_{k}^{N}\leq
t<t_{k+1}^{N},$

where the value $k_{N}$ depends on $N$ and it is equal to
$(\int_{a_{N}}^{a_{N}+1}\exp(-(t-a_{N})^{-1}(a_{N}+1-t)^{-1})\mathrm{d}t)^{-1}$.
Thus $\phi_{N}(u)$ is continuous at $u=a_{N}$. We will also make use of the
fact that $\phi_{N}$ is infinitely often differentibal.

  * (A4)

For $\xi(\theta,v)=H(\theta,v)-h(\theta)$ with
$\theta\in{\mathbb{R}}^{d},v\in{\mathbb{R}}$, the centered random variables
$\xi(\bar{\theta}_{t}+\sqrt{\gamma^{N}_{k}}$ $\chi_{N}(x),\eta_{k+1}^{N})$
have densities $f_{t_{k}^{N},x}(z)$ that are five times continuously
differentiable with derivatives that are at least of polynomial decay of order
$M>2d+6$, i.e. for $x\in\mathbb{R}^{d}$, for
$t\in\Gamma_{N}=\\{0,t_{1}^{N},...,t_{M(N)}^{N}\\}$ and for multi-indices
$\nu$ , $|\nu|\leq 5$ it holds with a constant $C>0$ that

$|D_{z}^{\nu}f_{t,x}(z)|\leq CQ_{M}(z),$

where, for $r>d$ the function $Q_{r}:\mathbb{R}^{d}\to\mathbb{R}$ is defined
by

$Q_{r}(z)=c_{r}(1+\|z\|)^{-r},$

with $c_{r}$ chosen such that $\int_{\mathbb{R}^{d}}Q_{r}(z)\mathrm{d}z=1$.
Furthermore, it holds for $x,y,z\in\mathbb{R}^{d}$,
$t=t_{i}^{N}\in\Gamma_{N}$, $1\leq i,j\leq d$ that

$\displaystyle\int_{\mathbb{R}^{d}}f_{t,x}(z)z_{i}\mathrm{d}z=0,$
$\displaystyle\int_{\mathbb{R}^{d}}f_{t,x}(z)z_{i}z_{j}\mathrm{d}z=R_{ij}(\bar{\theta}_{t}+\chi_{N}(x)\sqrt{\gamma_{i}^{N}}),$
$\displaystyle|f_{t,x}(z)-f_{t,y}(z)|\leq
C\sqrt{\gamma_{i+1}^{N}}\|x-y\|Q_{M}(z)$

for some constant $C>0$.

  * (A5)

The covariance matrix $R(\theta)$ of $H(\theta,\eta)$ exists and has a
smallest eigenvalue bounded away from 0, uniformly for $\theta$. The elements
$R_{ij}$ of the covariance matrix are absolutely bounded and are Lipschitz
continuous with a uniformly valid Lipschitz constant in a tubular neighborhood
of $\bar{\theta}_{t}$ for all $t\in\Gamma_{N}$ for $N$ large enough.

In Subsection 2.2 we introduce a truncated version $V_{t}^{N}$ of the Robbins-
Monro procedure and show that it approximates the untruncated version
$U_{t}^{N}$. More precisely, we will show that the supremum of the absolute
difference of the two processes is of order
$O_{P}(\sqrt{\gamma_{1}^{N}})=O_{P}(N^{-\beta/2})$. In the following
subsection 2.3 we will state our main result. We will show that the truncated
Robbins-Monro procedure can be approximated by the diffusion $X_{t}$ in the
following sense. We will give bounds on the total variation distance and
Hellinger distance between the transition densities of the two processes.
Furthermore, we will consider the joint distributions on an increasing grid of
time points for the two processes. We will state bounds on the total variation
distance and Hellinger distance for the distribution of the processes on the
grid. This result allows a strong approximation of the truncated Robbins-Monro
procedure can be approximated by the diffusion $X_{t}$. We will give bounds on
the total variation distance and Hellinger distance between the values of the
truncated Robbins-Monro procedure and the diffusion on an increasing grid of
time points. For the proof of these results we define a truncated modification
of the diffusion in Subsection 2.4. For the comparison of the truncated
Robbins-Monro process to the truncated diffusion we will make use of the
parametrix method. How this approach can be adapted to our setting will be
explained in Subsection 2.5. Subsection 2.6 states our result on the
comparison of the truncated processes and outlines its proof. In Subsection
2.7 we will discuss convergence of the Robbins-Monro process under an
additional Lyapunov condition which allows approximation by a stationary
diffusion, see (11). All proofs of our results will be given in Section 3 and
in the supplement.

###  2.2 Approximation of Robbins-Monro algorithm by a truncated modification

In this section we introduce a truncated modification of $(U_{t}^{N})_{t\geq
0}$ for which we will show uniform convergence to the untruncated version. For
a discussion of practical and theoretical aspects of truncated Robbins-Monro
procedures we refer to [18]. For a motivation how we truncate $U^{N}_{t_{k}}$
we rewrite the process $U^{N}_{t_{k}}$ as specified in the following lemma.

######  Lemma 2.1.

With

$\displaystyle\beta_{k+1}^{N}$ $\displaystyle=$
$\displaystyle\sqrt{\gamma^{N}_{k+1}}\left(-h(\bar{\theta}_{t^{N}_{k}})-\frac{\bar{\theta}_{t^{N}_{k+1}}-\bar{\theta}_{t^{N}_{k}}}{\gamma^{N}_{k+1}}\right),$
$\displaystyle\alpha_{t_{k}^{N}}^{N}$ $\displaystyle=$
$\displaystyle\frac{\sqrt{\gamma_{k}^{N}}-\sqrt{\gamma^{N}_{k+1}}}{(\gamma^{N}_{k+1})^{3/2}}$

the Markov chain $(U_{t_{k}^{N}}^{N})$ has the following representation:

$\displaystyle U^{N}_{t^{N}_{k+1}}$ $\displaystyle=$ $\displaystyle
U_{t^{N}_{k}}^{N}+G_{N}(t_{k}^{N},U_{t^{N}_{k}}^{N})\gamma^{N}_{k+1}U_{t^{N}_{k}}^{N}-\sqrt{\gamma^{N}_{k+1}}\xi\left(\bar{\theta}_{t_{k}^{N}}+\sqrt{\gamma^{N}_{k}}U_{t^{N}_{k}}^{N},\eta_{k+1}^{N}\right)+\beta_{k+1}^{N},$
(14)

where

$\displaystyle G_{N}(t_{k}^{N},x)$ $\displaystyle=$
$\displaystyle\alpha_{t^{N}_{k}}^{N}I-\sqrt{\frac{\gamma^{N}_{k}}{\gamma^{N}_{k+1}}}\int_{0}^{1}\mathcal{D}h\left(\bar{\theta}_{t^{N}_{k}}+\delta
x\sqrt{\gamma^{N}_{k}}\right)\mathrm{d}\delta.$

The proof of the lemma will be given in Subsection A.1 of the supplementary
material. The representation (14) motivates the following truncated process
$V^{N}_{t_{k}}$:

$\displaystyle V^{N}_{t^{N}_{k+1}}$ $\displaystyle=$ $\displaystyle
V_{t^{N}_{k}}^{N}+F_{N}(t_{k}^{N},V_{t^{N}_{k}}^{N})\gamma^{N}_{k+1}\chi_{N}(V_{t^{N}_{k}}^{N})-\sqrt{\gamma^{N}_{k+1}}\xi\left(\bar{\theta}_{t_{k}^{N}}+\sqrt{\gamma^{N}_{k}}\chi_{N}(V_{t^{N}_{k}}^{N}),\eta_{k+1}^{N}\right),$

where

$\displaystyle F_{N}(t,x)$ $\displaystyle=$
$\displaystyle(\alpha_{t}^{N}\mathbb{I}_{\|x\|\leq
a_{N}}+\bar{\alpha}\mathbb{I}_{\|x\|>a_{N}})I-\sqrt{\frac{\gamma^{N}_{k+\mathbb{I}_{\|x\|\geq
a_{N}}}}{\gamma^{N}_{k+1}}}\int_{0}^{1}\mathcal{D}h\left(\bar{\theta}_{t}+\delta\chi_{N}(x)\sqrt{\gamma^{N}_{k}}\right)\mathrm{d}\delta,$

and where $I$ is the $d\times d$ identity matrix. The term $a_{N}$ and the
functions $\chi_{N}$ and $\phi_{N}$ were defined after the statement of
Assumption (A3). Note that in the definition of the truncated process the term
$\beta_{k+1}^{N}$ is omitted. This is done because this term is asymptotically
negligible, as stated in the following lemma.

######  Lemma 2.2.

It holds that:

$\beta_{k+1}^{N}\underset{N\rightarrow+\infty}{\longrightarrow}0.$

For a proof of the lemma see Subsection A.2 of the supplementary material.
Before we will come in the next subsection to the statement of our main result
we will compare the truncated and the untruncated Robbins-Monro algorithm in
the following proposition.

######  Proposition 2.3.

For $C>0$ large enough it holds that:

$\mathbb{P}\left(\sup_{1\leq k\leq
M(N)}\left\|U_{t_{k}^{N}}^{N}-V_{t_{k}^{N}}^{N}\right\|>C\sqrt{\gamma_{1}^{N}}\right)\underset{N\rightarrow+\infty}{\longrightarrow}0.$

For a proof of the proposition see Subsection 3.2.

###  2.3 Main result

The following theorem compares the truncated Robbins-Monro procedure and the
diffusion process. It states a bound for the difference in total variation
norm and Hellinger norm for the transition densities $p_{N}$ of the truncated
Robbins-Monro procedure $V^{N}_{t}$ and the transition density $q$ of the
diffusion process $X_{t}$. For $x,z\in{\mathbb{R}}^{d}$ and $0\leq s<t$ we
denote the conditional density of $X_{t}$ at $z$ given $X_{s}=x$ by
$q(s,t,x,z)$ and for $x,z\in{\mathbb{R}}^{d}$ and $s,t\in\\{t_{0}^{N},\
\dots,\ t_{M(N)}^{N}\\}$ with $s<t$ we write $p_{N}(s,t,x,z)$ for the
conditional density of $V_{t}^{N}$ at $z$ given $V_{s}^{N}=x$.

######  Theorem 2.1.

For $s,t\in\\{t_{0}^{N},\ \dots,\ t_{M(N)}^{N}\\}$ with $s<t$ and
$x\in{\mathbb{R}}^{d}$ with $|x|\leq a_{N}/2$ it holds that

$\displaystyle\int_{{\mathbb{R}}^{d}}(\sqrt{p_{N}}-\sqrt{q})^{2}(s,t,x,z)\mathrm{d}z\leq\int_{{\mathbb{R}}^{d}}|p_{N}-q|(s,t,x,z)\mathrm{d}z$
$\displaystyle\qquad\leq
C(t-s)^{1/2}I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{\beta^{-1}-1}+C\ln(1/\gamma_{1}^{N})^{2}(\gamma_{1}^{N})^{1/2}$
$\displaystyle\qquad\leq
C(t-s)^{1/2}I_{\\{\frac{1}{2}<\beta<1\\}}N^{-(1-\beta)}+C\ln(N)^{2}N^{-\beta/2}.$

The theorem follows directly by application of Proposition 2.5 in Subsection
2.4 and Theorem 2.2 in Subsection 2.6.

The propositions can also be used for getting a result on the distributions of
the truncated Robbins-Monro procedure and the diffusion process on an
increasing grid of time points
$\tau^{N}_{0}=0<\tau^{N}_{1}<...<\tau^{N}_{m_{N}}$ with
$\tau^{N}_{1},...,\tau^{N}_{m_{N}}\in\\{t_{0}^{N},\ \dots,\ t_{M(N)-1}^{N}\\}$
and $\tau^{N}_{m_{N}}=t_{M(N)-1}^{N}$. For $m_{N}\geq 1$,
$z_{1},...,z_{m_{N}},x\in{\mathbb{R}}^{d}$ put $z=(z_{1},...,z_{m_{N}})$ and
denote the conditional distribution of $(X_{\tau^{N}_{j}}:1\leq j\leq m_{N})$
given $X_{0}=x$ and of $(V^{N}_{\tau^{N}_{j}}:1\leq j\leq m_{N})$ given
$V^{N}_{0}=x$ by $Q_{x}^{m_{N}}$ or $P_{N,x}^{m_{N}}$, respectively. We get
the following corollary of Proposition 2.6 and Proposition 2.11 for the
L1-distance between these measures.

######  Corollary 2.4.

Suppose that

$\displaystyle m_{N}\to\infty,\quad
C^{-1}m_{N}^{-1}\leq{|\tau^{N}_{j}-\tau^{N}_{j-1}|}\leq Cm_{N}^{-1}\text{ for
}1\leq j\leq m_{N},\quad N^{-\beta}m_{N}\ln(N)^{4}\to 0.$ (15)

With a measure $\nu$ that dominates $Q_{x}^{m_{N}}$ and $P_{N,x}^{m_{N}}$ it
holds for $x\in{\mathbb{R}}^{d}$ with $|x|\leq a_{N}/2$ that

$\displaystyle\int\left|\frac{\mathrm{d}Q_{x}^{m_{N}}}{\mathrm{d}\nu}-\frac{\mathrm{d}P_{N,x}^{m_{N}}}{\mathrm{d}\nu}\right|\mathrm{d}\nu\leq
C\delta_{N},$

where

$\displaystyle\delta_{N}=m_{N}^{1/4}\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{(\beta^{-1}-1)/2}+\sqrt{\ln(1/\gamma_{1}^{N})}(\gamma_{1}^{N})^{1/4}\right)$
$\displaystyle\qquad\qquad+m_{N}\ln(1/\gamma_{1}^{N})^{2}(\gamma_{1}^{N})^{1/2}$
$\displaystyle\qquad=O\bigg{(}m_{N}^{1/4}\left(I_{\\{\frac{1}{2}<\beta<1\\}}N^{-(1-\beta)/2}+\sqrt{\ln(N)}N^{-\beta/4}\right)$
$\displaystyle\qquad\qquad+m_{N}\ln(N)^{2}N^{-\beta/2}\bigg{)}.$

In particular, we have that the upper bound $\delta_{N}$ in the corollary
converges to 0 if $m_{N}$ is of the form $m_{N}=CN^{\mu}$ with $\mu<\beta/2$
for $\frac{1}{2}<\beta\leq\frac{4}{5}$, $\mu<2(1-\beta)$ for
$\frac{4}{5}\leq\beta<1$ and $\mu<1/2$ for $\beta=1$. Note also that the
corollary implies that on a large enough probability space we can construct
versions of $V^{N}_{t}$ and $X_{t}$ such that

$\mathbb{P}\left(V^{N}_{\tau^{N}_{j}}=X_{\tau^{N}_{j}}:0\leq j\leq
m_{N}\right)\geq 1-C\delta_{N}.$

###  2.4 Comparison of truncated and untruncated diffusion

In this section we will compare $X_{t}$ with a truncated modification
$X_{t}^{N}$ defined by

$\mathrm{d}X_{t}^{N}=F_{N}(t,X_{t}^{N})\chi_{N}(X_{t}^{N})\mathrm{d}t+R^{1/2}(\bar{\theta}_{t})\mathrm{d}B_{t}.$

The conditional density of $X^{N}_{t}$ at $z$ given $X^{N}_{0}=x$ is denoted
by $q_{N}(0,t,x,z)$.

The following proposition states a bound for the difference of the two
densities in total variation norm.

######  Proposition 2.5.

For $0\leq t\leq T$ and $x\in{\mathbb{R}}^{d}$ with $|x|\leq a_{N}/2$ it holds
that

$\displaystyle\int_{{\mathbb{R}}^{d}}(\sqrt{q_{N}}-\sqrt{q})^{2}(s,t,x,z)\mathrm{d}z\leq\int_{{\mathbb{R}}^{d}}|q_{N}-q|(s,t,x,z)\mathrm{d}z$
$\displaystyle\qquad\leq
C(t-s)^{1/2}\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{\beta^{-1}-1}+\ln(1/\gamma_{1}^{N})(\gamma_{1}^{N})^{1/2}\right)$
$\displaystyle\qquad\leq
C(t-s)^{1/2}\left(I_{\\{\frac{1}{2}<\beta<1\\}}N^{-(1-\beta)}+\ln(N)N^{-\beta/2}\right).$

The proof of the proposition will be given in Section A.3. The proposition can
be used for getting a result on the distributions of the diffusions on an
increasing grid of time points. With $m_{N}\geq 1$,
$z_{1},...,z_{m_{N}},x\in{\mathbb{R}}^{d}$, $z$, $\tau^{N}_{j}$ and
$Q_{x}^{m_{N}}$ defined as in the last subsection, denote the conditional
distribution of $(X_{N,t_{j}}:1\leq j\leq m_{N})$ given $X_{N,0}=x$ by
$Q_{N,x}^{m_{N}}$. We get the following corollary of Proposition 2.5 for the
Hellinger distance and L1-distance between the measures $Q_{x}^{m_{N}}$ and
$Q_{N,x}^{m_{N}}$. The proof of this result can also be found in Section A.3.

######  Proposition 2.6.

Suppose that (15) holds. With a measure $\nu$ that dominates $Q_{x}^{m_{N}}$
and $Q_{N,x}^{m_{N}}$ it holds for $x\in{\mathbb{R}}^{d}$ with $|x|\leq
a_{N}/2$ that

$\displaystyle\frac{1}{2}\int\left|\frac{\mathrm{d}Q_{x}^{m_{N}}}{\mathrm{d}\nu}-\frac{\mathrm{d}Q_{N,x}^{m_{N}}}{\mathrm{d}\nu}\right|\mathrm{d}\nu\leq
H(Q_{x}^{m_{N}},Q_{N,x}^{m_{N}})$ $\displaystyle\qquad\leq
Cm_{N}^{1/4}\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{(\beta^{-1}-1)/2}+\sqrt{\ln(1/\gamma_{1}^{N})}(\gamma_{1}^{N})^{1/4}\right)$
$\displaystyle\qquad\leq
Cm_{N}^{1/4}\left(I_{\\{\frac{1}{2}<\beta<1\\}}N^{-(1-\beta)/2}+\sqrt{\ln(N)}N^{-\beta/4}\right).$

In particular, we have that the upper bound in the proposition converges to 0
if $m_{N}$ is of the form $m_{N}=CN^{\mu}$ with $\mu<\beta$ for
$\frac{1}{2}<\beta\leq\frac{2}{3}$, $\mu<2(1-\beta)$ for
$\frac{2}{3}\leq\beta<1$ and $\mu<1$ for $\beta=1$.

###  2.5 The parametrix method

The main tool of our proofs is the parametrix method. This method allows to
represent transition densities of certain processes by so-called parametrix
series. A parametrix for a differential operator is often easier to construct
than a fundamental solution and for many purposes it is almost as powerful.
Sometimes it is also possible to construct a fundamental solution from a
parametrix by iteratively improving it. The idea of parametrix representations
is old and goes back to E. Lévy, who considered non-degenerate second-order
operators in non-divergent form, see [13]. It is based on perturbative theory
methods. In real time, the density of an SDE with variable coefficients is a
priori close to the density of an SDE with constant coefficients, for which we
have good density controls. The idea of the method is to use Kolmogorov
equations satisfied by the two densities for precise estimates of their
difference. In addition to Lévy’s approach, other versions of the parametrix
method have been developed. E.g. an approach has been proposed by [19] to
obtain asymptotic expansions of the Laplacian spectrum on a manifold as a
function of its curvature. This approach allows to study errors of discrete
approximation schemes. In the framework of inhomogeneous non-degenerate
diffusion processes it has been used in [15] to obtain local limit
propositions for approximating Markov processes and in [16] to get error
bounds for Euler schemes. In recent years there were some progresses of the
parametrix method in the literature. Without claiming to be complete, we only
mention extensions to processes with jumps, to McKean–Vlasov type equations,
see [23], to degenrate Kolmogorov type diffusions, see [11, 17] and to
parabolic SPDEs, see [25]. We now explain the core of the method for the
example of a classical Brownian diffusion without going into technical
details. We just explain the "Master formula" as it appeared in the celebrated
article [19]. In this example we are interested in studying Brownian SDEs of
the form

$Z_{t}=z+\int_{0}^{t}b(s,Z_{s})\mathrm{d}s+\int_{0}^{t}\sigma(s,Z_{s})\mathrm{d}W_{s},$
(16)

where $(W_{s})_{s\geq 0}$ is an ${\mathbb{R}}^{k}$-valued Brownian motion on
some filtered probability space $(\Omega.\mathcal{F},(\mathcal{F}_{t})_{t\geq
0},\mathbb{P})$ and where the process $Z_{t}$ is ${\mathbb{R}}^{k}$-valued.
The coefficients $b$ and $\sigma$ are ${\mathbb{R}}^{k}$-valued or
${\mathbb{R}}^{k}\times{\mathbb{R}}^{k}$-valued, respectively and under
appropriate assumptions on $b$ and $\sigma$ a unique weak solution of (16)
exists and admits a transition density/fundamental solution $p(s,t,x,y)$.
Along with equation (16), consider the equation with coefficients "frozen" at
the point $y$

$\tilde{Z}_{t}=z+\int_{0}^{t}b(s,y)\mathrm{d}s+\int_{0}^{t}\sigma(s,y)\mathrm{d}W_{s}$
(17)

with Gaussian transition density $\tilde{p}(s,t,x,y)$. Below we will make use
of the backward and forward Kolmogorov equations:

$\frac{\partial\tilde{p}}{\partial s}+\tilde{L}\tilde{p}=0,\quad\frac{\partial
p}{\partial s}+Lp=0,\quad-\frac{\partial\tilde{p}}{\partial
t}+\tilde{L}^{*}\tilde{p}=0,\quad-\frac{\partial p}{\partial t}+L^{*}p=0.$
(18)

together with the initial conditions

$\tilde{p}(t,t,x,y)=\delta(x-y)\text{ and }p(t,t,x,y)=\delta(x-y).$ (19)

With the help of (18) and (19) we can write the basic equality for the
parametrix method:

$\displaystyle p(s,t,x,y)-\tilde{p}(s,t,x,y)$
$\displaystyle\qquad=\int_{s}^{t}\mathrm{d}u\frac{\partial}{\partial
u}\left[\int_{{\mathbb{R}}^{k}}p(s,u,x,z)\tilde{p}(u,t,z,y)\mathrm{d}z\right]$
$\displaystyle\qquad=\int_{s}^{t}\mathrm{d}u\int_{{\mathbb{R}}^{k}}\left[\tilde{p}(u,t,z,y)L^{*}p(s,u,x,z)-p(s,u,x,z)\tilde{L}\tilde{p}(u,t,z,y)\right]\mathrm{d}z$
$\displaystyle\qquad=\int_{s}^{t}\mathrm{d}u\int_{{\mathbb{R}}^{k}}\left[p(s,u,x,z)(L-\tilde{L})\tilde{p}(u,t,z,y)\right]\mathrm{d}z.$

This equation can be written as

$p=\tilde{p}(t,t,x,y)+p\otimes H,$ (20)

where $H=[L-\tilde{L}]\tilde{p}$ and the convolution type binary operation
$\otimes$ is defined by

$(f\otimes
g)(s,t,x,y)=\int_{s}^{t}\mathrm{d}u\int_{\mathbb{R}^{d}}f(s,u,x,z)g(u,t,z,y)\mathrm{d}z.$
(21)

Iterative application of (20) gives an infinite series

$p=\sum_{r=0}^{\infty}\tilde{p}\otimes H^{(r)},$ (22)

where $\tilde{p}\otimes H^{(0)}=\tilde{p}$ and $\tilde{p}\otimes
H^{(r+1)}=(\tilde{p}\otimes H^{(r)})\otimes H$ for $r=0,1,2,...$ . An
important property of representation (22) is that it allows us to express the
non-Gaussian density $p$ in terms of Gaussian densities $\tilde{p}$. Equation
(22) is the "Master formula" in our proof. We will apply it twice, to the
truncated diffusion $X_{t}^{N}$ and to the truncated Robbins-Monro process
$U_{t}^{N}$. In the latter application the parametrix method allows us to
express transition densities of the Robbins-Monro process as an expression
depending on the densities of sums of independent random variables. The main
idea of the proof is based on the comparison of the densities of sums with the
Gaussian densities showing up in the parametrix expansion of the truncated
diffusion $X_{t}^{N}$.

We will make use of infinite series expansions of $q_{N}$

$\displaystyle q_{N}(t,s,x,y)$ $\displaystyle=$
$\displaystyle\sum_{r=0}^{\infty}\tilde{q}_{N}\otimes H_{N}^{(r)}(t,s,x,y),$
(23)

where the notation on the right hand side of this equation will be explained
now. The equation is based on looking at a frozen diffusion process
$\tilde{X}_{t}^{s,x,y}$

$\tilde{X}_{t}^{s,x,y}=x+\int_{s}^{t}F_{N}(u,\theta_{u,s}^{N}(y))\chi_{N}(\theta_{u,s}^{N}(y)\mathrm{d}u+\int_{s}^{t}R^{1/2}(\bar{\theta}_{u})\mathrm{d}B_{u},$

where the functions $\theta_{t,s}^{N}$ ($0\leq t\leq s$) are the solutions of
the following ordinary differential equations

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\theta_{t,s}^{N}(y)=F_{N}(t,\theta_{t,s}^{N}(y))\chi_{N}(\theta_{t,s}^{N}(y))$
(24)

with boundary condition $\theta_{s,s}^{N}(y)=y$. This is a Gaussian process
with transition density

$\tilde{q}_{N}(t,s,x,y)=\tilde{q}_{N}^{t,x,y}(t,s,x,z)|_{z=y}=g_{\bar{\sigma}}(t,s,\theta_{t,s}^{N}(y)-x),$

where

$\displaystyle g_{\bar{\sigma}}(t,s,z)$ $\displaystyle=$
$\displaystyle\frac{1}{(2\pi)^{d/2}\sqrt{\det\bar{\sigma}(t,s)}}\exp\left(-\frac{1}{2}z^{\intercal}\bar{\sigma}(t,s)^{-1}z\right),$
$\displaystyle\bar{\sigma}(t,s)$ $\displaystyle=$
$\displaystyle\int_{t}^{s}R(\bar{\theta}_{u})\mathrm{d}u.$

We define

$H_{N}(t,s,x,y)=(L_{t}^{N}-\tilde{L}_{t}^{N})\tilde{q}_{N}(t,s,x,y),$

where $L_{t}^{N}$ and $\tilde{L}_{t}^{N}$ are the following generators:

$\displaystyle L_{t}^{N}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i,j=1}^{d}R_{ij}(\bar{\theta}_{t})\frac{\partial^{2}}{\partial
x_{i}\partial
x_{j}}+\sum_{i=1}^{d}\left(\sum_{j=1}^{d}\left[F_{N}(t,x)\right]_{i,j}\left[\chi_{N}(x)\right]_{j}\right)\frac{\partial}{\partial
x_{i}},$ $\displaystyle\tilde{L}_{t}^{N}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i,j=1}^{d}R_{ij}(\bar{\theta}_{t})\frac{\partial^{2}}{\partial
x_{i}\partial
x_{j}}+\sum_{i=1}^{d}\left(\sum_{j=1}^{d}\left[F_{N}(t,\theta_{t,s}^{N}(y))\right]_{i,j}\left[\chi_{N}(\theta_{t,s}^{N}(y))\right]_{j}\right)\frac{\partial}{\partial
x_{i}},$

where the flow $\theta_{t,s}^{N}(y)$ is defined in (24). The convolution type
operation $\otimes$ is defined as in (21) and for $r=1,2,...$ the $r$-fold
convolution is given by $g\otimes H_{N}^{(r)}=(g\otimes H_{N}^{(r-1)})\otimes
H_{N}$ with $g\otimes H_{N}^{(0)}=g$. The validity of formula (22) for our
choices of $p,\tilde{p}$, and $H$ and the correctness of (23) for
$q_{N},\tilde{q}_{N}$, and $H_{N}$ follows from Lemmas 3.3 and 3.4 stated in
Subsection 3.1.

For the proof in the next subsection we will make use of the following series
expansion of $p_{N}$ for $l<k$

$\displaystyle p_{N}(t_{l}^{N},t_{k}^{N},x,y)$ $\displaystyle=$
$\displaystyle\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}^{(r)}(t_{l}^{N},t_{k}^{N},x,y),$
(25)

where the notation on the right hand side of this equation will be explained
now. For this purpose we consider the following frozen Markov chain

$\displaystyle\tilde{V}_{t_{i+1}^{N}}^{N,y}$ $\displaystyle=$
$\displaystyle\tilde{V}_{t_{i}^{N}}^{N,y}+\left(\theta_{t_{i+1}^{N},t_{j}^{N}}^{N}(y)-\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)\right)$
$\displaystyle\quad-\sqrt{\gamma_{i+1}^{N}}\xi\left(\bar{\theta}_{t_{i}^{N}}+\chi_{N}\left(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)\right)\sqrt{\gamma_{i}^{N}},\eta_{i+1}\right)$
$\displaystyle=$
$\displaystyle\tilde{V}_{t_{i}^{N}}^{N,y}+\int_{t_{i}^{N}}^{t_{i+1}^{N}}F_{N}\left(u,\theta_{u,t_{j}^{N}}^{N}(y)\right)\chi_{N}\left(\theta_{u,t_{j}^{N}}^{N}(y)\right)\mathrm{d}u$
$\displaystyle\quad-\sqrt{\gamma_{i+1}^{N}}\xi\left(\bar{\theta}_{t_{i}^{N}}+\chi_{N}\left(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)\right)\sqrt{\gamma_{i}^{N}},\eta_{i+1}\right).$

By iterative application of (2.5) we get that for $k<j$

$\tilde{V}_{t_{j}^{N}}^{N,y}=\tilde{V}_{t_{k}^{N}}^{N,y}+y-\theta_{t_{k}^{N},t_{j}^{N}}^{N}(y)-\sum_{i=k}^{j-1}\sqrt{\gamma_{i+1}^{N}}\xi_{i},$

where

$\xi_{i}=\xi\left(\bar{\theta}_{t_{i}^{N}}+\chi_{N}\left(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)\right)\sqrt{\gamma_{i}^{N}},\eta_{i+1}\right).$

For the transition density $\tilde{p}_{N}^{y}(t_{k}^{N},t_{j}^{N},x,z)$ of the
frozen Markov chain we have

$\displaystyle\tilde{p}_{N}^{y}(t_{k}^{N},t_{j}^{N},x,z)$ $\displaystyle=$
$\displaystyle\frac{\mathrm{d}}{\mathrm{d}z}P\left(\tilde{V}_{t_{j}^{N}}^{N,y}\in\mathrm{d}z\big{|}\tilde{V}_{t_{k}^{N}}^{N,y}=x\right)$
$\displaystyle=$ $\displaystyle
p_{S_{N}}\left(z-x-y+\theta_{t_{k}^{N},t_{j}^{N}}^{N}(y)\right),$

where $p_{S_{N}}$ denotes the density of
$-\sum_{i=k}^{j-1}\sqrt{\gamma_{i+1}^{N}}\xi_{i}$. Note that

$\tilde{p}_{N}^{y}(t_{k}^{N},t_{j}^{N},x,y)=p_{S_{N}}\left(\theta_{t_{k}^{N},t_{j}^{N}}^{N}(y)-x\right).$

For a test function $\phi$ we define now the one step generators:

$\displaystyle\mathcal{L}_{N}\phi(t_{i}^{N},t_{j}^{N},x,y)$ $\displaystyle=$
$\displaystyle\frac{1}{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}\left(\phi(t_{i+1}^{N},t_{j}^{N},z,y)-\phi(t_{i+1}^{N},t_{j}^{N},x,y)\right)p_{N}(t_{i}^{N},t_{i+1}^{N},x,z)\mathrm{d}z,$
$\displaystyle\tilde{\mathcal{L}}_{N}\phi(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle=$
$\displaystyle\frac{1}{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}\left(\phi(t_{i+1}^{N},t_{j}^{N},z,y)-\phi(t_{i+1}^{N},t_{j}^{N},x,y)\right)\tilde{p}^{y}_{N}(t_{i}^{N},t_{i+1}^{N},x,z)\mathrm{d}z.$

We put

$\mathcal{K}_{N}(t_{i}^{N},t_{j}^{N},x,y)=\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)\tilde{p}^{y}_{N}(t_{i}^{N},t_{j}^{N},x,y)$

and with the discretized time convolution

$(f\otimes_{N}g)(t_{i}^{N},t_{j}^{N},x,y)=\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{\mathbb{R}^{d}}f(t_{i}^{N},t_{k}^{N},x,z)g(t_{k}^{N},t_{j}^{N},z,y)\mathrm{d}z$

we define for $r=1,2,...$ the $r$-fold convolution as
$g\otimes_{N}\mathcal{K}_{N}^{(r)}=(g\otimes_{N}\mathcal{K}_{N}^{(r-1)})\otimes_{N}\mathcal{K}_{N}$
with $g\otimes_{N}\mathcal{K}_{N}^{(0)}=g$. With this notation one can show
that (25) holds. For the proof one makes repeated use of the Markov property,
see also Lemma 3.6 in Konakov and Mammen (2000).

###  2.6 Comparison of the truncated version of the Robbins-Monro algorithm
with the truncated diffusion

In this subsection we want to prove the following bound for the difference
$|q_{N}-p_{N}|$ of the transition density $p_{N}$ of the truncated Robbins-
Monro procedure $V_{t_{i}^{N}}^{N}$ and of the transition density $q_{N}$ of
the truncated diffusion $X_{t}^{N}$. The main result of this section is the
following result

######  Theorem 2.2.

For $i<j$ it holds that for all $x,y\in\mathbb{R}^{n}$ with a constant $C>0$

$\displaystyle|q_{N}(t_{i}^{N},t_{j}^{N},x,y)-p_{N}(t_{i}^{N},t_{j}^{N},x,y)|$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln^{2}(1/\gamma_{1}^{N})\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta_{t_{j}^{N},t_{i}^{N}}^{N}(x)).$

In the statement of the theorem for a natural number $m$ and positive real
numbers $t$ we define

$\mathcal{Q}_{m}(t,x)=t^{-d/2}Q_{m}(t^{-1/2}x)$

with $Q_{m}$ defined in (A4).

We now come to the proof of Theorem 2.2. Note that by (23) and (25) we have
for $i<j$ that

$q_{N}(t_{i}^{N},t_{j}^{N},x,y)-p_{N}(t_{i}^{N},t_{j}^{N},x,y)=\sum_{r=0}^{\infty}\tilde{q}_{N}\otimes
H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)-\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y).$

With this expansion Theorem 2.2 follows immediately from the following lemmas.
In the first lemma we replace in the parametrix expansion of $q_{N}$ the
convolution operation $\otimes$ by the discrete convolution $\otimes_{N}$. In
Lemma 2.8 we show that it suffices to consider only the first $N$ terms in the
expansion. Lemma 2.9 now is the heart of our argument. We replace in the
parametrix expansions the Gaussian densities $\tilde{q}_{N}$ by the densities
$\tilde{p}_{N}$ of normed sums of independent random variables. We use
Edgeworth expansion arguments and local limit propositions that offer powerful
tools to bound the errors of this replacement. Note that $\tilde{q}_{N}$ is
replaced by $\tilde{p}_{N}$ at two places: in the summation and in the
definition of the kernels $H_{N}$ and $K_{N}$. At this point we apply Lemma
3.2. The kernel $K_{N}$ is defined as $H_{N}$, but with $\tilde{q}_{N}$
replaced by $\tilde{p}_{N}$. Finally, Lemma 2.10 bounds errors that show up by
replacing the kernel $K_{N}$ by the kernel $\mathcal{K}_{N}$, which is used in
the parametrix expansion of the Robbins-Monro algorithm. Lemma 2.9 will be
proved in Subsection 3.3. The proofs of the other lemmas can be found in
Subsection A.4 of the supplementary material.

######  Lemma 2.7.

For $i<j$ it holds with some constant $C>0$ that

$\displaystyle\left|\sum_{r=0}^{\infty}\tilde{q}_{N}\otimes
H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)-\sum_{r=0}^{\infty}\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y),$

where $\bar{q}_{N}$ is the transition density of the diffusion
$\bar{X}_{t}^{N}$ defined in (89).

Note that we have the bound (94) for $\bar{q}_{N}$.

######  Lemma 2.8.

For $i<j$ it holds with some constant $C>0$ that

$\displaystyle\left|\sum_{r=N+1}^{\infty}\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq C\exp(-CN)\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$

######  Lemma 2.9.

For $i<j$ it holds with some constant $C>0$ that

$\displaystyle\left|\sum_{r=0}^{N}\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)-\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq
C\ln(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}(t_{j}^{N}-t_{i}^{N})^{1/2}\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},y-\theta_{t_{j}^{N},t_{i}^{N}}^{N}(x)),$

where

$K_{N}(t_{i}^{N},t_{j}^{N},x,y)=(L_{t_{i}^{N}}^{N}-\tilde{L}_{t_{i}^{N}}^{N})\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y).$

The convolutions $K_{N}^{(r)}$ are calculated using the convolution
$\otimes_{N}$, in contrast to $H_{N}^{(r)}$ where as above the convolution
operation $\otimes$ is used.

######  Lemma 2.10.

For $i<j$ it holds with some constant $C>0$ that

$\displaystyle\left|\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)-\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln{(1/\gamma_{1}^{N})}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta_{t_{j}^{N},t_{i}^{N}}^{N}(x)).$

Theorem 2.2 can be used for getting a result on the distributions of the
truncated diffusion and truncated Robbins-Monro procedure on an increasing
grid of time points. With $m_{N}\geq 1$,
$z_{1},...,z_{m_{N}},x\in{\mathbb{R}}^{d}$, $z$, $\tau^{N}_{j}$,
$Q_{N,x}^{m_{N}}$ and $P_{N,x}^{m_{N}}$ defined as in the subsections 2.3 and
2.4 we get the following corollary of Proposition 2.5 for the Hellinger
distance and L1-distance between the measures $Q_{x}^{m_{N}}$ and
$Q_{N,x}^{m_{N}}$. The proof of this result can be found in Section 3.4.

######  Proposition 2.11.

Suppose that (15) holds. With a measure $\nu$ that dominates $P_{N,x}^{m_{N}}$
and $Q_{N,x}^{m_{N}}$ it holds for $x\in{\mathbb{R}}^{d}$ with $|x|\leq
a_{N}/2$ that

$\displaystyle\int\left|\frac{\mathrm{d}Q_{N,x}^{m_{N}}}{\mathrm{d}\nu}-\frac{\mathrm{d}P_{N,x}^{m_{N}}}{\mathrm{d}\nu}\right|\mathrm{d}\nu\leq
C\sqrt{\gamma_{1}^{N}}m_{N}\ln^{2}(1/\gamma_{1}^{N})\leq
Cm_{N}N^{-\beta/2}\ln^{2}(N)$

In particular, we have that the upper bound in the proposition converges to 0
if $m_{N}$ is of the form $m_{N}=CN^{\mu}$ with $\mu<\beta$.

###  2.7 Convergence of the Robbins-Monro process under an additional Lyapunov
condition to a stationary diffusion

If the Lyapunov condition (A6) stated below is satisfied for the system (6)
then the stationary point $\theta^{*}$ is asymptotically stable for this
system. That is for any $\varepsilon>0$ there exists $\delta>0$ with the
following property: for all $\theta_{0}$ such that
$|\theta_{0}-\theta^{*}|<\delta$, the solution $\bar{\theta}_{t}$ of (6) with
starting value $\bar{\theta}_{0}=\theta_{0}$ is defined for all $t>0$ and it
holds that

$\sup_{t\in[0,\infty)}|\bar{\theta}_{t}-\theta^{*}|<\varepsilon.$

Moreover,

$|\bar{\theta}_{t}-\theta^{*}|\to 0$

for $t\to\infty$, provided $|\theta_{0}-\theta^{*}|$ is small enough.

  * (A6)

(Lyapunov condition) All eigen values of the matrix
$\bar{\alpha}+\mathcal{D}h(\theta^{*})$ have strictly negative real parts.

The next assumption states that the mean vector field $h(\theta)$ is "inward".

  * (A7)

There exists $\delta>0$ with

$\liminf_{k\to\infty}\left(2\delta\frac{\gamma_{k}}{\gamma_{k+1}}+\frac{\gamma_{k+1}-\gamma_{k}}{\gamma^{2}_{k+1}}\right)>0$

such that

$\langle\theta-\theta^{*},h(\theta)\rangle\leq-\delta|\theta-\theta^{*}|^{2}.$

For $\gamma_{k}=A(k^{\beta}+B)^{-1}$ with $A>0,B\geq 0$ the first assumption
of Assumption (A7) holds for all $\delta>0$ if $\frac{1}{2}<\beta<1$ and for
$\delta>\beta/(2A)$ if $\beta=1$. Under Assumptions (A1)–(A7) one can analyze

$U^{*,N}_{t}=\frac{\theta_{k}^{N}-\theta^{*}}{\sqrt{\gamma_{k}^{N}}}\text{ for
}t\in[t_{k}^{N},t_{k+1}^{N})$

with limiting diffusion $X_{t}^{*}$, see (11), similarly as $U^{N}_{t}$ with
limiting diffusion $X_{t}^{N}$. Cutoff points for the definition of the
truncated processes can be chosen similarly as in the nonstationary case. For
the cutoff error of the diffusion one can apply again the results in [4]
getting a bound similar to the one stated in Proposition 2.5. For the
comparison of the truncated and untruncated Robbins-Monro procedure one gets
the bound of Proposition 2.3 proceeding exactly as in the proof of this
proposition. Finally, one can show the local limit theorem 2.2 for the
truncated versions of $U^{*,N}_{t}$ and $X_{t}^{*}$ proceeding essentially
again as for the truncated versions of $U^{N}_{t}$ and $X_{t}$. Note also that
for this result it is not necessary that Assumption (A3) holds in a tubular
neighborhood of the ODE solution. It suffices that the assumption holds in a
neighborhood of the unique stationary point $\theta^{*}$, $h(\theta^{*})=0$.

##  3 Proofs

###  3.1 Some bounds

In this subsection we will state some bounds that will be used in the proofs
in the following subsections. The proofs of the lemmas of this subsection can
be found in Subsection A.6 in the supplement. The first lemma states that
$F_{N}(t_{k}^{N},x)\chi_{N}(x)$ is uniformly Lipschitz in
$x\in{\mathbb{R}}^{d}$ for $N$ large enough:

######  Lemma 3.1.

For $x,y\in{\mathbb{R}}^{d}$ and $N$ large enough it holds with some constant
$L>0$ that

$\displaystyle\|F_{N}(t_{k}^{N},x)\chi_{N}(x)-F_{N}(t_{k}^{N},y)\chi_{N}(y)\|\leq
L\|x-y\|.$ (27)

Furthermore, in the following subsections we will make use of the following
inequalities stated in the next lemma.

######  Lemma 3.2.

For $0\leq|\nu|\leq 4$ and $z,y\in{\mathbb{R}}^{d}$ it holds that

$\displaystyle|D_{z}^{\nu}(\tilde{p}_{N}-\tilde{q}_{N})(t_{i}^{N},t_{j}^{N},z,y)|\leq
C{\sqrt{\gamma_{1}^{N}}}\left({(t_{j}^{N}-t_{i}^{N})^{-(|\nu|+1)/2}}+(t_{j}^{N}-t_{i}^{N})^{1-|\nu|/2}a_{N}\right)$
(28)
$\displaystyle\quad\quad\quad\times\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-z),$
$\displaystyle|D_{z}^{\nu}\varphi(t_{i}^{N},t_{j}^{N},z,y)|\leq
C(t_{j}^{N}-t_{i}^{N})^{-|\nu|/2}\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-z)$
(29)

for $\varphi=\tilde{q}_{N}$ and $\varphi=\tilde{p}_{N}$.

We now compare the flow $\theta_{t,s}^{N}$ defined in (24) with the flow
$\theta_{t,s}$ ($0\leq t\leq s$) that is defined as the solutions of the
following ordinary differential equations

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\theta_{t,s}(y)=\left(\bar{\alpha}I+\mathcal{D}h(\bar{\theta}_{t})\right)\theta_{t,s}(y)$
(30)

with boundary condition $\theta_{s,s}(y)=y$. The following lemma collects
bounds for and between $\theta_{t,T}(y)$, $\theta^{N}_{t,T}(y)$, and $y$.

######  Lemma 3.3.

For all $t,t_{k}^{N}\in[0,T]$ and for all $x,y\in{\mathbb{R}}^{d}$ the
following bounds hold with a constant $C>1$, depending only on $T$.

$\displaystyle
C^{-1}\|\theta^{N}_{s,t}(y)-x\|\leq\|\theta^{N}_{t,s}(x)-y\|\leq
C\|\theta^{N}_{s,t}(y)-x\|,$ (31) $\displaystyle
C^{-1}\|\theta^{N}_{s,t}(y)\|\leq\|y\|\leq C\|\theta^{N}_{s,t}(y)\|$ (32)
$\displaystyle C^{-1}\|y\|\leq\|\theta_{t,T}(y)\|\leq C\|y\|,$ (33)
$\displaystyle\|\theta^{N}_{t,T}(y)-\theta_{t,T}(y)\|\leq
C(T-t)\left(\ln(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\mathbb{I}_{\beta=1}+(\gamma_{1}^{N})^{-1+1/\beta}\mathbb{I}_{1/2<\beta<1}\right).$
(34)

######  Lemma 3.4.

For all $t,v\in[0,T]$ and for all $x,y\in{\mathbb{R}}^{d}$ the following
bounds hold with some constant $C>0$

$\displaystyle\left|H^{(r)}(t,v,x,y)\right|\leq
C^{r}\frac{\Gamma^{r}(1/2)}{\Gamma(r/2)}(v-t)^{(r-d-2)/2}\exp\left(-\frac{(x-\theta_{t,v}^{N}(y))^{2}}{C|v-t|}\right),$
(35) $\displaystyle\left|H_{N}^{(r)}(t,v,x,y)\right|\leq
C^{r}\frac{\Gamma^{r}(1/2)}{\Gamma(r/2)}(v-t)^{(r-d-2)/2}\exp\left(-\frac{(x-\theta_{t,v}^{N}(y))^{2}}{C|v-t|}\right),$
(36) $\displaystyle\left|\left(H^{(r)}-H_{N}^{(r)}\right)(t,v,x,y)\right|$
(37)
$\displaystyle\qquad\leq(r+1)C^{r+1}\frac{\Gamma^{r}(1/2)}{\Gamma(r/2)}\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}(v-t)^{{(r-d-2)/2}}(1+|y|)\exp\left(-\frac{(x-\theta_{t,v}^{N}(y))^{2}}{C|v-t|}\right).$

The following bound follows from Theorem 1.2 in [20].

######  Lemma 3.5.

For $s<t$ and $x,y\in{\mathbb{R}}^{d}$ it holds that

$\displaystyle q_{N}(s,t,x,y)\leq
C(t-s)^{-d/2}\exp\left(-C\frac{(y-\theta^{N}_{t,s}(x))^{2}}{t-s}\right).$

We conclude this subsection by stating the following simple lemma without
proof.

######  Lemma 3.6.

For $r\geq 1$ and $t>0$, $z,\delta\in{\mathbb{R}}^{d}$ it holds that

$\displaystyle\mathcal{Q}_{r}(t,z+\delta)$ $\displaystyle\leq$ $\displaystyle
C{}\mathcal{Q}_{r}(t,z)(1+\|t^{-1/2}\delta\|)^{r},$ (38)
$\displaystyle\|t^{-1/2}z\|\mathcal{Q}_{r}(t,z)$ $\displaystyle\leq$
$\displaystyle C{}\mathcal{Q}_{r-1}(t,z).$ (39)

###  3.2 Proof of Proposition 2.3

We introduce the exit time

$\tau^{N}_{a_{N}}=\inf\\{k\in[1,M(N)]:\|V_{t_{k}^{N}}^{N}\|\geq a_{N}\\}$

and consider the processes $U_{t_{k}^{N}}^{N}$ and $V_{t_{k}^{N}}^{N}$ for
$k\leq\tau^{N}_{a_{N}}$. Note that $U_{t_{0}^{N}}^{N}=V_{t_{0}^{N}}^{N}$ and
$U_{t_{1}^{N}}^{N}=V_{t_{1}^{N}}^{N}+\beta_{1}^{N}$. Furthermore, for $1\leq
k<\tau^{N}_{a_{N}}$ we have because of $F_{N}(t,x)=G_{N}(t,x)$ for $\|x\|\leq
a_{N}$ that

$\displaystyle\left\|U_{t_{k+1}^{N}}^{N}-V_{t_{k+1}^{N}}^{N}\right\|\leq(1+L\gamma_{k+1}^{N})\left\|U_{t_{k}^{N}}^{N}-V_{t_{k}^{N}}^{N}\right\|+C\sqrt{\gamma_{k+1}^{N}\gamma_{k}^{N})}\left\|U_{t_{k}^{N}}^{N}-\chi_{N}(V_{t_{k}^{N}}^{N})\right\|+\left\|\beta_{k+1}^{N}\right\|,$

where $C/2$ is a bound for the Lipschitz constants of $h$ and of $H$ in its
first argument, see Assumption (A1). We now use that
$\left\|U_{t_{k}^{N}}^{N}-\chi_{N}(V_{t_{k}^{N}}^{N})\right\|=\left\|U_{t_{k}^{N}}^{N}-V_{t_{k}^{N}}^{N}\right\|$
for $k\leq\tau^{N}_{a_{N}}$, that $\gamma_{k}$ is monotone decreasing, and
that $\left\|\beta_{k+1}^{N}\right\|$ is bounded by a constant times
$(\gamma^{N}_{k+1})^{3/2}$, see the Proof of Proposition 2.2. This gives that:

$\displaystyle\left\|U_{t_{k+1}^{N}}^{N}-V_{t_{k+1}^{N}}^{N}\right\|\leq(1+C^{\prime}\gamma_{k}^{N})\left\|U_{t_{k}^{N}}^{N}-V_{t_{k}^{N}}^{N}\right\|+C^{\prime}(\gamma^{N}_{k})^{3/2},$

where $C^{\prime}>0$ is a new constant. Now, by definition of $M(N)$ we have
that $\gamma_{1}^{N}+...+\gamma_{M(N)-1}^{N}<T$. Because of $k<M(N)$ for
$k<\tau^{N}_{N}$ we have that

$\displaystyle\left\|U_{t_{k+1}^{N}}^{N}-V_{t_{k+1}^{N}}^{N}\right\|$
$\displaystyle\leq$
$\displaystyle(1+C^{\prime}\gamma_{k}^{N})\cdot...\cdot(1+C^{\prime}\gamma_{1}^{N})\sum_{l=1}^{k}C^{\prime}(\gamma^{N}_{l})^{3/2}$
$\displaystyle\leq$ $\displaystyle C^{\prime\prime}(\gamma^{N}_{1})^{1/2}$

with $C^{\prime\prime}=\exp(C^{\prime}T)C^{\prime}T$. We conclude that

$\displaystyle\mathbb{P}\left(\sup_{1\leq k\leq
M(N)}\left\|U_{t_{k}^{N}}^{N}-V_{t_{k}^{N}}^{N}\right\|>C^{\prime\prime}\sqrt{\gamma_{1}^{N}}\right)\leq\mathbb{P}\left(\tau^{N}_{N}<M(N)\right)$
$\displaystyle\qquad=\mathbb{P}\left(\sup_{1\leq k\leq
M(N)-1}\left\|V_{t_{k}^{N}}^{N}\right\|>a_{N}\right)$
$\displaystyle\qquad\leq\mathbb{P}\left(\sup_{1\leq k\leq
M(N)}\left\|V_{t_{k}^{N}}^{N}\right\|>a_{N}\right).$

As mentioned after the statement of our assumptions the process $U_{t}^{N}$
converges in distribution to the diffusion $(X_{t}:0\leq t\leq T)$ defined in
(9). The same holds for the process $V_{t}^{N}$, see Lemma 11.2.1, Theorem
10.2.2 and Theorem 11.2.3 in [27]. Now for any $\epsilon>0$ there exists a
level $K_{\epsilon}$ with

$\mathbb{P}(\sup_{0\leq t\leq T}\|X_{t}\|\geq K_{\epsilon})\leq\epsilon.$

This shows that the upper bound $\mathbb{P}\left(\sup_{1\leq k\leq
M(N)}\left\|V_{t_{k}^{N}}^{N}\right\|>a_{N}\right)$ converges to 0 because of
$a_{N}\to\infty$ for $N\to\infty$ which concludes the proof of the
proposition. $\square$

###  3.3 Proof of Lemma 2.9

We will show

$\displaystyle|(H_{N}-K_{N})(t_{i}^{N},t_{j}^{N},z,y)|$ $\displaystyle\leq$
$\displaystyle C|z-\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)|\
|\nabla_{z}(\tilde{p}_{N}-\tilde{q}_{N})(t_{i}^{N},t_{j}^{N},z,y)|,$ (40)
$\displaystyle|H_{N}(t_{i}^{N},t_{j}^{N},z,y)|$ $\displaystyle\leq$
$\displaystyle
C\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)),$
(41)
$\displaystyle|(\tilde{p}_{N}\otimes_{N}K^{(r)}_{N})(t_{i}^{N},t_{j}^{N},z,y)|$
$\displaystyle\leq$
$\displaystyle\frac{(C(t_{j}^{N}-t_{i}^{N}))^{r}}{r!}\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},z-\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y)),$
(42)

and with some constant $\bar{c}$ and with $m=M-d-5-\gamma$

$\displaystyle\int_{\mathbb{R}^{d}}\mathcal{Q}_{m}(t_{k}^{N}-t_{i}^{N},z-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))\mathcal{Q}_{m}(t_{j}^{N}-t_{k}^{N},y-\theta^{N}_{t_{j}^{N},t_{k}^{N}}(z))\mathrm{d}z$
(43)
$\displaystyle\qquad\leq\bar{c}\mathcal{Q}_{m}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$

for all $1\leq i<k\leq j,x,y\in\mathbb{R}^{d}$. In the proof of the lemma we
will make repeated use of Lemma 3.2. We will use it to bound the right hand
side of (40) and in the proof of (42). At both places we replace the Gaussian
densities $\tilde{q}_{N}$ by the densities $\tilde{p}_{N}$ of normed sums of
independent random variables. By application of the lemma we get with the help
of (39) from (40) that

$\displaystyle|(H_{N}-K_{N})(t_{i}^{N},t_{j}^{N},z,y)|$ $\displaystyle\leq$
$\displaystyle
C\left(\frac{\sqrt{\gamma_{1}^{N}}}{\sqrt{t_{j}^{N}-t_{i}^{N}}}+(t_{j}^{N}-t_{i}^{N})a_{N}\sqrt{\gamma_{1}^{N}}\right)$
$\displaystyle\quad\times\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-z).$

We now show that (41)–(3.3) imply the statement of the lemma. For a poof of
this claim we write

$\displaystyle\tilde{q}_{N}\otimes_{N}H_{N}^{(r+1)}(t_{i}^{N},t_{j}^{N},x,y)-\tilde{p}_{N}\otimes_{N}K_{N}^{(r+1)}(t_{i}^{N},t_{j}^{N},x,y)=I+II,$
(45)

where

$\displaystyle I$ $\displaystyle=$
$\displaystyle(\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}-\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\otimes_{N}H_{N}(t_{i}^{N},t_{j}^{N},x,y),$
$\displaystyle II$ $\displaystyle=$
$\displaystyle\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}\otimes_{N}(H_{N}-K_{N})(t_{i}^{N},t_{j}^{N},x,y).$

For a discusssion of the second term II note that we get directly from
(42)–(3.3) that

$\displaystyle II$ $\displaystyle\leq$
$\displaystyle\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{\mathbb{R}^{d}}|\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}|(t_{i}^{N},t_{k}^{N},x,z)|H_{N}-K_{N}|(t_{k}^{N},t_{j}^{N},z,y)\mathrm{d}z$
$\displaystyle\leq$ $\displaystyle
a_{N}\sqrt{\gamma_{1}^{N}}\frac{C^{r+1}}{r!}\sum_{k=i}^{j-1}\frac{(t_{k}^{N}-t_{i}^{N})^{r}}{\sqrt{t_{j}^{N}-t_{k}^{N}}}\gamma_{k+1}^{N}\int_{\mathbb{R}^{d}}\mathcal{Q}_{M-d-1}(t_{k}^{N}-t_{i}^{N},x-\theta^{N}_{t_{i}^{N},t_{k}^{N}}(z))$
$\displaystyle\qquad\times\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{k}^{N},\theta^{N}_{t_{k}^{N},t_{j}^{N}}(y)-z)\mathrm{d}z$
$\displaystyle\leq$ $\displaystyle
a_{N}\sqrt{\gamma_{1}^{N}}\frac{C^{r+1}}{r!}\sum_{k=i}^{j-1}\frac{(t_{k}^{N}-t_{i}^{N})^{r}}{\sqrt{t_{j}^{N}-t_{k}^{N}}}\gamma_{k+1}^{N}\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\leq$ $\displaystyle
a_{N}\sqrt{\gamma_{1}^{N}}\frac{C^{r+1}}{r!}(t_{j}^{N}-t_{i}^{N})^{r+\frac{1}{2}}B\left(\frac{1}{2},r+1\right)\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\leq$ $\displaystyle
a_{N}\sqrt{\gamma_{1}^{N}}\frac{C^{r+2}}{\Gamma\left(r+\frac{3}{2}\right)}(t_{j}^{N}-t_{i}^{N})^{r+\frac{1}{2}}\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)).$

We can apply this inequality to show that

$\displaystyle\sum_{r=0}^{\infty}\left|\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}\otimes_{N}(H_{N}-K_{N})(t_{i}^{N},t_{j}^{N},x,y)\right|$
(46) $\displaystyle\qquad\leq
a_{N}(t_{j}^{N}-t_{i}^{N})^{1/2}\sqrt{\gamma_{1}^{N}}\mathcal{Q}_{M-d-1}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)).$

Iterative application of (45) gives:

$\displaystyle\left|\sum_{r=0}^{N}\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)-\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad=\left|\sum_{r=0}^{N}\tilde{q}_{N}\otimes_{N}(H_{N}^{(r)}-H_{N}^{(r),\otimes_{N}})(t_{i}^{N},t_{j}^{N},x,y)\right.$
$\displaystyle\qquad\qquad+\left.\sum_{r=0}^{N}(\tilde{q}_{N}-\tilde{p}_{N})\otimes_{N}H_{N}^{(r),\otimes_{N}}(t_{i}^{N},t_{j}^{N},x,y)\right.$
$\displaystyle\qquad\qquad+\left.\sum_{r=0}^{N}\sum_{k=0}^{r-1}\tilde{p}_{N}\otimes_{N}K_{N}^{(k)}\otimes_{N}(H_{N}-K_{N})\otimes_{N}H_{N}^{(r-1-k),\otimes_{N}}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq\left|\tilde{q}_{N}\right|\otimes_{N}\sum_{r=0}^{N}\left|H_{N}^{(r)}-H_{N}^{(r),\otimes_{N}}\right|(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad\qquad+\left|\tilde{q}_{N}-\tilde{p}_{N}\right|\otimes_{N}\sum_{r=0}^{N}\left|H_{N}^{(r),\otimes_{N}}\right|(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad\qquad+\sum_{k\geq
0}\left|\tilde{p}_{N}\otimes_{N}K_{N}^{(k)}\otimes_{N}(H_{N}-K_{N})\right|\otimes_{N}\sum_{r=0}^{N}\left|H_{N}^{(r),\otimes_{N}}\right|(t_{i}^{N},t_{j}^{N},x,y).$

The first term of this upper bound can be bounded as follows.

$\displaystyle\left|\tilde{q}_{N}\right|\otimes_{N}\sum_{r=0}^{N}\left|H_{N}^{(r)}-H_{N}^{(r),\otimes_{N}}\right|(t_{i}^{N},t_{j}^{N},x,y)\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$
(47)

This inequality can shown by similar arguments as the proof of (58) in the
proof of Lemma 2.7.

The statement of the lemma now follows by application of (28), (35), (46),
(47) and (63). It remains to show (40)– (43).

For a proof of (40) note that

$\displaystyle(H_{N}-K_{N})(t_{i}^{N},t_{j}^{N},z,y)=(L_{t_{i}^{N}}^{N}-\tilde{L}_{t_{i}^{N}}^{N})(\tilde{q}_{N}-\tilde{p}_{N})(t_{i}^{N},t_{j}^{N},z,y)$
$\displaystyle\qquad=\left(\sum_{i=1}^{d}\left(\sum_{j=1}^{d}[F_{N}(t_{i}^{N},z)]_{i,j}[\chi_{N}(z)]_{j}\right)-\sum_{i=1}^{d}\left(\sum_{j=1}^{d}[F_{N}(t_{i}^{N},\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{i,j}[\chi_{N}(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{j}\right)\right)$
$\displaystyle\qquad\qquad\times\frac{\partial}{\partial
z_{i}}(\tilde{q}_{N}-\tilde{p}_{N})(t_{i}^{N},t_{j}^{N},z,y).$

With the help of (27) this shows (40).

Claim (41) follows directly by application of (36).

We now come to the proof of (43). For this purpose we show for $0\leq
t<s<u\leq T$, $x,y\in\mathbb{R}^{d}$ that with some constant $C>0$

$\displaystyle I(s,t,u,x,y)\leq C\mathcal{Q}_{m}(u-t,y-\theta^{N}_{u,t}(x)),$

where

$I(s,t,u,x,y)=\int_{\mathbb{R}^{d}}\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))\mathcal{Q}_{m}(u-s,y-\theta^{N}_{u,s}(z))\mathrm{d}z.$

This is equivalent to (43). For the proof of this claim note first that we get
from (32) that

$I(s,t,u,x,y)\leq
C\int_{\mathbb{R}^{d}}\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))\mathcal{Q}_{m}(u-s,z-\theta^{N}_{s,u}(y))\mathrm{d}z.$

We now consider two cases: I. $\|y-\theta^{N}_{u,t}(x)\|\leq\sqrt{u-t}$ and
II. $\|y-\theta^{N}_{u,t}(x)\|>\sqrt{u-t}$. We start by considering case I. We
make the additional assumption that $s-t\geq\frac{1}{2}(u-t)$. The case
$s-t\leq\frac{1}{2}(u-t)$ can be treated with the same type of arguments and
for this reason its discussion is omitted. Now we get in the latter case:

$\displaystyle\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))=\frac{1}{(s-t)^{d/2}}Q_{m}\left(\frac{z-\theta^{N}_{s,t}(x)}{\sqrt{s-t}}\right)$
$\displaystyle\qquad\leq\frac{2^{d/2}}{(u-t)^{d/2}}c_{m}\leq\frac{2^{d/2}}{(u-t)^{d/2}}c_{m}\frac{2^{m}}{\left(1+\frac{\|y-\theta^{N}_{u,t}(x)\|}{\sqrt{u-t}}\right)^{m}}$
$\displaystyle\qquad=2^{m+d/2}\mathcal{Q}_{m}(u-t,y-\theta^{N}_{u,t}(x))$

This gives in case I the following bound for $I(s,t,u,x,y)$

$I(s,t,u,x,y)\leq
C\mathcal{Q}_{m}(u-t,y-\theta^{N}_{u,t}(x))\int_{\mathbb{R}^{d}}\mathcal{Q}_{m}(u-s,z-\theta^{N}_{s,u}(y))\mathrm{d}z\leq
C\mathcal{Q}_{m}(u-t,y-\theta^{N}_{u,t}(x)),$

which shows (43) for case I. We now consider case II:
$\|y-\theta^{N}_{u,t}(x)\|>\sqrt{u-t}$. We define the following two sets:

$\displaystyle A_{1}$ $\displaystyle=$
$\displaystyle\\{z\in\mathbb{R}^{d}:\|z-\theta_{s,t}^{N}(x)\|\geq\frac{1}{2}\|\theta_{s,u}^{N}(y)-\theta_{s,t}^{N}(x)\|\\},$
$\displaystyle A_{2}$ $\displaystyle=$
$\displaystyle\\{z\in\mathbb{R}^{d}:\|z-\theta_{s,u}^{N}y)\|\geq\frac{1}{2}\|\theta_{s,u}^{N}(y)-\theta_{s,t}^{N}(x)\|\\}.$

It holds $A_{1}\cup A_{2}=\mathbb{R}^{d}$. We only consider values of $z$ in
$A_{2}$. For such $z$ we get by application of (32)

$\displaystyle\|z-\theta_{s,u}^{N}(y)\|\geq\frac{1}{2}\|\theta_{s,u}^{N}(y)-\theta_{s,t}^{N}(x)\|$
$\displaystyle\qquad=\frac{1}{2}\|\theta_{s,u}^{N}(y)-\theta_{s,u}^{N}(\theta_{u,t}^{N}(x))\|\geq
C\|y-\theta_{u,t}^{N}(x)\|.$

This gives in case II

$\displaystyle\int_{A_{2}}\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))\mathcal{Q}_{m}(u-s,z-\theta^{N}_{s,u}(y))\mathrm{d}z$
$\displaystyle\qquad\leq\int_{A_{2}}\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))\frac{c_{m}(u-s)^{(m-d)/2}}{\|\theta_{s,u}^{N}(y)-z\|^{m}}\mathrm{d}z$
$\displaystyle\qquad\leq\int_{A_{2}}\mathcal{Q}_{m}(s-t,z-\theta^{N}_{s,t}(x))\mathrm{d}z\frac{c_{m}(u-s)^{(m-d)/2}}{C^{m}\|y-\theta_{u,t}^{N}(x)\|^{m}}$
$\displaystyle\qquad\leq
c_{m}(u-t)^{-d/2}C^{-m}2^{m}\left(1+\|y-\theta_{u,t}^{N}(x)\|\right)^{-m}$
$\displaystyle\qquad\leq\mathcal{Q}_{m}(u-t,y-\theta^{N}_{u,t}(x)).$

The same bound can be shown for integrals over the set $A_{1}$. This completes
the proof of (43) for case II.

It remains to show (42). For a proof of this claim note that

$\displaystyle
K_{N}(t_{i}^{N},t_{j}^{N},z,y)=(L_{t_{i}^{N}}^{N}-\tilde{L}_{t_{i}^{N}}^{N})\tilde{p}_{N}(t_{i}^{N},t_{j}^{N},z,y)$
(48)
$\displaystyle\qquad=\left(\sum_{i=1}^{d}\left(\sum_{j=1}^{d}[F_{N}(t_{i}^{N},z)]_{i,j}[\chi_{N}(z)]_{j}\right)-\sum_{i=1}^{d}\left(\sum_{j=1}^{d}[F_{N}(t_{i}^{N},\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{i,j}[\chi_{N}(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{j}\right)\right)$
$\displaystyle\qquad\qquad\times\left(\frac{\partial}{\partial
z_{i}}(\tilde{p}_{N}-\tilde{q}_{N})+\frac{\partial}{\partial
z_{i}}\tilde{q}_{N}\right)(t_{i}^{N},t_{j}^{N},z,y)$ $\displaystyle\qquad\leq
C\left(\frac{\sqrt{\gamma_{1}^{N}}}{\sqrt{t_{j}^{N}-t_{i}^{N}}}+(t_{j}^{N}-t_{i}^{N})a_{N}\sqrt{\gamma_{1}^{N}}\right)\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-z)$
$\displaystyle\qquad\qquad+C\bar{q}_{N}(t_{i}^{N},t_{j}^{N},z,y)$
$\displaystyle\qquad\leq
C\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-z),$

where again Lemma 3.2 has been used. With the help of (43) this gives that

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}K_{N})(t_{i}^{N},t_{j}^{N},x,z)\right|\leq\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}|\tilde{p}_{N}(t_{i}^{N},t_{k}^{N},x,v)||K_{N}(t_{k}^{N},t_{j}^{N},v,z)|\mathrm{d}v$
$\displaystyle\qquad\leq
C\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-2}(t_{k}^{N}-t_{i}^{N},v-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{k}^{N},\theta^{N}_{t_{k}^{N},t_{j}^{N}}(z)-v)\mathrm{d}v$
$\displaystyle\qquad\leq
C\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))\sum_{k=i}^{j-1}\gamma_{k+1}^{N}$
$\displaystyle\qquad\leq
C\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))(t_{j}^{N}-t_{i}^{N}).$

Similarly we get that

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}K_{N}^{(2)})(t_{i}^{N},t_{j}^{N},x,z)\right|\leq\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}|(\tilde{p}_{N}\otimes_{N}K_{N})(t_{i}^{N},t_{k}^{N},x,v)||K_{N}(t_{k}^{N},t_{j}^{N},v,z)|\mathrm{d}v$
$\displaystyle\qquad\leq
C^{2}\sum_{k=i}^{j-1}\gamma_{k+1}^{N}(t_{k}^{N}-t_{i}^{N})\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-2}(t_{k}^{N}-t_{i}^{N},v-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\qquad\times\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{k}^{N},\theta^{N}_{t_{k}^{N},t_{j}^{N}}(z)-v)\mathrm{d}v$
$\displaystyle\qquad\leq\frac{(C(t_{j}^{N}-t_{i}^{N}))^{2}}{2!}\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)),$

where $C$ is the same constant as in the last inequality. By induction we
conclude that

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}K_{N}^{(r+1)})(t_{i}^{N},t_{j}^{N},x,z)\right|\leq\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}|(\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})(t_{i}^{N},t_{k}^{N},x,v)||K_{N}(t_{k}^{N},t_{j}^{N},v,z)|\mathrm{d}v$
$\displaystyle\qquad\leq\frac{C^{r+1}}{r!}\sum_{k=i}^{j-1}\gamma_{k+1}^{N}(t_{k}^{N}-t_{i}^{N})^{r}\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-2}(t_{k}^{N}-t_{i}^{N},v-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\qquad\times\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{k}^{N},\theta^{N}_{t_{k}^{N},t_{j}^{N}}(z)-v)\mathrm{d}v$
$\displaystyle\qquad\leq\frac{C^{r+1}}{r!}\int_{0}^{t_{j}^{N}-t_{i}^{N}}u^{r}\mathrm{d}u\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\leq\frac{(C(t_{j}^{N}-t_{i}^{N}))^{r+1}}{(r+1)!}\mathcal{Q}_{M-d-2}(t_{j}^{N}-t_{i}^{N},z-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)),$

which shows (42) and concludes the proof of Lemma 2.9. $\square$

###  3.4 Proof of Proposition 2.11

Using a telescopic sum, we have with putting $x_{0}=x$, $x_{m_{N}}=y$,
$\prod_{k=1}^{0}...=1$, and $\prod_{k=l+1}^{l}...=1$

$\displaystyle\int\left|\frac{\mathrm{d}Q_{N,x}^{m_{N}}}{\mathrm{d}\nu}-\frac{\mathrm{d}P_{N,x}^{m_{N}}}{\mathrm{d}\nu}\right|\mathrm{d}\nu$
(49)
$\displaystyle=\int_{\mathbb{R}^{d(m_{N}-1)}}\left|\prod_{i=1}^{m_{N}}q_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})-\prod_{i=1}^{m_{N}}p_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})\right|\mathrm{d}x_{1}...\mathrm{d}x_{m_{N}}$
$\displaystyle\leq\sum_{i=1}^{m_{N}}\int_{\mathbb{R}^{d(m_{N}-1)}}\left|q_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})-p_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})\right|$
$\displaystyle\qquad\times\prod_{k=1}^{i-1}q_{N}(\tau^{N}_{k-1},\tau^{N}_{k},x_{k-1},x_{k})\prod_{l=i+1}^{m_{N}}p_{N}(\tau^{N}_{l-1},\tau^{N}_{l},x_{l-1},x_{l})\mathrm{d}x_{1}...\mathrm{d}x_{m_{N}}$
$\displaystyle\leq\sum_{i=1}^{m_{N}}\int_{\mathbb{R}^{d(m_{N}-1)}}\left|q_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})-p_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})\right|$
$\displaystyle\qquad\times
q_{N}(0,\tau^{N}_{i-1},x,x_{i-1})p_{N}(\tau^{N}_{i},T,x_{i},y)\mathrm{d}x_{i-1}\
\mathrm{d}x_{i}\ \mathrm{d}y$
$\displaystyle=\sum_{i=1}^{m_{N}}\int_{\mathbb{R}^{d(m_{N}-1)}}\left|q_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})-p_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})\right|$
(50) $\displaystyle\qquad\times
q_{N}(0,\tau^{N}_{i-1},x,x_{i-1})\mathrm{d}x_{i-1}\ \mathrm{d}x_{i}.$

Now by Lemma 3.5 we have that

$\displaystyle q_{N}(0,\tau^{N}_{i-1},x,x_{i-1})$ $\displaystyle\leq$
$\displaystyle
C(\tau^{N}_{i-1})^{-d/2}\exp(-C(x_{i-1}-\theta_{\tau^{N}_{i-1},0}^{N}(x))^{2}/\tau^{N}_{i-1})$
$\displaystyle\leq$ $\displaystyle
C\mathcal{Q}_{M-d-6}(\tau^{N}_{i-1},x_{i-1}-\theta_{\tau^{N}_{i-1},0}^{N}(x)).$

Furthermore, we have by Theorem 2.2 and (15)

$\displaystyle\left|q_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})-p_{N}(\tau^{N}_{i-1},\tau^{N}_{i},x_{i-1},x_{i})\right|$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln^{2}(1/\gamma_{1}^{N})\mathcal{Q}_{M-d-6}(\tau^{N}_{i}-\tau^{N}_{i-1},x_{i}-\theta_{\tau^{N}_{i},\tau^{N}_{i-1}}^{N}(x_{i-1}))$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln^{2}(1/\gamma_{1}^{N})\mathcal{Q}_{M-d-6}(\tau^{N}_{i}-\tau^{N}_{i-1},x_{i}-\theta_{\tau^{N}_{i},\tau^{N}_{i-1}}^{N}(x_{i-1})).$

Now the statement of the proposition follows by application of (43). $\square$

## References

  * [1] A. Benveniste, M. Métivier, and P. Priouret.  Adaptive algorithms and stochastic approximations, volume 22 of Applications of Mathematics (New York).  Springer-Verlag, Berlin, 1990.  Translated from French by Stephen S. Wilson.
  * [2] R.N. Bhattacharya and R.R. Rao.  Normal approximation and asymptotic expansions.  SIAM, 1986.
  * [3] I. Bitter and V. Konakov.  L1 and l∞ stability of transition densities of perturbed diffusions.  Random Oper. Stoch. Equ., 29:287–308, 2021.
  * [4] V.I. Bogachev, M. Röckner, and S. Shaposhnikov.  Distances between transition probabilities of diffusions and applications to nonlinear Fokker-Planck-Kolmogorov equations..  J. Funct. Anal., 271(50):1262–1300, 2016.
  * [5] V. Borkar, S. Chen, A. Devraj, I. Kontoyiannis, and S. Meyn.  The ODE method for asymptotic statistics in stochastic approximation and reinforcement learning.  arXiv preprint arXiv:2110.14427, 2021.
  * [6] E. Clément.  Hellinger and total variation distance in approximating Lévy driven SDEs.  arXiv:2103.09648 [math.PR], 2022.
  * [7] Y. A. Davydov.  Local invariance principle I.  Zap. Nauchn. Sem. LOMI, 97:45–50, 1980.  English transl. in J. Soviet Math. 24 (1984), No. 5.
  * [8] Y. A. Davydov.  On the strong convergence of distributions of functionals of stochastic processes I.  Theory of Probability and its Applications, 25:782–799, 1980.
  * [9] Y. A. Davydov.  On the strong convergence of distributions of functionals of stochastic processes II.  Theory of Probability and its Applications, 26:266–286, 1981.
  * [10] Y. A. Davydov.  Local invariance principle II.  Zap. Nauchn. Sem. LOMI, 130:69–77, 1983.  English transl. in J. Soviet Math. 27 (1984), No. 6.
  * [11] F. Delarue and S. Menozzi.  Density estimates for a random noise propagating through a chain of differential equations.  Journal of Functional Analysis, 259(6):1577–1630, 2010.
  * [12] S. Dereich, A. Jentzen, and S. Kassing.  On the existence of minimizers in shallow residual ReLU neural network optimization landscapes.  arXiv:2302.14690v1, 2023.
  * [13] A. Friedman.  Partial differential equations of parabolic type. Prentice-Hall, 1964.
  * [14] A. Jentzen and A. Riekert.  A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions.  Z. Angew. Math. Phys., 73:188, 2022.
  * [15] V. Konakov and E. Mammen.  Local limit theorems for transition densities of Markov chains converging to diffusions.  Prob. Th. Rel. Fields, 117:551–587, 2000.
  * [16] V. Konakov and E. Mammen.  Edgeworth type expansions for Euler schemes for stochastic differential equations.  Monte Carlo Methods Appl., 8(3):271–285, 2002.
  * [17] V. Konakov, S. Menozzi, and S. Molchanov.  Explicit parametrix and local limit theorems for some degenerate diffusion processes.  Annales de l’Institut Henri Poincaré, Série B, 46–4:908–923, 2010.
  * [18] H. J. Kushner and G. G. Yin.  Stochastic approximation and recursive algorithms and applications, volume 35 of Applications of Mathematics (New York).  Springer-Verlag, New York, second edition, 2003.  Stochastic Modelling and Applied Probability.
  * [19] H. P. McKean and I. M. Singer.  Curvature and the eigenvalues of the Laplacian.  J. Differential Geometry, 1:43–69, 1967.
  * [20] S. Menozzi, A. Pesce, and X. Zhang.  Density and gradient estimation for non degenerate Brownian SDEs with unbounded measurable drift.  Journal of Differential Equations, 272:330–369, 2021.
  * [21] S. Meyn.  Control Systems and Reinforcement Learning.  Cambridge University Press,, 2022.
  * [22] E. Moulines and F. Bach.  Non-asymptotic analysis of stochastic approximation algorithms for machine learning.  Advances in Neural Information Processing Systems, 24:451–459, 2011\.
  * [23] Frikha N., Konakov V., and Menozzi S.  Well-posedness of some non- linear stable driven SDEs.  Discrete and Continuous Dynamical Systems, 41(2):849–898, 2021\.
  * [24] M.B. Nevel’son and R.Z. Khas’minskiĭ.  Stochastic approximation and recursive estimation, volume 47.  Amer Mathematical Society, 1973.
  * [25] A. Pascucci and A. Pesce.  The parametrix method for parabolic SPDEs.  Stoch. Proc. Appl., 130(10):6226–6245, 2020.
  * [26] H. Robbins and S. Monro.  A stochastic approximation method.  Ann. Math. Statistics, 22:400–407, 1951.
  * [27] D.W. Stroock and S.R.S. Varadhan.  Multidimensional diffusion processes.  Springer-Verlag Berlin Heidelberg New-York, 1979.

##  Appendix A Supplement to: "Local Limit Propositions and Strong
Approximations for Robbins-Monro Procedures" by V. Konakov and E. Mammen

###  A.1 Proof of Lemma 2.1

Note first that

$\displaystyle U^{N}_{t_{k+1}^{N}}$ $\displaystyle=$
$\displaystyle\frac{\theta_{k+1}^{N}-\bar{\theta}_{t_{k+1}^{N}}}{\sqrt{\gamma^{N}_{k+1}}}$
$\displaystyle=$
$\displaystyle\frac{\theta^{N}_{k}-\gamma_{k+1}^{N}H(\theta_{k}^{N},\eta_{k+1}^{N})-\bar{\theta}_{t_{k+1}^{N}}}{\sqrt{\gamma^{N}_{k+1}}}$
$\displaystyle=$
$\displaystyle\frac{\theta^{N}_{k}-\bar{\theta}_{t_{k}^{N}}}{\sqrt{\gamma^{N}_{k}}}\frac{\sqrt{\gamma^{N}_{k}}}{\sqrt{\gamma^{N}_{k+1}}}-\sqrt{\gamma^{N}_{k+1}}H(\theta_{k}^{N},\eta_{k+1}^{N})-\frac{\bar{\theta}_{t_{k+1}^{N}}-\bar{\theta}_{t_{k}^{N}}}{\sqrt{\gamma^{N}_{k+1}}}$
$\displaystyle=$ $\displaystyle
U_{t_{k}}^{N}\frac{\sqrt{\gamma^{N}_{k}}}{\sqrt{\gamma^{N}_{k+1}}}-\sqrt{\gamma^{N}_{k+1}}H(\bar{\theta}_{t_{k}^{N}}+U_{t_{k}}^{N}\sqrt{\gamma^{N}_{k}},\eta_{k+1}^{N})-\frac{\bar{\theta}_{t_{k+1}^{N}}-\bar{\theta}_{t_{k}^{N}}}{\sqrt{\gamma^{N}_{k+1}}}.$

Now, we can write:

$\displaystyle
U_{t_{k}}^{N}\frac{\sqrt{\gamma^{N}_{k}}}{\sqrt{\gamma^{N}_{k+1}}}$
$\displaystyle=$ $\displaystyle
U_{t_{k}}^{N}+U_{t_{k}}^{N}\frac{\sqrt{\gamma^{N}_{k}}-\sqrt{\gamma^{N}_{k+1}}}{\sqrt{\gamma^{N}_{k+1}}}$
$\displaystyle=$ $\displaystyle
U_{t_{k}}^{N}+\alpha_{t_{k}}^{N}U_{t_{k}}^{N}\gamma^{N}_{k+1}.$

The dynamics of $U_{t_{k}}^{N}$ becomes:

$\displaystyle U^{N}_{t_{k+1}^{N}}$ $\displaystyle=$ $\displaystyle
U_{t_{k}}^{N}+\alpha_{t_{k}}^{N}U_{t_{k}}^{N}\gamma^{N}_{k+1}-\sqrt{\gamma^{N}_{k+1}}\Big{(}H(\bar{\theta}_{t_{k}^{N}}+U_{t_{k}}^{N}\sqrt{\gamma^{N}_{k}},\eta_{k+1}^{N})-h(\bar{\theta}_{t_{k}^{N}}+U_{t_{k}}^{N}\sqrt{\gamma^{N}_{k}})\Big{)}$
$\displaystyle-\sqrt{\gamma^{N}_{k+1}}\Big{(}h(\bar{\theta}_{t_{k}^{N}}+U_{t_{k}}^{N}\sqrt{\gamma^{N}_{k}})-h(\bar{\theta}_{t_{k}^{N}})\Big{)}+\sqrt{\gamma^{N}_{k+1}}\left(-h(\bar{\theta}_{t_{k}^{N}})-\frac{\bar{\theta}_{t_{k+1}^{N}}-\bar{\theta}_{t_{k}^{N}}}{\gamma^{N}_{k+1}}\right)$
$\displaystyle=$ $\displaystyle
U_{t_{k}}^{N}+G_{N}(t_{k}^{N},U_{t_{k}}^{N})U_{t_{k}}^{N}\gamma^{N}_{k+1}-\sqrt{\gamma^{N}_{k+1}}\xi(\bar{\theta}_{t_{k}^{N}}+U_{t_{k}}^{N}\sqrt{\gamma^{N}_{k}},\eta_{k+1}^{N})+\beta_{k+1}^{N},$

which shows the statement of the lemma. $\square$

###  A.2 Proof of Lemma 2.2

Here we need that $\|\theta_{0}-\theta^{*}\|$ is small enough. And that for
all eigenvalues of $\mathcal{D}h(\theta^{*})$ the real part is strictly
negative. Then by Lyapunov’s proposition we have that
$\bar{\theta}_{t}\to\theta^{*}$ for $t\to\infty$ and thus
$\|h(\bar{\theta}_{t})\|\leq C$ and $\|\mathcal{D}h(\bar{\theta}_{t})\|\leq C$
for some $C>0$ for all $t$. This gives

$\displaystyle\|\beta_{k+1}^{N}\|$ $\displaystyle\leq$
$\displaystyle\sqrt{\gamma^{N}_{k+1}}\left\|-h(\bar{\theta}_{t^{N}_{k}})-\frac{\bar{\theta}_{t^{N}_{k+1}}-\bar{\theta}_{t^{N}_{k}}}{\gamma^{N}_{k+1}}\right\|$
$\displaystyle=$
$\displaystyle\frac{\sqrt{\gamma^{N}_{k+1}}}{t^{N}_{k+1}-t^{N}_{k}}\left\|\int_{t^{N}_{k}}^{t^{N}_{k+1}}\left(-h(\bar{\theta}_{t^{N}_{k}})+h(\bar{\theta}_{t})\right)\mathrm{d}t\right\|$
$\displaystyle=$
$\displaystyle\frac{\sqrt{\gamma^{N}_{k+1}}}{t^{N}_{k+1}-t^{N}_{k}}\left\|\int_{t^{N}_{k}}^{t^{N}_{k+1}}\int_{0}^{1}\mathcal{D}h(\bar{\theta}_{t^{N}_{k}}+\delta(\bar{\theta}_{t}-\bar{\theta}_{t^{N}_{k}}))\left(\bar{\theta}_{t^{N}_{k}}-\bar{\theta}_{t}\right)\mathrm{d}\delta\mathrm{d}t\right\|$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma^{N}_{k+1}}}{t^{N}_{k+1}-t^{N}_{k}}\int_{t^{N}_{k}}^{t^{N}_{k+1}}\left\|\bar{\theta}_{t^{N}_{k}}-\bar{\theta}_{t}\right\|\mathrm{d}t$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma^{N}_{k+1}}}{t^{N}_{k+1}-t^{N}_{k}}\int_{t^{N}_{k}}^{t^{N}_{k+1}}\left\|\int_{0}^{1}h(\bar{\theta}_{t^{N}_{k}+\delta(t-t^{N}_{k})})(t-t^{N}_{k})\mathrm{d}\delta\right\|\mathrm{d}t$
$\displaystyle\leq$ $\displaystyle
C^{2}{\sqrt{\gamma^{N}_{k+1}}}(t^{N}_{k+1}-t^{N}_{k})$ $\displaystyle=$
$\displaystyle C^{2}(\gamma^{N}_{k+1})^{3/2}.$

Now $\gamma^{N}_{k+1}$ converges to 0 under our assumptions. This implies the
statement of the lemma. $\square$

###  A.3 Proofs of Propositions 2.5 and 2.6

###### Proof of Proposition 2.5.

We start by showing that

$\displaystyle\left(\int_{{\mathbb{R}}^{d}}\left|q_{N}-q\right|(s,t,x,z)\mathrm{d}z\right)^{2}$
(51) $\displaystyle\qquad\leq
C\int_{s}^{t}\int_{{\mathbb{R}}^{d}}\left|R^{-1/2}(\bar{\theta}_{u})\left([\bar{\alpha}I+\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y)\right)\right|^{2}q(s,u,x,y)\mathrm{d}y\mathrm{d}u.$

This claim follows from Corollary 1.2 in [4]. We have to check the assumptions
of the corollary. First, for both diffusions $X_{t}$ and $X_{t}^{N}$, the
drift coefficients have Borel measurable locally bounded entries, i.e., for
every ball $U\subset{\mathbb{R}}^{d}$, there is a number $B=B(U)$ such that

$\displaystyle\left|\left[\bar{\alpha}I+\mathcal{D}h(\bar{\theta}_{t})\right]x\right|\leq
B(U)\text{ for all }x\in U,t\in[0,T],$
$\displaystyle\left|F_{N}(t,x)\chi_{N}(x)\right|\leq B(U)\text{ for all }x\in
U,t\in[0,T],$

and $R^{1/2}(\bar{\theta}_{t})$ is locally strictly positive, i.e, for every
ball $U\subset{\mathbb{R}}^{d}$, there exist $0<c_{R}(U)<C_{R}(U)$ with
$c_{R}(U)I\leq R^{1/2}(\bar{\theta}_{t})\leq C_{R}(U)I$ for all $x\in
U,t\in[0,T]$. Furthermore, we have to check that
$R^{-1/2}(\bar{\theta}_{u})([\bar{\alpha}I+\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y))$
is square integrable with respect to the measure $q(s,u,x,$ $y)$ $\mathrm{d}y\
\mathrm{d}u$ on ${\mathbb{R}}^{d}\times[s,t]$ and, choosing Assumption (a) in
Theorem 1.1 in [4], that $(1+|y|)^{-2}|R^{i,j}(\bar{\theta}_{u})|$,
$(1+|y|)^{-1}|F_{N}(u,y)\chi_{N}(y)|$ and
$(1+|y|)^{-1}|[\bar{\alpha}I+\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y)|$
are integrable, again with respect to the measure $q(s,u,x,y)\mathrm{d}y\
\mathrm{d}u$ on ${\mathbb{R}}^{d}\times[s,t]$. All these assumptions can be
easily verified using in particular the fast decay of Gaussian densities.

We now argue that for $u\in[t_{k}^{N},t_{k+1}^{N})$, $|y|<a_{N}$

$\displaystyle\left|[\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\right|$
(52) $\displaystyle\qquad\leq
C\left(\left|\bar{\alpha}-\alpha_{t_{k}^{N}}^{N}\right|+\sqrt{\gamma_{k}^{N}/\gamma_{k+1}^{N}}-1+\left|\bar{\theta}_{u}-\bar{\theta}_{t_{k}^{N}}\right|+a_{N}\sqrt{\gamma_{k}^{N}}\right).$

For a proof of this bound note that for $u\in[t_{k}^{N},t_{k+1}^{N})$,
$|y|<a_{N}$ the function $F_{N}(u,y)$ can be bounded as follows:

$\displaystyle\left|\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\right|$
$\displaystyle\qquad\leq|\bar{\alpha}-\alpha_{t_{k}^{N}}^{N})|+\left|\left(\sqrt{\gamma_{k}^{N}/\gamma_{k+1}^{N}}-1\right)\int_{0}^{1}\mathcal{D}h(\bar{\theta}_{t_{k}^{N}}+\delta\sqrt{\gamma_{k}^{N}}y)\mathrm{d}\delta\right|,$

which can be used to show (52). Finally, we argue that

$\displaystyle\sqrt{\gamma_{k}^{N}/\gamma_{k+1}^{N}}-1\leq
C(\gamma_{1}^{N})^{\beta^{-1}},$
$\displaystyle|\bar{\alpha}-\alpha_{t_{k+1}^{N}}^{N}|\leq
CI_{\\{\beta=1\\}}\gamma_{1}^{N}+CI_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{\beta^{-1}-1},$
$\displaystyle\left|\bar{\theta}_{u}-\bar{\theta}_{t_{k}^{N}}\right|\leq
C\gamma_{1}^{N},$

which can be easily shown and implies with (52) that for $|y|<a_{N}$

$\displaystyle\left|[\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\right|$
(53) $\displaystyle\qquad\leq
C\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{\beta^{-1}-1}+a_{N}\sqrt{\gamma_{1}^{N}}\right).$

Thus it holds that

$\displaystyle\int_{s}^{t}\int_{|y|<a_{N}}\left|R^{-1/2}(\bar{\theta}_{u})\left([\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y)\right)\right|^{2}q(s,u,x,y)\mathrm{d}y\mathrm{d}u$
(54) $\displaystyle\qquad\leq
C(t-s)\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{2\beta^{-1}-2}+a_{N}^{2}\gamma_{1}^{N}\right).$

For $|y|\geq a_{N}$ we have that

$\left|[\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y)\right|\leq
C|y|.$

This implies for $|x|\leq a_{N}/2$ that

$\displaystyle\int_{s}^{t}\int_{|y|\geq
a_{N}}\left|R^{-1/2}(\bar{\theta}_{u})\left([\bar{\alpha}I-\mathcal{D}h(\bar{\theta}_{u})]y-F_{N}(u,y)\chi_{N}(y)\right)\right|^{2}q(s,u,x,y)\mathrm{d}y\mathrm{d}u$
(55) $\displaystyle\qquad\leq C\int_{s}^{t}\int_{|y|\geq
a_{N}}|y|^{2}q(s,u,x,y)\mathrm{d}y\mathrm{d}u$ $\displaystyle\qquad\leq
C\int_{s}^{t}\int_{|y|\geq
a_{N}}|y|^{2}u^{-d/2}\exp\left(-C\frac{|x-\theta_{0,u}^{N}(y)|^{2}}{u}\right)\mathrm{d}y\mathrm{d}u$
$\displaystyle\qquad\leq C\int_{s}^{t}\int_{|y|\geq
a_{N}}|y|^{2}u^{-d/2}\exp\left(-C\frac{|x-y|^{2}}{u}\right)\mathrm{d}y\mathrm{d}u$
$\displaystyle\qquad\leq C(t-s)\int_{|y|\geq
Ca_{N}}|y|^{2}\exp\left(-C|y|^{2}\right)\mathrm{d}y$ $\displaystyle\qquad\leq
C(t-s)\exp(-Ca_{N}^{2})$ $\displaystyle\qquad\leq
C(t-s)(\gamma_{1}^{N})^{\rho}$

for all $\rho>0$. The second inequality of Proposition 2.5 now follows from
(51), (54) and (55). The first inequality follows by using the bound

$H^{2}(P,Q)\leq\int\left|\frac{\mathrm{d}P}{\mathrm{d}\nu}-\frac{\mathrm{d}Q}{\mathrm{d}\nu}\right|\mathrm{d}\nu$
(56)

for the Hellinger distance between two measures $P$ and $Q$ with dominating
measure $\nu$. The last inequality follows by definition of $\gamma_{1}^{N}$.
∎

###### Proof of Proposition 2.6.

The first inequality follows by application of the bound

$\int\left|\frac{\mathrm{d}P}{\mathrm{d}\nu}-\frac{\mathrm{d}Q}{\mathrm{d}\nu}\right|\mathrm{d}\nu\leq
2H(P,Q)$

for two measures $P$ and $Q$ with dominating measure $\nu$. The third
inequality follows as in the proof of Proposition 2.5 by definition of
$\gamma_{1}^{N}$.

For the proof of the second inequality we apply Proposition 2.1 in [6]. This
proposition is stated for one-dimensional homogeneous Markov chains but it can
be easily generalized to multi-dimensional non-homogeneous Markov chains.
Stated for the measures $Q_{x}^{m_{N}}$ and $Q_{N,x}^{m_{N}}$ the proposition
says that

$\displaystyle
H^{2}(Q_{x}^{m_{N}},Q_{N,x}^{m_{N}})\leq\frac{1}{2}\sum_{j=1}^{m_{N}}\left(\mathbb{E}[H^{2}_{X_{t_{j-1}},j}]+\mathbb{E}[H^{2}_{X_{t_{j-1}}^{N},j}]\right),$

where for $x\in{\mathbb{R}}^{d}$

$\displaystyle
H^{2}_{x,j}=\int\left(\sqrt{q(t_{j-1},t_{j},x,z)}-\sqrt{q_{N}(t_{j-1},t_{j},x,z)}\right)^{2}\mathrm{d}z.$

By application of (56) we get that

$\displaystyle H^{2}(Q_{x}^{m_{N}},Q_{N,x}^{m_{N}})$ $\displaystyle\leq$
$\displaystyle\frac{1}{2}\sum_{j=1}^{m_{N}}\left(\mathbb{E}\left[\int\left|{q(t_{j-1},t_{j},X_{t_{j-1}},z)}-{q_{N}(t_{j-1},t_{j},X_{t_{j-1}},z)}\right|\mathrm{d}z\right]\right.$
$\displaystyle\qquad+\left.\mathbb{E}\left[\int\left|{q(t_{j-1},t_{j},X_{t_{j-1}}^{N},z)}-{q_{N}(t_{j-1},t_{j},X_{t_{j-1}}^{N},z)}\right|\mathrm{d}z\right]\right).$

By application of Proposition 2.5 we get that

$\displaystyle\mathbb{E}\left[\int\left|{q(t_{j-1},t_{j},X_{t_{j-1}},z)}-{q_{N}(t_{j-1},t_{j},X_{t_{j-1}},z)}\right|\mathrm{d}z\right]$
$\displaystyle\qquad\leq
Cm_{N}^{-1/2}\left(I_{\\{\frac{1}{2}<\beta<1\\}}(\gamma_{1}^{N})^{\beta^{-1}-1}+\ln(1/\gamma_{1}^{N})(\gamma_{1}^{N})^{1/2}\right)+\mathbb{P}[|X_{t_{j-1}}|>a_{N}/2].$

The same inequality applies with $X_{t_{j-1}}$ replaced by $X_{t_{j-1}}^{N}$.
Thus for the proof of the proposition it suffices to show that:

$\displaystyle\mathbb{P}[|X_{t_{j-1}}|>a_{N}/2]+\mathbb{P}[|X_{t_{j-1}}^{N}|>a_{N}/2]\leq
Cm_{N}^{-1/2}\ln(1/\gamma_{1}^{N})(\gamma_{1}^{N})^{1/2}.$ (57)

Now, by application of Gaussian bounds for the transition densities of $X_{t}$
and $X_{t}^{N}$, stated in [11], we get that

$\displaystyle\mathbb{P}[|X_{t_{j-1}}|>a_{N}/2]+\mathbb{P}[|X_{t_{j-1}}^{N}|>a_{N}/2]\leq
C\exp(-Ca_{N}^{2}).$

Furthermore, note that we can assume that $m_{N}\leq CN^{\mu}$ for some
$\mu>0$. Otherwise the second inequality of the proposition holds trivially
because the Hellinger distance is bounded and the right hand side of the
inequality would converge to infinity. But for the case that $m_{N}\leq
CN^{\mu}$ for some $\mu>0$ one can easily verify that (57) holds. This
completes the proof of the proposition. ∎

###  A.4 Proofs of Lemmas 2.7, 2.8, and 2.10 from Subsection 2.6

###### Proof of Lemma 2.7.

With
$q_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)=\sum_{r=0}^{\infty}\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)$
we have to show that

$\displaystyle\left|(q_{N}-q_{N}^{discr})(t_{i}^{N},t_{j}^{N},x,y)\right|\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$
(58)

For a proof of this claim we write

$\displaystyle(\tilde{q}_{N}\otimes
H_{N}^{(r)}-\tilde{q}_{N}\otimes_{N}H_{N}^{(r)})(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad=\left[\left(\tilde{q}_{N}\otimes
H_{N}^{(r-1)}\right)\otimes H_{N}-\left(\tilde{q}_{N}\otimes
H_{N}^{(r-1)}\right)\otimes_{N}H_{N}\right](t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad\qquad+\left[\left(\tilde{q}_{N}\otimes
H_{N}^{(r-1)}\right)-\left(\tilde{q}_{N}\otimes_{N}H_{N}^{(r-1)}\right)\right]\otimes_{N}H_{N}(t_{i}^{N},t_{j}^{N},x,y).$

By summing this up from $r=1$ to $r=\infty$ we get by using linearity of the
operations $\otimes$ and $\otimes_{N}$

$\displaystyle\left(q_{N}-q_{N}^{discr}\right)(t_{i}^{N},t_{j}^{N},x,y)=\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad+\left(q_{N}-q_{N}^{discr}\right)\otimes_{N}H_{N}(t_{i}^{N},t_{j}^{N},x,y).$

Iterative application of this equation gives

$\displaystyle\left(q_{N}-q_{N}^{discr}\right)(t_{i}^{N},t_{j}^{N},x,y)=\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad+\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)\otimes_{N}\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y),$
(59)

where $\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)=\sum_{k\geq
1}H_{N}^{(k),\otimes_{N}}(t_{i}^{N},t_{j}^{N},x,y)$. Here
$H_{N}^{(k),\otimes_{N}}$ is similarly defined as $H_{N}^{(k)}$ as a $k$-times
convolution of $H_{N}$ but now with using the convolution operator
$\otimes_{N}$ instead of $\otimes$. Furthermore, instead of $H_{N}$ we use the
convolution of
$H_{N}(t_{i}^{N},t_{j}^{N},x,y)\mathbb{I}_{t_{j}^{N}>t_{i}^{N}}$. For the
latter change note that $\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)$ with
$t_{i}^{N}=t_{j}^{N}$ is not used in (59) because
$\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)$ only appears on the right hand
side of the convolution operator $\otimes_{N}$. Note that for this reason it
holds for functions $f$ that
$f\otimes_{N}\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)=((f\otimes_{N}H_{N})+(f\otimes_{N}H_{N})\otimes_{N}H_{N}+((f\otimes_{N}H_{N})\otimes_{N}H_{N})\otimes_{N}H_{N}+...)(t_{i}^{N},t_{j}^{N},x,y)$.

For the statement of the lemma it suffices to show that for some constant
$C>0$

$\displaystyle\left|\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)(t_{i}^{N},t_{j}^{N},x,y)\right|$ (60)
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}(1+|y|)\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y),$
$\displaystyle\left|\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)\otimes_{N}\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)\right|$
(61) $\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}(t_{j}^{N}-t_{i}^{N})(1+|y|)\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$

Before we come to the proof of these claims we first argue that the following
bound for $\Phi_{N}^{discr}$ applies

$\displaystyle|\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)|$ $\displaystyle\leq$
$\displaystyle
C(t_{j}^{N}-t_{i}^{N})^{-1/2}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$ (62)

This bound follows from

$\displaystyle|H_{N}^{(r),\otimes_{N}}(t_{i}^{N},t_{j}^{N},x,y)|$
$\displaystyle\leq$ $\displaystyle
C^{r}\frac{\Gamma^{r}(1/2)}{\Gamma(r/2)}(t_{j}^{N}-t_{i}^{N})^{\frac{r}{2}-1}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$
(63)

For a proof of (63) we remark that for the function
$f(u)=(u-t_{i}^{N})^{\frac{r}{2}-1}(t_{j}^{N}-u)^{-1/2}$ for $r\geq 2$ the
following estimate holds because $f$ is monotonically increasing on
$[t_{i}^{N},t_{j}^{N}]$

$\displaystyle\sum_{k=i}^{j-1}\gamma_{k+1}^{N}f(t_{k}^{N})$
$\displaystyle\leq$
$\displaystyle\int_{t_{i}^{N}}^{t_{j}^{N}}f(u)\mathrm{d}u.$

Using this bound we can carry over the arguments of the proof of (36) to get
(63).

We now show that (60) and (62) imply (61). By application of the first two
inequalities we get

$\displaystyle\left|\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)\otimes_{N}\Phi_{N}^{discr}(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}\left|\left(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N}\right)(t_{i}^{N},t_{k}^{N},x,z)\right|\
\left|\Phi_{N}^{discr}(t_{k}^{N},t_{j}^{N},z,y)\right|\mathrm{d}z$
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\frac{\sqrt{t_{k}^{N}-t_{i}^{N}}}{\sqrt{t_{j}^{N}-t_{k}^{N}}}\int_{{\mathbb{R}}^{d}}(1+|z|)\bar{q}_{N}(t_{i}^{N},t_{k}^{N},x,z)\bar{q}_{N}(t_{k}^{N},t_{j}^{N},z,y)\mathrm{d}z$
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\frac{\sqrt{t_{k}^{N}-t_{i}^{N}}}{\sqrt{t_{j}^{N}-t_{k}^{N}}}\int_{{\mathbb{R}}^{d}}\left(1+|y|\right)\bar{q}_{N}(t_{i}^{N},t_{k}^{N},x,z)\bar{q}_{N}(t_{k}^{N},t_{j}^{N},z,y)\mathrm{d}z$
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}(t_{j}^{N}-t_{i}^{N})\left(1+|y|\right)\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y)\int_{0}^{1}\frac{\sqrt{u}}{\sqrt{1-u}}\mathrm{d}u$
$\displaystyle\qquad\leq
C\ln^{2}(1/\gamma_{1}^{N})\sqrt{\gamma_{1}^{N}}(t_{j}^{N}-t_{i}^{N})\left(1+|y|\right)\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$

This shows (61). For the proof of the lemma it remains to show (60). For a
proof of this claim we introduce

$\lambda_{u}(z)=q_{N}(t_{i}^{N},u,x,z)H_{N}(u,t_{j}^{N},z,y).$

Using a Taylor expansion of second order we get

$\displaystyle(q_{N}\otimes
H_{N}-q_{N}\otimes_{N}H_{N})(t_{i}^{N},t_{j}^{N},x,y)=\sum_{k=i}^{j-1}\int_{t_{k}^{N}}^{t_{k+1}^{N}}\mathrm{d}u\int_{{\mathbb{R}}^{d}}\left(\lambda_{u}(z)-\lambda_{t_{k}^{N}}(z)\right)\mathrm{d}z$
$\displaystyle\qquad=\sum_{k=i}^{j-1}\int_{t_{k}^{N}}^{t_{k+1}^{N}}(u-t_{k}^{N})\int_{0}^{1}\int_{{\mathbb{R}}^{d}}\frac{\partial}{\partial
s}\lambda_{s}(z)\bigg{|}_{s=s_{k}}\mathrm{d}z\ \mathrm{d}\delta\ \mathrm{d}u$
(64)

with $s_{k}=t_{k}^{N}+\delta(u-t_{k}^{N})$. Next by application of forward and
backward Kolmogorov equations we get that

$\displaystyle\int_{{\mathbb{R}}^{d}}\frac{\partial}{\partial
s}\lambda_{s}(z)\bigg{|}_{s=s_{k}}\mathrm{d}z=\int_{{\mathbb{R}}^{d}}\frac{\partial}{\partial
s}\left(q_{N}(t_{i}^{N},s,x,z)H_{N}(s,t_{j}^{N},z,y)\right)\bigg{|}_{s=s_{k}}\mathrm{d}z$
$\displaystyle\qquad=\int_{{\mathbb{R}}^{d}}\left((L_{s}^{N})^{*}q_{N}(t_{i}^{N},s,x,z)(L_{s}^{N}-\tilde{L}_{s}^{N})\tilde{q}_{N}(s,t_{j}^{N},z,y)\right)\bigg{|}_{s=s_{k}}\mathrm{d}z$
$\displaystyle\qquad\qquad-\int_{{\mathbb{R}}^{d}}\left(q_{N}(t_{i}^{N},s,x,z)(L_{s}^{N}-\tilde{L}_{s}^{N})\tilde{L}_{s}^{N}\tilde{q}_{N}(s,t_{j}^{N},z,y)\right)\bigg{|}_{s=s_{k}}\mathrm{d}z$
$\displaystyle\qquad\qquad+\int_{{\mathbb{R}}^{d}}q_{N}(t_{i}^{N},s,x,z)\sum_{h=1}^{d}\bigg{[}\frac{\partial}{\partial
s}\sum_{l=1}^{d}\bigg{(}[F_{N}(s,z)]_{h,l}[\chi_{N}(z)]_{l}$
$\displaystyle\qquad\qquad\qquad-[F_{N}(s,\theta^{N}_{s,t_{j}^{N}}(y))]_{h,l}[\chi_{N}(\theta^{N}_{s,t_{j}^{N}}(y)]_{l}\bigg{)}\bigg{|}_{s=s_{k}}\frac{\partial\tilde{q}_{N}(s_{k},t_{j}^{N},z,y)}{\partial
z_{h}}\bigg{]}\mathrm{d}z$ $\displaystyle\qquad=I+II$

with

$\displaystyle I$ $\displaystyle=$
$\displaystyle\int_{{\mathbb{R}}^{d}}q_{N}(t_{i}^{N},s,x,z)\big{(}(L_{s}^{N})^{2}-2L_{s}^{N}\tilde{L}_{s}^{N}+(\tilde{L}_{s}^{N})^{2}\big{)}\tilde{q}_{N}(s,t_{j}^{N},z,y)\bigg{|}_{s=s_{k}}\mathrm{d}z,$
$\displaystyle II$ $\displaystyle=$
$\displaystyle\int_{{\mathbb{R}}^{d}}q_{N}(t_{i}^{N},s_{k},x,z)\sum_{r=1}^{d}\bigg{[}\frac{\partial}{\partial
s}\sum_{l=1}^{d}\bigg{(}[F_{N}(s,z)]_{r,l}[\chi_{N}(z)]_{l}$
$\displaystyle\qquad\qquad\qquad-[F_{N}(s,\theta^{N}_{s,t_{j}^{N}}(y))]_{r,l}[\chi_{N}(\theta^{N}_{s,t_{j}^{N}}(y)]_{l}\bigg{)}\bigg{|}_{s=s_{k}}\frac{\partial\tilde{q}_{N}(s_{k},t_{j}^{N},z,y)}{\partial
z_{r}}\bigg{]}\mathrm{d}z.$

We will show below that

$\displaystyle|I|$ $\displaystyle\leq$ $\displaystyle
C\frac{\ln^{2}(1/\gamma_{1}^{k})}{\sqrt{s_{k}-t_{i}^{N}}\sqrt{t_{j}^{N}-s_{k}}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y),$
(65) $\displaystyle|II|$ $\displaystyle\leq$ $\displaystyle
C\frac{\ln(1/\gamma_{1}^{k})}{\sqrt{t_{j}^{N}-s_{k}}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$
(66)

Before we will come to the proof of (65) and (66) we now show that these
inequalities can be used to prove the lemma. Note first that from (65), (66)
we get that

$\left|\int_{{\mathbb{R}}^{d}}\frac{\partial}{\partial
s}\lambda_{s}(z)\bigg{|}_{s=s_{k}}\mathrm{d}z\right|\leq
C\frac{\ln^{2}(1/\gamma_{1}^{k})}{\sqrt{s_{k}-t_{i}^{N}}\sqrt{t_{j}^{N}-s_{k}}}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y).$

We now use this bound to estimate the summand on the right hand side of (64).
For the statement of the lemma we have to show that for some constant $C>0$

$\displaystyle\sum_{k=i}^{j-1}\int_{t_{k}^{N}}^{t_{k+1}^{N}}\int_{0}^{1}\frac{(u-t_{k}^{N})}{\sqrt{s_{k}-t_{i}^{N}}\sqrt{t_{j}^{N}-s_{k}}}\mathrm{d}\delta\
\mathrm{d}u\leq C\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}.$ (67)

Note that the left hand side of (67) is equal to

$\sum_{k=i}^{j-1}\int_{0}^{1}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}+v(t_{k+1}^{N}-t_{k}^{N})}\sqrt{t_{j}^{N}-t_{k}^{N}-v(t_{k+1}^{N}-t_{k}^{N})}}\mathrm{d}v.$

We now consider the summands of this sum for $i+1\leq k\leq j-2$, $k=i$ and
$k=j-1$. We have

$\displaystyle\sum_{k=i+1}^{j-2}\int_{0}^{1}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}+v(t_{k+1}^{N}-t_{k}^{N})}\sqrt{t_{j}^{N}-t_{k}^{N}-v(t_{k+1}^{N}-t_{k}^{N})}}\mathrm{d}v$
$\displaystyle\qquad\leq\sum_{k=i+1}^{j-2}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}}\sqrt{t_{j}^{N}-t_{k+1}^{N}}}$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\sum_{k=i+1}^{j-2}\frac{t_{k+1}^{N}-t_{k}^{N}}{\sqrt{t_{k}^{N}-t_{i}^{N}}}$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\int_{t_{i}^{N}}^{t_{j}^{N}}\frac{1}{\sqrt{v-t_{i}^{N}}}\mathrm{d}v$
$\displaystyle\qquad\leq C\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}.$

For $k=i<j-1$ we have that

$\displaystyle\int_{0}^{1}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}+v(t_{k+1}^{N}-t_{k}^{N})}\sqrt{t_{j}^{N}-t_{k}^{N}-v(t_{k+1}^{N}-t_{k}^{N})}}\mathrm{d}v$
$\displaystyle\qquad\leq\int_{0}^{1}\frac{1}{\sqrt{v}}\mathrm{d}v\frac{(t_{i+1}^{N}-t_{i}^{N})^{3/2}}{\sqrt{t_{j}^{N}-t_{i+1}^{N}}}$
$\displaystyle\qquad\leq C\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}.$

For $k=j-1>i$ we have that

$\displaystyle\int_{0}^{1}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}+v(t_{k+1}^{N}-t_{k}^{N})}\sqrt{t_{j}^{N}-t_{k}^{N}-v(t_{k+1}^{N}-t_{k}^{N})}}\mathrm{d}v$
$\displaystyle\qquad\leq\int_{0}^{1}\frac{1}{\sqrt{v}}\mathrm{d}v\frac{(t_{j}^{N}-t_{j-1}^{N})^{3/2}}{\sqrt{t_{j-1}^{N}-t_{i}^{N}}}$
$\displaystyle\qquad\leq C\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}.$

Finally, we have for $k=j-1=i$ that

$\displaystyle\int_{0}^{1}\frac{(t_{k+1}^{N}-t_{k}^{N})^{2}}{\sqrt{t_{k}^{N}-t_{i}^{N}+v(t_{k+1}^{N}-t_{k}^{N})}\sqrt{t_{j}^{N}-t_{k}^{N}-v(t_{k+1}^{N}-t_{k}^{N})}}\mathrm{d}v$
$\displaystyle\qquad\leq\int_{0}^{1}\frac{1}{\sqrt{v(1-v)}}\mathrm{d}v(t_{j}^{N}-t_{i}^{N})$
$\displaystyle\qquad\leq C\sqrt{\gamma_{1}^{N}}\sqrt{t_{j}^{N}-t_{i}^{N}}.$

This concludes the proof of (67). For the statement of the lemma it remains to
show (65) and (66).

For a proof of (65) note first that terms with derivatives of second order in
$L_{s}^{N}$ and $\tilde{L}_{s}^{N}$ are equivalent. Hence the terms with
derivatives of forth order in
$(L_{s}^{N})^{2}-2L_{s}^{N}\tilde{L}_{s}^{N}+(\tilde{L}_{s}^{N})^{2}$ cancels.
Thus the operator
$(L_{s}^{N})^{2}-2L_{s}^{N}\tilde{L}_{s}^{N}+(\tilde{L}_{s}^{N})^{2}$ contains
only derivatives up to second order. Because of the truncation we have that
the coefficients of the operator are bounded by $C\ln^{2}(1/\gamma_{1}^{N})$.
The integrand of the integral in I contains terms with first and second order
derivatives of $\tilde{q}_{N}(s_{k},t_{i}^{N},t_{j}^{N},z,y)$ with respect to
$z$. Terms with first order derivatives contain only integrable singularities.
For terms with second order derivatives we make use of partial integration and
get products of first order derivatives of
$\tilde{q}_{N}(s_{k},t_{i}^{N},t_{j}^{N},z,y)$ and of
$q_{N}(t_{i}^{N},s_{k},y,z)$ with respect to $z$. We now use that the norm of
the derivative of $q_{N}(t_{i}^{N},s_{k},y,z)$ with respect to $z$ can be
bounded by $C(s_{k}-t_{i}^{N})^{-1/2}q_{N}(t_{i}^{N},s_{k},y,z)$, see (3.12)
in [20]. Using these arguments we get the bound (65).

For the proof of (66) we define
$v=(v_{1},...,v_{d})^{t}=\bar{\theta}_{s}+\delta\sqrt{\gamma_{k}^{N}}\chi_{N}(x)$
and we get for fixed $1\leq i,l\leq d$

$\displaystyle\frac{\partial}{\partial s}[F_{N}(s,z)]_{i,l}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{\gamma_{k+\mathbf{1}_{\|x\|\geq
a_{N}}}^{N}}{\gamma_{k+1}^{N}}}\int_{0}^{1}\frac{\partial}{\partial
s}\frac{\partial h_{i}}{\partial v_{l}}(v)\mathrm{d}\delta$ (68)
$\displaystyle=$ $\displaystyle\sqrt{\frac{\gamma_{k+\mathbf{1}_{\|x\|\geq
a_{N}}}^{N}}{\gamma_{k+1}^{N}}}\int_{0}^{1}\sum_{r=1}^{d}\frac{\partial^{2}h_{i}}{\partial
v_{l}\ \partial v_{r}}(v)h_{n}(\bar{\theta}_{s})\mathrm{d}\delta$

Furthermore, with
$w=(w_{1},...,w_{d})^{t}=z_{s}+\delta\sqrt{\gamma_{k}^{N}}\chi_{N}(z_{s})$ and
$z_{s}=\theta_{s,t_{j}^{N}}^{N}(y)$ we get that

$\displaystyle\frac{\partial}{\partial
s}[F_{N}(s,z_{s})]_{i,l}[\chi_{N}(z_{s})]_{l}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{\gamma_{k+\mathbf{1}_{\|x\|\geq
a_{N}}}^{N}}{\gamma_{k+1}^{N}}}\int_{0}^{1}\frac{\partial}{\partial
s}\frac{\partial h_{i}}{\partial
w_{l}}(w)\mathrm{d}\delta[\chi_{N}(z_{s})]_{l}$
$\displaystyle\qquad+[F_{N}(s,z_{s})]_{i,l}\sum_{r,q=1}^{d}\frac{\partial[\chi_{N}(z_{s})]_{l}}{\partial
w_{r}}[F_{N}(s,z_{s})]_{rq}[\chi_{N}(z_{s})]_{q}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{\gamma_{k+\mathbf{1}_{\|x\|\geq
a_{N}}}^{N}}{\gamma_{k+1}^{N}}}\int_{0}^{1}\sum_{r=1}^{d}\frac{\partial^{2}h_{i}}{\partial
w_{l}\ \partial w_{r}}(w)\frac{\partial w_{r}}{\partial s}\mathrm{d}\delta$
$\displaystyle\qquad+[F_{N}(s,z_{s})]_{i,l}\sum_{r,q=1}^{d}\frac{\partial[\chi_{N}(z_{s})]_{l}}{\partial
w_{r}}[F_{N}(s,z_{s})]_{rq}[\chi_{N}(z_{s})]_{q}.$

Now, we have that

$\displaystyle\frac{\partial w_{r}}{\partial s}$ $\displaystyle=$
$\displaystyle\frac{\partial(z_{s,r}+\delta\sqrt{\gamma_{k}^{N}}[\chi_{N}(z_{s})]_{r})}{\partial
s}$ $\displaystyle=$
$\displaystyle\sum_{q=1}^{d}[F_{N}(s,z_{s})]_{rq}[\chi_{N}(z_{s})]_{q}+\delta\sqrt{\gamma_{k}^{N}}\sum_{p,q=1}^{d}\frac{[\chi_{N}(z_{s})]_{r}}{\partial
z_{s,p}}[F_{N}(s,z_{s})]_{pq}[\chi_{N}(z_{s})]_{q}.$

This shows that $\|\frac{\partial w_{r}}{\partial s}\|\leq Ca_{N}$. Using this
bound we get from (A.4) that

$\left\|\frac{\partial}{\partial
s}[F_{N}(s,z_{s})]_{i,l}[\chi_{N}(z_{s})]_{l}\right\|\leq Ca_{N}.$

Furthermore from (68) we have that

$\left\|\frac{\partial}{\partial s}[F_{N}(s,z)]_{i,l}\right\|\leq C.$

From the last two inequalities we conclude that

$\displaystyle\bigg{|}\frac{\partial}{\partial
s}\bigg{(}[F_{N}(s,z)]_{r,l}[\chi_{N}(z)]_{l}-[F_{N}(s,z_{s}]_{r,l}[\chi_{N}(z_{s}]_{l}\bigg{)}\bigg{|}_{s=s_{k}}\bigg{|}\
\bigg{|}\frac{\partial\tilde{q}_{N}(s_{k},t_{j}^{N},z,y)}{\partial
z_{r}}\bigg{|}$ $\displaystyle\qquad\leq
C\bigg{|}\frac{\partial\tilde{q}_{N}(s_{k},t_{j}^{N},z,y)}{\partial
z_{r}}\bigg{|}\ \bigg{|}\frac{\partial}{\partial
s}[F_{N}(s,z)]_{r,l}\bigg{|}_{s=s_{k}}|z-z_{s}|+a_{N}\bigg{|}$
$\displaystyle\qquad\leq\frac{C}{\sqrt{t_{j}^{N}-s_{k}}}(1+a_{N})\frac{\partial\tilde{q}_{N}(s_{k},t_{j}^{N},z,y)}{\partial
z_{r}}.$

This bound can be used to show (65). ∎

###### Proof of Lemma 2.8.

For the proof of the lemma we use the bound

$\displaystyle\left|\tilde{q}_{N}\otimes_{N}H_{N}^{(r)}(t_{i}^{N},t_{j}^{N},x,y)\right|\leq
C^{r}\frac{\Gamma^{r}\left(\frac{1}{2}\right)}{\Gamma\left(\frac{r}{2}\right)}\gamma_{k+1}^{N}(t_{j}^{N}-t_{i}^{N})^{\frac{r}{2}-1}\int_{\mathbb{R}^{d}}\bar{q}_{N}(t_{i}^{N},t_{k}^{N},x,z)$
$\displaystyle\qquad\qquad\times\bar{q}_{N}(t_{k}^{N},t_{j}^{N},z,y)\mathrm{d}z$
$\displaystyle\qquad\leq\frac{C^{r}}{\Gamma\left(1+\frac{r}{2}\right)}\bar{q}_{N}(t_{i}^{N},t_{j}^{N},x,y),$

where the constant $C$ does not depend on $r$.With another constant
$C^{\prime}$ it holds that

$\sum_{r=N+1}^{\infty}\frac{C^{r}}{\Gamma\left(1+\frac{r}{2}\right)}\leq
C^{\prime}e^{-C^{\prime}N}.$

This immediately implies the statement of the lemma. ∎

###### Proof of Lemma 2.10.

The proof of this lemma heavily depends on the bound

$\displaystyle\left|M_{N}(t_{i}^{N},t_{j}^{N},x,y)\right|\leq
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$
(70)

for $j>i$ where

$\displaystyle
M_{N}(t_{i}^{N},t_{j}^{N},x,y)=\mathcal{K}_{N}(t_{i}^{N},t_{j}^{N},x,y)-K_{N}(t_{i}^{N},t_{j}^{N},x,y).$

The proof of this bound can be found in Subsection A.5. For a proof of the
lemma note first that

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}-\tilde{p}_{N}\otimes_{N}K_{N})(t_{i}^{N},t_{j}^{N},x,y)\right|=\left|(\tilde{p}_{N}\otimes_{N}M_{N})(t_{i}^{N},t_{j}^{N},x,y)\right|$
$\displaystyle\qquad\leq\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}|\tilde{p}_{N}(t_{i}^{N},t_{k}^{N},x,z)||M_{N}(t_{k}^{N},t_{j}^{N},z,y)|\mathrm{d}z$
$\displaystyle\qquad\leq
C\sum_{k=i}^{j-1}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-6}(t_{k}^{N}-t_{i}^{N},z-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{k}^{N},y-\theta^{N}_{t_{j}^{N},t_{k}^{N}}(z))\mathrm{d}z$
$\displaystyle\qquad\leq
C\sum_{k=i}^{j-1}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\gamma_{k+1}^{N}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\left(\int_{\gamma_{j}^{N}}^{t_{j}^{N}-t_{i}^{N}}\frac{\mathrm{d}u}{u}+1\right)\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln{\left(\frac{e(t_{j}^{N}-t_{i}^{N})}{\gamma_{j}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln{\left(\frac{C(t_{j}^{N}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)).$

For $r>1$ we use the following recursion argument

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}^{(r+1)}-\tilde{p}_{N}\otimes_{N}K_{N}^{(r+1)})(t_{i}^{N},t_{j}^{N},x,y)\right|$
(71)
$\displaystyle\leq\left|(\tilde{p}_{N}\otimes_{N}\mathcal{K}_{N}^{(r)}-\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\right|\otimes_{N}|\mathcal{K}_{N}|(t_{i}^{N},t_{j}^{N},x,y)+\left|(\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\right|\otimes_{N}|\mathcal{M}_{N}|(t_{i}^{N},t_{j}^{N},x,y).$

For the second term on the right hand side of (71) we get with the help of
(42)

$\displaystyle\left|(\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\right|\otimes_{N}|\mathcal{M}_{N}|(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\leq\sum_{k=i}^{j-1}\frac{(C(t_{k}^{N}-t_{i}^{N}))^{r}}{r!}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\gamma_{k+1}^{N}\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-6}(t_{k}^{N}-t_{i}^{N},z-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\times\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{k}^{N},y-\theta^{N}_{t_{i}^{N},t_{k}^{N}}(z))\mathrm{d}z$
$\displaystyle\leq\sum_{k=i}^{j-1}\frac{(C(t_{k}^{N}-t_{i}^{N}))^{r}}{r!}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\gamma_{k+1}^{N}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)).$

This gives

$\displaystyle\sum_{r=0}^{\infty}\left|(\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\right|\otimes_{N}|\mathcal{M}_{N}|(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\leq
C\sum_{k=i}^{j-1}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\gamma_{k+1}^{N}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\leq
C\sqrt{\gamma_{1}^{N}}\ln{\left(\frac{C(t_{j}^{N}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x)).$

We now treat the first term on the right hand side of (71). First note that

$\displaystyle\left|\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(1)})-\tilde{p}_{N}\otimes_{N}K_{N}^{(1)})\right|\otimes_{N}|\mathcal{K}_{N}|(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\sum_{k=i}^{j-1}\gamma_{k+1}^{N}\ln{\frac{C(t_{k}^{N}-t^{N}_{i})}{\gamma_{1}^{N}}}\int_{{\mathbb{R}}^{d}}\mathcal{Q}_{M-d-6}(t_{k}^{N}-t_{i}^{N},z-\theta^{N}_{t_{k}^{N},t_{i}^{N}}(x))$
$\displaystyle\qquad\qquad\times\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{k}^{N},\theta^{N}_{t_{k}^{N},t_{j}^{N}}(y)-z)\mathrm{d}z$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\int_{0}^{t_{j}^{N}-t^{N}_{i}}\ln{\left(\frac{Cu}{\gamma_{1}^{N}}\right)}\mathrm{d}u\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{i}^{N},0}(x))$
$\displaystyle=C\sqrt{\gamma_{1}^{N}}{(t_{j}^{N}-t_{i}^{N})}\left(\ln{\left(\frac{C(t_{j}^{N}-t^{N}_{i})}{\gamma_{1}^{N}}\right)}-1\right)\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t_{i}^{N}}(x))$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}{(t^{N}_{i}-t_{i}^{N})}\ln{\left(\frac{C(t^{N}_{j}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x)).$

Similarly, we get that

$\displaystyle\left|\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(1)})-\tilde{p}_{N}\otimes_{N}K_{N}^{(1)})\right|\otimes_{N}|M_{N}|(t_{i}^{N},t^{N}_{j},x,y)$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\sum_{k=0}^{i-1}\gamma_{k+1}^{N}\frac{\sqrt{\gamma_{k+1}^{N}}}{t_{j}^{N}-t_{k}^{N}}\ln{\frac{C(t_{k}^{N}-t^{N}_{i})}{\gamma_{1}^{N}}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x))$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\sum_{k=0}^{i-1}\gamma_{k+1}^{N}\frac{1}{\sqrt{t_{j}^{N}-t_{k}^{N}}}\ln{\left(\frac{C(t_{k}^{N}-t^{N}_{i})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x))$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\int_{t^{N}_{i}}^{t_{j}^{N}}\frac{1}{\sqrt{t_{j}^{N}-u}}\ln{\left(\frac{C(u-t^{N}_{i})}{\gamma_{1}^{N}}\right)}\mathrm{d}u\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x))$
$\displaystyle\leq
2C\sqrt{\gamma_{1}^{N}}{\sqrt{t^{N}_{j}-t_{i}^{N}}}\ln{\left(\frac{C(t_{j}^{N}-t^{N}_{i})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x)).$

It follows that

$\displaystyle\left|\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(1)})-\tilde{p}_{N}\otimes_{N}K_{N}^{(1)})\right|\otimes_{N}|K_{N}+M_{N}|(t^{N}_{i},t_{j}^{N},x,y)$
$\displaystyle\leq
2C\sqrt{\gamma_{1}^{N}}{\sqrt{t_{i}^{N}-t^{N}_{j}}}\ln{\left(\frac{C(t^{N}_{j}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{j}^{N},t^{N}_{i}}(x))$

and

$\displaystyle\left|\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(2)})-\tilde{p}_{N}\otimes_{N}K_{N}^{(2)})\right|(t^{N}_{i},t_{j}^{N},x,y)$
$\displaystyle\leq
3C\sqrt{\gamma_{1}^{N}}{\sqrt{t^{N}_{j}-t_{i}^{N}}}\ln{\left(\frac{C(t^{N}_{j}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t_{i}^{N},0}(x)).$

Note that in the upper bound we now have the factor
${\sqrt{t^{N}_{j}-t_{i}^{N}}}\ln{\left(\frac{C(t^{N}_{j}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}$
instead of $\ln{\left(\frac{C(t^{N}_{j}-t_{i}^{N})}{\gamma_{1}^{N}}\right)}$.
In the following iterations we make use of

$\int_{0}^{s}\sqrt{u}\ln{\left(\frac{Cu}{\gamma_{1}^{N}}\right)}\mathrm{d}u\leq\frac{2}{3}s^{\frac{3}{2}}\ln{\left(\frac{Cs}{\gamma_{1}^{N}}\right)}$

and

$\frac{2}{3}\int_{0}^{s}u^{\frac{3}{2}}\ln{\left(\frac{Cu}{\gamma_{1}^{N}}\right)}\mathrm{d}u\leq\frac{2}{3}\frac{2}{5}s^{\frac{5}{2}}\ln{\left(\frac{Cs}{\gamma_{1}^{N}}\right)}.$

Continuing in this way we obtain

$\displaystyle\left|\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(r)})-\tilde{p}_{N}\otimes_{N}K_{N}^{(r)})\right|(t^{N}_{i},t^{N}_{j},x,y)$
$\displaystyle\leq
C^{r}\frac{(t^{N}_{j}-t^{N}_{i})^{r}}{(2r-1)!!}\sqrt{\gamma_{1}^{N}}\ln{\left(\frac{C(t^{N}_{j}-t^{N}_{i})}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t^{N}_{j}-t^{N}_{i},y-\theta^{N}_{t^{N}_{j},t^{N}_{i}}(x)).$

Summing this up we get the bound

$\displaystyle\left|\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}(K_{N}+M_{N})^{(r)})(t^{N}_{i},t^{N}_{j},x,y)-\sum_{r=0}^{N}\tilde{p}_{N}\otimes_{N}K_{N}^{(r)}(t^{N}_{i},t^{N}_{j},x,y)\right|$
$\displaystyle\leq
C\sqrt{\gamma_{1}^{N}}\ln{\left(\frac{1}{\gamma_{1}^{N}}\right)}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},y-\theta^{N}_{t^{N}_{j},t^{N}_{i}}(x)),$

which concludes the proof of the lemma. ∎

###  A.5 Proof of inequality (70)

We start by showing (70) for $j-i>1$. The case $j=i+1$ will be treated
afterwards. First we will show that

$\displaystyle M_{N}(t_{i}^{N},t_{j}^{N},x,y)=T_{1}+T_{2}+T_{3}+T_{4},$ (72)

where with
$\Delta_{i}\theta^{N}(y)=\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)$
and $T_{4}=T_{4,1}-T_{4,2}$

$\displaystyle T_{1}$ $\displaystyle=$
$\displaystyle\sum_{l=1}^{d}\left[F_{N}(t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))-\frac{1}{\gamma_{i+1}^{N}}\Delta_{i}\theta^{N}(y)\right]\frac{\partial}{\partial
x_{l}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y),$ $\displaystyle T_{2}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{l,m=1}^{d}\left[R_{l,m}\left(\bar{\theta}_{t_{i}}+\chi_{N}(x)\sqrt{\gamma_{i}^{N}}\right)-R_{l,m}\left(\bar{\theta}_{t_{i}}+\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\sqrt{\gamma_{i}^{N}}\right)\right]$
$\displaystyle\qquad\times\frac{\partial^{2}}{\partial x_{l}\partial
x_{m}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y),$ $\displaystyle T_{3}$
$\displaystyle=$
$\displaystyle\frac{\gamma_{i+1}^{N}}{2}\sum_{l,m=1}^{d}\left(\left[F_{N}\left(t_{i}^{N},x\right)\chi_{N}(x)\right]_{l}\left[F_{N}\left(t_{i}^{N},x\right)\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\right]_{m}-\left[F_{N}\left(t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)\right)\right.\right.$
$\displaystyle\qquad\left.\left.\times\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\right]_{l}\right.\left.\left[F_{N}\left(t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)\right)\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\right]_{m}\right)\frac{\partial^{2}}{\partial
x_{l}\partial x_{m}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y),$ $\displaystyle
T_{4,1}$ $\displaystyle=$ $\displaystyle
3\sqrt{\gamma_{i+1}^{N}}\int_{{\mathbb{R}}^{d}}f_{t_{i}^{N},x}(v)\sum_{|\nu|=3}\frac{\left[v+F_{N}(t_{i}^{N},x)\chi_{N}(x)\sqrt{\gamma_{i+1}^{N}}\right]^{\nu}}{\nu!}$
$\displaystyle\qquad\times\int_{0}^{1}(1-\delta)^{2}D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+\delta\left(v\sqrt{\gamma_{i+1}^{N}}+F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N}\right),y\right)\mathrm{d}\delta\mathrm{d}v,$
$\displaystyle T_{4,2}$ $\displaystyle=$ $\displaystyle
3\sqrt{\gamma_{i+1}^{N}}\int_{{\mathbb{R}}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v)\sum_{|\nu|=3}\frac{\left[v+\Delta_{i}\theta^{N}(y)\right]^{\nu}}{\nu!}$
$\displaystyle\qquad\times\int_{0}^{1}(1-\delta)^{2}D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+\delta\left(v\sqrt{\gamma_{i+1}^{N}}+\Delta_{i}\theta^{N}(y)\right),y\right)\mathrm{d}\delta\mathrm{d}v,$

where $f_{t_{i}^{N},\theta}$ is the density of

$\xi_{i,\theta}=H\left(\bar{\theta}_{t_{i}}+\chi_{N}(\theta)\sqrt{\gamma_{i}^{N}},\eta_{i+1}\right)-{\mathbb{E}}\left[H\left(\bar{\theta}_{t_{i}}+\chi_{N}(\theta)\sqrt{\gamma_{i}^{N}},\eta_{i+1}\right)\right].$

For $1\leq k\leq 4$ we will show that

$\displaystyle\left|T_{k}\right|\leq
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$
(73)

for $j-i>1$. Claim (70) follows then from (72) and (73). We start with a proof
of (72). Note first that for the one-step transition densities we have

$\displaystyle p_{N}(t_{i}^{N},t_{i+1}^{N},x,z)$ $\displaystyle=$
$\displaystyle(\gamma_{i+1}^{N})^{-d/2}f_{t_{i}^{N},x}\left(\frac{y-x-
F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N}}{\sqrt{\gamma_{i+1}^{N}}}\right),$
$\displaystyle\tilde{p}_{N}^{y}(t_{i}^{N},t_{i+1}^{N},x,z)$ $\displaystyle=$
$\displaystyle(\gamma_{i+1}^{N})^{-d/2}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}\left(\frac{y-x-\int_{t_{i}^{N}}^{t_{j}^{N}}F_{N}(u,\theta^{N}_{u,t_{j}^{N}}(y))\chi_{N}(\theta^{N}_{u,t_{j}^{N}}(y))\gamma_{i+1}^{N}\mathrm{d}u}{\sqrt{\gamma_{i+1}^{N}}}\right).$

With these representations of the one-step transition densities we get with
$\varphi=\tilde{p}^{y}_{N}$ that

$\displaystyle\mathcal{K}_{N}(t_{i}^{N},t_{j}^{N},x,y)=\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)\varphi(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad=\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{1}\varphi(t_{i}^{N},t_{j}^{N},x,y)-\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{2}\varphi(t_{i}^{N},t_{j}^{N},x,y),$

where

$\displaystyle\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{1}\varphi(t_{i}^{N},t_{j}^{N},x,y)=\frac{1}{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}f_{t_{i}^{N},x}\left(\frac{z-x-
F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N}}{\sqrt{\gamma_{i+1}^{N}}}\right)$
$\displaystyle\qquad\times\left(\sum_{1\leq|\nu|\leq
2}\frac{(z-x)^{\nu}}{\nu!}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x,y)\right.$
$\displaystyle\qquad\left.+3\sum_{|\nu|=3}\frac{(z-x)^{\nu}}{\nu!}\int_{0}^{1}(1-\delta)^{2}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x+\delta(z-x),y)\mathrm{d}\delta\right)\mathrm{d}z$

and

$\displaystyle\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{2}\varphi(t_{i}^{N},t_{j}^{N},x,y)=\frac{1}{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}(y)}}\left(\frac{z-x-\Delta_{i}\theta^{N}(y)}{\sqrt{\gamma_{i+1}^{N}}}\right)$
$\displaystyle\qquad\times\left(\sum_{1\leq|\nu|\leq
2}\frac{(z-x)^{\nu}}{\nu!}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x,y)\right.$
$\displaystyle\qquad\left.+3\sum_{|\nu|=3}\frac{(z-x)^{\nu}}{\nu!}\int_{0}^{1}(1-\delta)^{2}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x+\delta(z-x),y)\mathrm{d}\delta\right)\mathrm{d}z.$

We now note that

$\displaystyle\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{1}\varphi(t_{i}^{N},t_{j}^{N},x,y)=\frac{1}{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}f_{t_{i}^{N},x}(v))$
(74) $\displaystyle\qquad\times\left(\sum_{1\leq|\nu|\leq
2}\frac{\left(v\sqrt{\gamma_{i+1}^{N}}+F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N}\right)^{\nu}}{\nu!}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x,y)\right.$
$\displaystyle\qquad\left.+3\sum_{|\nu|=3}\frac{(z-x)^{\nu}}{\nu!}\int_{0}^{1}(1-\delta)^{2}(D^{\nu}\varphi)(t_{i+1}^{N},t_{j}^{N},x+\delta\left(v\sqrt{\gamma_{i+1}^{N}}+F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N}\right),y)\mathrm{d}\delta\right)\mathrm{d}z$
$\displaystyle=\sum_{l=1}^{d}\left[F_{N}(t_{i}^{N},x)\chi_{N}(x)\right]_{l}\frac{\partial}{\partial
x_{l}}\varphi(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad+\frac{1}{2}\sum_{l,m=1}^{d}\left(R_{l,m}\left(\bar{\theta}_{t_{i}^{N}}+\chi_{N}(x)\sqrt{\gamma_{i+1}^{N}}\right)+\frac{\gamma_{i+1}^{N}}{2}\left[F_{N}(t_{i}^{N},x)\chi_{N}(x)\right]_{l}\left[F_{N}(t_{i}^{N},x)\chi_{N}(x)\right]_{m}\right)$
$\displaystyle\qquad\qquad\times\frac{\partial^{2}}{\partial x_{l}\partial
x_{m}}\varphi(t_{i}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad+3\sqrt{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}f_{t_{i}^{N},x}(v)\sum_{|\nu|=3}\frac{\left(v+F_{N}(t_{i}^{N},x)\chi_{N}(x)\sqrt{\gamma_{i+1}^{N}}\right)^{\nu}}{\nu!}\int_{0}^{1}(1-\delta)^{2}$
$\displaystyle\qquad\qquad\times(D^{\nu}\varphi)\left(t_{i+1}^{N},t_{j}^{N},x+\delta\left(v\sqrt{\gamma_{i+1}^{N}}+F_{N}(t_{i}^{N},x)\chi_{N}(x)\gamma_{i+1}^{N},y\right)\right)\mathrm{d}\delta\mathrm{d}v.$

By similar arguments one obtains that

$\displaystyle\left({\mathcal{L}}_{N}-\tilde{\mathcal{L}}_{N}\right)_{2}\varphi(t_{i}^{N},t_{j}^{N},x,y)=\frac{1}{\gamma_{i+1}^{N}}\sum_{l=1}^{d}\left[\Delta_{i}\theta^{N}(y)\right]_{l}\frac{\partial}{\partial
x_{l}}\varphi(t_{i+1}^{N},t_{j}^{N},x,y)$ (75)
$\displaystyle\qquad+\frac{1}{2}\sum_{l,m=1}^{d}\left(R_{l,m}\left(\bar{\theta}_{t_{i}^{N}}+\chi_{N}(\theta^{N}_{t_{i}^{N}},t_{j}^{N}(y))\sqrt{\gamma_{i+1}^{N}}\right)+\frac{\gamma_{i+1}^{N}}{2}\left[\Delta_{i}\theta^{N}(y)\right]_{l}\left[\Delta_{i}\theta^{N}(y)\right]_{m}\right)$
$\displaystyle\qquad\qquad\times\frac{\partial^{2}}{\partial x_{l}\partial
x_{m}}\varphi(t_{i+1}^{N},t_{j}^{N},x,y)$
$\displaystyle\qquad+3\sqrt{\gamma_{i+1}^{N}}\int_{\mathbb{R}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}(y)}}(v)\sum_{|\nu|=3}\frac{\left(v+\Delta_{i}\theta^{N}(y)\right)^{\nu}}{\nu!}$
$\displaystyle\qquad\qquad\times\int_{0}^{1}(1-\delta)^{2}(D^{\nu}\varphi)\left(t_{i+1}^{N},t_{j}^{N},x+\delta\left(v\sqrt{\gamma_{i+1}^{N}}+\Delta_{i}\theta^{N}(y),y\right)\right)\mathrm{d}\delta\mathrm{d}v.$

Claim (72) now follows for $M_{N}=\mathcal{K}_{N}-K_{N}$ by using (74), (75)
and

$\displaystyle
K_{N}(t_{i}^{N},t_{j}^{N},z,y)=(L_{t_{i}^{N}}^{N}-\tilde{L}_{t_{i}^{N}}^{N})\tilde{p}_{N}(t_{i}^{N},t_{j}^{N},z,y)$
$\displaystyle\qquad=\sum_{l,m=1}^{d}\left([F_{N}(t_{i}^{N},z)]_{l,m}[\chi_{N}(z)]_{m}-[F_{N}(t_{i}^{N},\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{l,m}[\chi_{N}(\theta_{t_{i}^{N},t_{j}^{N}}^{N}(y))]_{m}\right)$
$\displaystyle\qquad\qquad\times\frac{\partial}{\partial
z_{l}}\tilde{p}_{N}(t_{i}^{N},t_{j}^{N},z,y).$

We now come to the proof of (73) for $1\leq k\leq 4$. For $k=1$ note that by
application of (27) and (33)

$\displaystyle|T_{1}|$ $\displaystyle\leq$
$\displaystyle\sum_{l=1}^{d}\frac{1}{\gamma_{i+1}^{N}}\int_{t_{i}^{N}}^{t_{i+1}^{N}}\left|\left[F_{N}(t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))-F_{N}(u,\theta^{N}_{u,t_{j}^{N}}(y))\chi_{N}(\theta^{N}_{u,t_{j}^{N}}(y))\right]_{l}\right|\mathrm{d}u$
$\displaystyle\qquad\times\left|\frac{\partial}{\partial
x_{l}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y)\right|$ $\displaystyle\leq$
$\displaystyle C\left(\max_{t_{i}^{N}\leq u\leq
t_{i+1}^{N}}\left|\theta^{N}_{u,t_{j}^{N}}(y)-\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)\right|+|y|\gamma_{i+1}^{N}\right)$
$\displaystyle\qquad\times(t_{j}^{N}-t_{i}^{N})^{-1/2}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x)$
$\displaystyle\leq$ $\displaystyle
Ca_{N}\gamma_{i+1}^{N}(t_{j}^{N}-t_{i}^{N})^{-1/2}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$

where the last inequality follows directly if $|y|\leq a_{N}+1$. If
$|y|>a_{N}+1$ we get $\theta^{N}_{t,t_{j}^{N}}(y)\equiv y$ for $t\leq
t_{j}^{N}$ is a solution of
$\frac{\mathrm{d}}{\mathrm{d}t}\theta^{N}_{t,t_{j}^{N}}(y)=F_{N}(t,\theta^{N}_{t,t_{j}^{N}}(y))\chi_{N}(\theta^{N}_{t,t_{j}^{N}}(y))$
with terminating value $\theta^{N}_{t_{j}^{N},t_{j}^{N}}(y)=y$. Thus in the
latter case we have that $T_{1}=0$. This shows that in both cases (73) holds
for $k=1$.

For the proof of (73) for $k=2$,

$\displaystyle|T_{2}|$ $\displaystyle\leq$ $\displaystyle
C\sum_{l,m=1}^{d}\left|R_{l,m}\left(\bar{\theta}_{t_{i}}+\chi_{N}(x)\sqrt{\gamma_{i}^{N}}\right)-R_{l,m}\left(\bar{\theta}_{t_{i}}+\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\sqrt{\gamma_{i}^{N}}\right)\right|$
$\displaystyle\qquad\times\left|\frac{\partial^{2}}{\partial x_{l}\partial
x_{m}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y)\right|$ $\displaystyle\leq$
$\displaystyle
C\sqrt{\gamma_{i}^{N}}\sum_{l,m=1}^{d}\left|\chi_{N}(x)-\chi_{N}(\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y))\right|\
\left|\frac{\partial^{2}}{\partial x_{l}\partial
x_{m}}\tilde{p}_{N}(t_{i+1}^{N},t_{j}^{N},x,y)\right|$ $\displaystyle\leq$
$\displaystyle
C\sqrt{\gamma_{i}^{N}}(t_{j}^{N}-t_{i+1}^{N})^{-1/2}\left|\frac{\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x}{(t_{j}^{N}-t_{i+1}^{N})^{1/2}}\right|\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x)$
$\displaystyle\leq$ $\displaystyle
C\sqrt{\gamma_{i+1}^{N}}(t_{j}^{N}-t_{i}^{N})^{-1/2}\left|\frac{\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x}{(t_{j}^{N}-t_{i}^{N})^{1/2}}\right|\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$

where in the last inequality we used that for some $c>0$ small enough
$c/(c+1)\leq(t_{j}^{N}-t_{i+1}^{N})/(t_{j}^{N}-t_{i}^{N})\leq 1$ because of
$(t_{j}^{N}-t_{i+1}^{N})/(t_{j}^{N}-t_{i}^{N})=1-\gamma_{i+1}^{N}/(t_{j}^{N}-t_{i}^{N})$
and $\gamma_{i+1}^{N}/(t_{j}^{N}-t_{i}^{N})\leq
1/(1+\gamma_{i+2}^{N}/\gamma_{i+1}^{N})\leq 1/(1+c)$ for $c>0$ small enough.
We conclude that

$\displaystyle|T_{2}|$ $\displaystyle\leq$ $\displaystyle
C\sqrt{\gamma_{i+1}^{N}}(t_{j}^{N}-t_{i}^{N})^{-1/2}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$

which shows (73) for $k=2$. For $k=3$ we get by using (27) that

$\displaystyle T_{3}$ $\displaystyle\leq$
$\displaystyle\frac{a_{N}\gamma_{i+1}^{N}}{t_{j}^{N}-t_{i+1}^{N}}\left|\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x\right|\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$

which by arguments similar to the ones used for $k=2$ can be bounded by

$\displaystyle\frac{a_{N}\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i+1}^{N})^{1/2}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x).$

This shows (73) for $k=3$. For $k=4$ choose $\nu$ with $|\nu|=3$ and put

$\displaystyle p_{1}(v)$ $\displaystyle=$ $\displaystyle f_{t_{i}^{N},x}(v),$
$\displaystyle p_{2}(v)$ $\displaystyle=$ $\displaystyle
f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v),$ $\displaystyle
a_{1}(v)$ $\displaystyle=$ $\displaystyle(v+\delta_{1,N})^{\nu},$
$\displaystyle a_{2}(v)$ $\displaystyle=$
$\displaystyle(v+\delta_{2,N})^{\nu}$

with

$\displaystyle\delta_{1,N}$ $\displaystyle=$ $\displaystyle
F_{N}(t_{i}^{N},x)\chi_{N}(x)\sqrt{\gamma_{i+1}^{N}},$
$\displaystyle\delta_{2,N}$ $\displaystyle=$
$\displaystyle\Delta_{i}\theta^{N}(y)$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{\gamma_{i+1}^{N}}}\int_{t_{i}^{N}}^{t_{i+1}^{N}}F_{N}(u,\theta^{N}_{u,t_{j}^{N}}(y))\chi_{N}(\theta^{N}_{u,t_{j}^{N}}(y))\mathrm{d}u.$

From the latter representation of $\delta_{2,N}$ one gets the following bounds
with the help of (27), of (32) in Lemma 3.3, by using that $F_{N}$ is Lipshitz
with respect to its first argument and that $\chi_{N}$ is absolutely bounded
by $a_{N}$.

$\displaystyle\left|\delta_{2,N}-\delta_{1,N}\right|$ $\displaystyle=$
$\displaystyle\tilde{\delta}_{1,N}+\tilde{\delta}_{2,N},$ (76)
$\displaystyle\left|\delta_{l,N}\right|$ $\displaystyle\leq$ $\displaystyle
Ca_{N}\sqrt{\gamma_{i+1}^{N}},$ (77) $\displaystyle|a_{l}(v)|$
$\displaystyle\leq$ $\displaystyle C(v+1)^{\nu}$ (78)

for $l=1,2$, where

$\displaystyle\tilde{\delta}_{1,N}$ $\displaystyle\leq$ $\displaystyle
C\sqrt{\gamma_{i+1}^{N}}\left|x-\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)\right|,$
$\displaystyle\tilde{\delta}_{2,N}$ $\displaystyle\leq$ $\displaystyle
Ca_{N}(\gamma_{i+1}^{N})^{3/2}.$

Furthermore, we put for $l=1,2$

$\displaystyle b_{l}(v)$ $\displaystyle=$
$\displaystyle\int_{0}^{1}(1-u)^{2}D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{l,N}),y\right)\mathrm{d}u.$

We will show

$\displaystyle\left|\int_{{\mathbb{R}}^{d}}(p_{1}(v)-p_{2}(v))a_{1}(v)b_{1}(v)\mathrm{d}v\right|$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x)$
(79)
$\displaystyle\left|\int_{{\mathbb{R}}^{d}}p_{2}(v)a_{1}(v)(b_{1}(v)-b_{2}(v))\mathrm{d}v\right|$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x),$
(80)
$\displaystyle\left|\int_{{\mathbb{R}}^{d}}p_{2}(v)b_{2}(v)(a_{1}(v)-a_{2}(v))\mathrm{d}v\right|$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x).$
(81)

These three inequalities imply that

$\displaystyle\left|\int_{{\mathbb{R}}^{d}}p_{1}(v)a_{1}(v)b_{1}(v)\mathrm{d}v-\int_{{\mathbb{R}}^{d}}p_{2}(v)a_{2}(v)b_{2}(v)\mathrm{d}v\right|$
$\displaystyle\leq$ $\displaystyle
C\frac{\sqrt{\gamma_{i+1}^{N}}}{t_{j}^{N}-t_{i}^{N}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x).$

By summing both sides of the last inequality over $\nu$ with $|\nu|=3$ we get
(73) for $k=4$. We now show (79)–(81). We start with the proof of (80). For
this proof we will use that

$\displaystyle\left|b_{1}(v)-b_{2}(v)\right|$ $\displaystyle\leq$
$\displaystyle
C\frac{\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i}^{N})^{3/2}}(1+|v|)^{M-d-5}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x).$
(82)

With this inequality and (78) we get that

$\displaystyle\left|\int_{{\mathbb{R}}^{d}}p_{2}(v)a_{1}(v)(b_{1}(v)-b_{2}(v))\mathrm{d}v\right|\leq
C\frac{\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i}^{N})^{3/2}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x)$
$\displaystyle\qquad\times\int_{{\mathbb{R}}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v)(1+|v|)^{M-d-5+3}\mathrm{d}v$
$\displaystyle\leq
C\frac{\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i}^{N})^{3/2}}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x),$

which implies (80) because of $\gamma_{i+1}^{N}\leq t_{j}^{N}-t_{i}^{N}$. For
(80) it remains to show (82). For this proof note that by (76) and (29) with
$\delta_{N}(\tau^{N})=\delta_{1N}+\tau^{N}(\delta_{2N}-\delta_{1N})$

$\displaystyle\left|b_{1}(v)-b_{2}(v)\right|$
$\displaystyle\qquad\leq\int_{0}^{1}(1-u)^{2}\left|D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{l,N}),y\right)\right.$
$\displaystyle\qquad\quad\left.-D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{l,N}),y\right)\right|\mathrm{d}u$
$\displaystyle\qquad\leq\int_{0}^{1}(1-u)^{2}\left|u\sqrt{\gamma_{i+1}^{N}}\sum_{k=1}^{d}(\delta_{2N}-\delta_{1N})_{k}\right.$
$\displaystyle\qquad\qquad\left.\times\int_{0}^{1}D^{\nu+e_{k}}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{N}(\tau^{N})),y\right)\mathrm{d}\tau^{N}\right|\mathrm{d}u$
$\displaystyle\qquad\leq
C\frac{\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i}^{N})^{2}}\left(\left|\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x\right|+a_{N}\gamma_{i+1}^{N}\right)\int_{0}^{1}(1-u)^{2}$
$\displaystyle\qquad\qquad\times\int_{0}^{1}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x-u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{N}(\tau^{N})))\mathrm{d}\tau^{N}\
\mathrm{d}u.$

Now by application of (38) we get that

$\displaystyle\left|b_{1}(v)-b_{2}(v)\right|$ $\displaystyle\qquad\leq
C\frac{\gamma_{i+1}^{N}}{(t_{j}^{N}-t_{i}^{N})^{2}}\left(\left|\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x\right|+a_{N}\gamma_{i+1}^{N}\right)\left(1+|v|+a_{N}\sqrt{\gamma_{i+1}^{N}}\right)^{M-d-5}\int_{0}^{1}(1-u)^{2}$
$\displaystyle\qquad\qquad\times\int_{0}^{1}\mathcal{Q}_{M-d-6}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x-u\sqrt{\gamma_{i+1}^{N}}\tau^{N}\tilde{\delta}_{1N}))\mathrm{d}\tau^{N}\
\mathrm{d}u.$

Claim (78) now follows by application of (39).

We now show (81). Note that by application of (29), (38)

$\displaystyle\left|\int_{{\mathbb{R}}^{d}}p_{2}(v)b_{2}(v)(a_{1}(v)-a_{2}(v))\mathrm{d}v\right|$
$\displaystyle\qquad\leq
C\int_{{\mathbb{R}}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v)\int_{0}^{1}(1-u)^{2}\left|D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{2,N}),y\right)\right|\mathrm{d}u$
$\displaystyle\qquad\qquad\times\left|(v+\delta_{1,N})^{\nu}-(v+\delta_{2,N})^{\nu}\right|\mathrm{d}v$
$\displaystyle\qquad\leq
C(t_{j}^{N}-t_{i+1}^{N})^{-3/2}\int_{{\mathbb{R}}^{d}}f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v)\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x)$
$\displaystyle\qquad\qquad\times(1+|v|+\delta_{2N})^{M-d-5}\left(|v|^{2}|\delta_{1N}-\delta_{2N}|+|v||\delta_{1N}^{2}-\delta_{2N}^{2}|+|\delta_{1N}^{3}-\delta_{2N}^{3}|\right)\mathrm{d}v$
$\displaystyle\qquad\leq
C(t_{j}^{N}-t_{i+1}^{N})^{-3/2}\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x)$
$\displaystyle\qquad\qquad\times\sqrt{\gamma_{i+1}^{N}}\left(|\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x|+\gamma_{i+1}^{N}\right),$

where in the last step (76)–(77) has been applied. Claim (81) now follows by
application of (39). We now prove (79). For the proof of this claim first note
that by our assumptions and by (28) and (38):

$\displaystyle|p_{1}(v)-p_{2}(v)|=\left|f_{t_{i}^{N},x}(v)-f_{t_{i}^{N},\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)}(v)\right|$
$\displaystyle\quad\leq
C\sqrt{\gamma_{i+1}^{N}}\left|\theta^{N}_{t_{i}^{N},t_{j}^{N}}(y)-x\right|Q_{m}(v),$
$\displaystyle|b_{1}(v)|\leq\int_{0}^{1}(1-u)^{2}\left|D^{\nu}\tilde{p}_{N}^{y}\left(t_{i+1}^{N},t_{j}^{N},x+u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{l,N}),y\right)\right|\mathrm{d}u$
$\displaystyle\quad\leq
C(t_{j}^{N}-t_{i+1}^{N})^{-3/2}\int_{0}^{1}(1-u)^{2}\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x-u\sqrt{\gamma_{i+1}^{N}}(v+\delta_{2,N}))\mathrm{d}u$
$\displaystyle\quad\leq
C(t_{j}^{N}-t_{i+1}^{N})^{-3/2}(1+|v|)^{M-d-5}\mathcal{Q}_{M-d-5}(t_{j}^{N}-t_{i}^{N},\theta^{N}_{t_{i+1}^{N},t_{j}^{N}}(y)-x).$

Claim (79) now follows by combining these bounds with the bound (78) for
$a_{1}(v)$ and by applying (39).

This concludes the proof of (70) for $j-i>1$. It remains to show (70) for the
case $j=i+1$. Because of (48) we have to show that

$\displaystyle\mathcal{K}_{N}(t_{i}^{N},t_{i+1}^{N},x,y)$ $\displaystyle\leq$