Neural Network Approximation of Optimal Controls for Stochastic
  Reaction-Diffusion Equations
We present a numerical algorithm that allows the approximation of optimal
controls for stochastic reaction-diffusion equations with additive noise by
first reducing the problem to controls of feedback form and then approximating
the feedback function using finitely based approximations. Using structural
assumptions on the finitely based approximations, rates for the approximation
error of the cost can be obtained. Our algorithm significantly reduces the
computational complexity of finding controls with asymptotically optimal cost.
Numerical experiments using artificial neural networks as well as radial basis
function networks illustrate the performance of our algorithm. Our approach can
also be applied to stochastic control problems for high dimensional stochastic
differential equations and more general stochastic partial differential
equations.
Wilhelm Stannat, Alexander Vogler, Lukas Wessels
# Neural Network Approximation of Optimal Controls for Stochastic Reaction-Diffusion Equations

###### Abstract

We present a numerical algorithm that allows the approximation of optimal controls for stochastic reaction-diffusion equations with additive noise by first reducing the problem to controls of feedback form and then approximating the feedback function using finitely based approximations. Using structural assumptions on the finitely based approximations, rates for the approximation error of the cost can be obtained. Our algorithm significantly reduces the computational complexity of finding controls with asymptotically optimal cost. Numerical experiments using artificial neural networks as well as radial basis function networks illustrate the performance of our algorithm. Our approach can also be applied to stochastic control problems for high dimensional stochastic differential equations and more general stochastic partial differential equations.

**There is a huge body of literature on optimal control problems with partial differential equation (PDE) constraints and their numerical treatment. Recent years have shown a rising interest in the optimal control of stochastic partial differential equations (SPDEs). However, the numerical approximation of optimal controls, let alone its practical implementation in the stochastic case, faces serious obstacles due to the computational complexity of classical algorithms. In this work, we present a new numerical algorithm that approximates feedback controls for SPDEs with asymptotically optimal cost. The algorithm is based on adjoint calculus applied to gradient descent. For the approximation of feedback controls, we use finitely based approximations such as artificial neural networks or radial basis function networks. The restriction to additive noise and our approach for the approximation of feedback functions enables us to significantly reduce the algorithmic complexity of our approach in comparison with classical algorithms.**

+
Footnote †: preprint: APS/123-QED

## I Introduction

For a fixed finite time horizon \(T>0\), we consider the randomly forced reaction-diffusion equation

\[\begin{cases}\partial_{t}u(t,x)=\Delta u(t,x)+f(u(t,x))+\xi(t,x),\,(t,x)\in[0, T]\times\Lambda\\ u(0,x)=u(x),\quad x\in\Lambda\end{cases} \tag{1}\]

on a bounded domain \(\Lambda\subset\mathbb{R}^{d}\) with Neumann boundary conditions. Here, \(\Delta:=\sum_{k=1}^{d}\frac{\partial^{2}}{\partial x_{k}^{2}}\) denotes the Laplace operator, \(f:\mathbb{R}\to\mathbb{R}\) models a local reaction term, and \(\xi(t,x)\) are random fluctuations. Deterministic reaction-diffusion equations are ubiquitous in the natural sciences and, in many situations, taking into account random fluctuations leads to the more realistic model (1). Typically, these random fluctuations are highly irregular, and therefore, in order to treat this equation rigorously, we reformulate it as the following \(L^{2}(\Lambda)\)-valued SPDE:

\[\begin{cases}\mathrm{d}u_{t}=[\Delta u_{t}+\mathcal{F}(u_{t})]\mathrm{d}t+ \sigma\mathrm{d}W_{t},\quad t\in[0,T]\\ u_{0}=u\in L^{2}(\Lambda),\end{cases} \tag{2}\]

where \(u_{t}(x)=u(t,x)\) for fixed \(t\in[0,T]\) is considered as an element in \(L^{2}(\Lambda)\). The random fluctuations \(\xi(t,x)\) are modelled by a cylindrical Wiener process \((W_{t})_{t\in[0,T]}\) on \(L^{2}(\Lambda)\), defined on some underlying probability space \((\Omega,\mathcal{F},\mathbb{P})\), and \(\sigma:L^{2}(\Lambda)\to L^{2}(\Lambda)\) is a Hilbert-Schmidt operator. For more details on the mathematical theory of SPDEs, see [1]. Furthermore, \(\mathcal{F}:L^{2}(\Lambda)\to L^{2}(\Lambda)\) denotes the Nemytskii operator associated with \(f\), i.e.,

\[\mathcal{F}(u)(x):=f(u(x)),\quad u\in L^{2}(\Lambda),x\in\Lambda.\]

The objective of control theory is to achieve a desired outcome for a dynamical system by applying an external input which can be chosen freely among a set of admissible inputs. In order to set up a mathematical formulation for the control of the SPDE (2), we introduce a control process \(\mathfrak{g}:[0,T]\times\Omega\to\mathcal{U}\subset L^{2}(\Lambda)\), adapted to the filtration generated by \(u_{t}\), as a forcing term on the right-hand side of the equation

\[\begin{cases}\mathrm{d}u_{t}^{\mathfrak{g}}=[\Delta u_{t}^{ \mathfrak{g}}+\mathcal{F}(u_{t}^{\mathfrak{g}})+\mathfrak{g}_{t}]\mathrm{d}t+ \sigma\mathrm{d}W_{t},\quad t\in[0,T]\\ u_{0}^{\mathfrak{g}}=u\in L^{2}(\Lambda),\end{cases} \tag{3}\]

and define the cost functional

\[J(\mathfrak{g}):=\mathbb{E}\bigg{[}\int_{0}^{T}\int_{\Lambda}l( t,x,u_{t}^{\mathfrak{g}}(x))\mathrm{d}x+\frac{\nu}{2}\|\mathfrak{g}_{t}\|_{L^{2}( \Lambda)}^{2}\mathrm{d}t\\ +\int_{\Lambda}m(x,u_{T}^{\mathfrak{g}}(x))\mathrm{d}x\bigg{]}. \tag{4}\]

Here, the running cost \(l(t,x,u):[0,T]\times\Lambda\times\mathbb{R}\to\mathbb{R}\) and the terminal cost \(m(x,u):\Lambda\times\mathbb{R}\to\mathbb{R}\) are assumed to be differentiable and Lipschitz continuous in the state variable \(u\). Now, the objective is to find a control process \(\mathfrak{g}\) that minimizes the cost functional (4) over some set of admissible controls subject to (3).
In the deterministic case, there is a huge body of literature on necessary and sufficient optimality conditions [2; 3; 4], as well as numerical algorithms which efficiently approximate optimal controls, see e.g. [5; 6; 7; 8]. In recent years, extensions to the stochastic case have seen a rising interest in the mathematical literature leading to necessary and sufficient optimality conditions in great generality [9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21]. However, classical algorithms for the numerical approximation of optimal controls in the stochastic case either require the approximation of backward SPDEs or the approximation of infinite dimensional Hamilton-Jacobi-Bellman (HJB) equations. Due to the curse of dimensionality, both of these alternatives are computationally very expensive, leading to an increased interest in the development of new, more efficient algorithms [22; 23; 24; 25; 26; 27; 28; 29; 30; 31].

A considerable reduction of the problem can be achieved in the case when an optimal control \(\mathfrak{g}^{*}\) is of feedback-type, i.e.,

\[\mathfrak{g}^{*}_{t}=G^{*}(t,u^{G^{*}}_{t}), \tag{5}\]

for some \(G^{*}:[0,T]\times L^{2}(\Lambda)\to L^{2}(\Lambda)\). In this work, we consider a mathematical setting in which the optimal control is indeed of the above type. In a first step we then use finitely based approximations for \(G^{*}\) in order to approximate the optimal control \(\mathfrak{g}^{*}\). This approach together with the restriction to additive noise enables us to replace the backward SPDE arising in adjoint calculus by a random backward PDE which significantly reduces its computational complexity. A similar idea was already used in [32].

The reduction to finitely based feedback controls then allows in a second step the local uniform approximation of finitely based feedback controls in appropriate ansatz spaces of variable, but finite, dimension. Solving the corresponding finite dimensional optimal control problem, e.g. with artificial neural networks, leads to an efficient computation of finite dimensional controls, whose costs approximate the optimal cost with increasing dimension, see Theorem 4. Imposing additional smoothness assumptions on the optimal control \(\mathfrak{g}^{*}_{t}=G^{*}(t,u^{G^{*}}_{t})\), in particular \(G^{*}\) globally Lipschitz, also enables us to derive rates for the approximation error in ansatz spaces consisting of Lipschitz continuous feedback functions, see Theorem 5. We discuss the example of neural network approximation in detail; see Theorem 8 for the universal approximation of finitely based feedback controls with 1-layer artificial neural networks and Proposition 9 for rates on the approximation error.

The practical implementation of the gradient descent algorithm for the approximation of finite dimensional feedback controls also requires the numerical discretization of the controlled state equation. In Proposition 11, we derive the combined approximation error for the optimal cost. Detailed proofs of our results, that also hold for more general ansatz spaces beyond artificial neural networks, can be found in [33]. The performance of our gradient descent algorithm is illustrated with three examples. The first example deals with the validation of our algorithm in the case of linear quadratic control. The other two examples consider the problem of stabilizing a bump solution of the stochastic Nagumo equation, in one case with general feedback controls, in the other case with feedback controls of Nemytskii-type.

The remainder of the paper is organized as follows: First, in Section II, we show that it is sufficient to consider feedback controls. In Section III, we explain how to approximate the optimal control by introducing ansatz spaces of controls that are suited for the numerical implementation. Section IV contains our main approximation results. In Section V, we describe our gradient descent algorithm. Next, in Section VI, we discuss the explicit example of artificial neural networks for the finite dimensional ansatz spaces. In Section VII, we investigate the error resulting from the numerical discretization. Finally, in Section VIII, we present numerical experiments using artificial neural networks and radial basis function networks.

## II Optimal controls of feedback-type

In order to reduce the complexity of the problem, we assume that there exists an optimal control \(\mathfrak{g}^{*}\) in feedback form (5), for some continuous \(G^{*}:[0,T]\times L^{2}(\Lambda)\to L^{2}(\Lambda)\) that satisfies a linear growth condition. This can be achieved, using the solution of the associated HJB equation

\[\begin{cases}\partial_{t}V+\frac{1}{2}\mathrm{tr}(\sigma^{*}D^{2}V\sigma)+ \left\langle DV,\Delta u+\mathcal{F}(u)\right\rangle_{L^{2}(\Lambda)}\\ +\int_{\Lambda}I(t,x,u(x))\mathrm{d}x+\inf_{G\in\mathcal{U}}\left\{\left\langle DV,G\right\rangle+\frac{\nu}{2}\|G\|^{2}_{L^{2}(\Lambda)}\right\}=0,\\ (t,u)\in[0,T]\times L^{2}(\Lambda)\\ V(T,u)=\int_{\Lambda}m(x,u(x))\mathrm{d}x,\quad u\in L^{2}(\Lambda).\end{cases} \tag{6}\]

For solution theories regarding equations of this type, see [11]. In particular, if (6) has a unique mild solution satisfying certain regularity assumptions, and

\[\gamma(p):=\mathrm{arginf}_{G\in\mathcal{U}}\left\{\left\langle p,G\right\rangle +\frac{\nu}{2}\|G\|^{2}_{L^{2}(\Lambda)}\right\}\]

is continuous, then

\[\mathfrak{g}^{*}_{t}:=G^{*}(t,u^{G^{*}}_{t}):=\gamma(DV(t,u^{G^{*}}_{t})).\]

is an optimal control in feedback form, provided that \(u^{G^{*}}\) is a unique strong solution of the closed loop equation

\[\begin{cases}\mathrm{d}u^{G}_{t}=[\Delta u^{G}_{t}+\mathcal{F}(u^{G}_{t})+G(t,u^{G}_{t})]\mathrm{d}t+\sigma\mathrm{d}W_{t},\;t\in[0,T]\\ u^{G}_{0}=u\in L^{2}(\Lambda)\end{cases} \tag{7}\]

with \(G=G^{*}\). A sufficient condition for (7) to have a unique strong solution is that \(G^{*}\) is Lipschitz continuous in \(u\), see e.g. [1]. This is in particular the case if the solution \(V\) of the HJB equation (6) has a bounded second derivative in \(u\), see Theorem 4.155 and Remark 4.202 in [11]. However, directly tackling the optimal control problem by approximating the solution of the HJB equation (6) numerically is very challenging due to the infinite dimensionality of the domain of \(V\). Instead, our approach is to approximate the feedback function directly. Therefore we consider the following feedback control prob
lem: Minimize

\[J(G):=\mathbb{E}\bigg{[}\int_{0}^{T}\int_{\Lambda}l(t,x,u_{t}^{G}(x) )\mathrm{d}x+\frac{\nu}{2}\|G(t,u_{t}^{G})\|_{L^{2}(\Lambda)}^{2}\mathrm{d}t\\ +\int_{\Lambda}m(x,u_{T}^{G}(x))\mathrm{d}x\bigg{]}\]

subject to equation (7). We seek to minimize \(J\) over the set of admissible controls

\[U_{\mathrm{ad}}:=\{G:[0,T]\times L^{2}(\Lambda)\to\mathcal{U}\,|\,G(\cdot,u_{ \cdot}^{G})\in\mathbb{A}\},\]

where \(\mathbb{A}=L^{2}([0,T]\times\Omega;L^{2}(\Lambda))\).

**Example 1** (Linear Quadratic Control): _Let us consider the linear quadratic control problem_

\[\begin{cases}\mathrm{d}u_{t}^{\mathfrak{g}}=[\Delta u_{t}^{\mathfrak{g}}+ \mathfrak{g}_{t}]\mathrm{d}t+\sigma\mathrm{d}W_{t},\quad t\in[0,T]\\ u_{0}^{\mathfrak{g}}=u\in L^{2}(\Lambda),\end{cases}\]

_where \(\mathfrak{g}:[0,T]\times\Omega\to L^{2}(\Lambda)\) is an adapted process, and_

\[J(\mathfrak{g}):=\frac{1}{2}\mathbb{E}\left[\int_{0}^{T}\int_{\Lambda}(u_{t}^ {\mathfrak{g}}(x))^{2}+\mathfrak{g}_{t}^{2}(x)\mathrm{d}x\mathrm{d}t+\int_{ \Lambda}(u_{T}^{\mathfrak{g}}(x))^{2}\mathrm{d}x\right].\]

_In this case, the optimal control \(\mathfrak{g}^{*}\) is indeed of feedback form, given by_

\[\mathfrak{g}_{t}^{*}=P(t)u_{t}^{\mathfrak{g}^{*}}\]

_where \(P:[0,T]\to L(L^{2}(\Lambda))\) is the solution of the Riccati equation34_

Footnote 3: The solution \(\mathcal{U}\) is not necessarily a solution of the Riccati equation (34).

\[\begin{cases}\partial_{t}P(t)+P(t)\Delta+\Delta P(t)-Id+P^{2}(t)=0,\,\,\,t\in[ 0,T]\\ P(T)=-Id.\end{cases} \tag{8}\]

## III Construction of ansatz spaces

For the efficient numerical implementation we need to restrict ourselves to a subset of controls \(\mathbb{U}\subset U_{\mathrm{ad}}\) that are suitable for the implementation in a gradient descent algorithm. However, we need to ensure that when minimizing over \(\mathbb{U}\), we do not end up much worse than in the original control problem. In this section we will provide a method to construct suitable ansatz spaces \(\mathbb{U}\) when we do not have any particular control constraints, i.e. \(\mathcal{U}=L^{2}(\Lambda)\).

In order to construct a suitable space \(\mathbb{U}\), we introduce the so-called finitely based approximation of a function \(G:[0,T]\times L^{2}(\Lambda)\to L^{2}(\Lambda)\). To this end, we consider finite dimensional subspaces

\[S_{n}\subset L^{2}(\Lambda)\]

with orthonormal basis \(e_{0},\ldots,e_{n}\) and orthogonal projections \(P_{n}:L^{2}(\Lambda)\to S_{n}\), such that

\[\|P_{n}u-u\|_{L^{2}(\Lambda)}\to 0.\]

The finitely based approximations of \(G\) with respect to \(S_{n}\) are then defined by

\[G^{n}(t,u) :=P_{n}G(t,P_{n}u) \tag{9}\] \[=\sum_{k=0}^{n}\langle G(t,\sum_{j=0}^{n}u_{j}e_{j}),e_{k}\rangle_ {L^{2}(\Lambda)}e_{k},\]

where \(u_{j}:=\langle u,e_{j}\rangle_{L^{2}(\Lambda)}\). One possible choice for the finite dimensional subspaces in the case of \(\Lambda=(0,1)\) is

\[S_{n}:=\mathrm{span}\left\{1,\sqrt{2}\cos(k\pi\cdot)\Big{|}\,k=1,\ldots,n \right\}.\]

In this case the finitely based approximation of \(G\) is given by

\[G^{n}(t,u)(x)\] \[:=\sum_{k=0}^{n}2\langle G(t,\sum_{j=0}^{n}u_{j}\sqrt{2}\cos(j\pi \cdot)),\cos(k\pi\cdot)\rangle_{L^{2}(0,1)}\cos(k\pi x),\]

for \(u=\sum_{j=0}^{n}u_{j}\cos(j\pi\cdot)\in L^{2}(\Lambda)\).

Our main results in Section IV provide approximation results for ansatz spaces \(\mathbb{U}\subset U_{\mathrm{ad}}\). The main assumption on the ansatz spaces \(\mathbb{U}\subset U_{\mathrm{ad}}\) is the following approximation property with respect to the optimal feedback \(G^{*}\):

**Definition 2**: _Let \(G:[0,T]\times L^{2}(\Lambda)\to L^{2}(\Lambda)\). We say that a subset \(\mathbb{U}\subset U_{\mathrm{ad}}\) satisfies the universal approximation property with respect to \(G\), if there exists a sequence \((G^{n,m})_{n,m\in\mathbb{N}}\subset\mathbb{U}\) that satisfies the following linear growth condition uniformly in \(m\)_

\[\|G^{n,m}(t,u)\|_{L^{2}(\Lambda)}\leq C_{n}(1+\|u\|_{L^{2}(\Lambda)}),\]

_for some constant \(C_{n}>0\), such that for any \(R>0\)_

\[\lim_{m\to\infty}\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,R)} \|G^{n}(t,u)-G^{n,m}(t,u)\|_{L^{2}(\Lambda)}^{2}=0,\]

_where_

\[\mathcal{B}_{L^{2}(\Lambda)}(0,R):=\{u\in L^{2}(\Lambda)|\|u\|_{L^{2}(\Lambda )}\leq R\}\]

_and \(G^{n}\) is given as in (9)._

We would like to stress that in the above definition we only require that the finitely based approximations of \(G\) can be approximated uniformly on bounded sets, and not \(G\) itself.

In the first part of this section we will explain how to construct ansatz spaces \(\mathbb{U}\) that satisfies the universal approximation property with respect to the optimal feedback \(G^{*}\). These ansatz spaces consider finitely based controls of arbitrary dimension \(n\in\mathbb{N}\), which are of the type

\[G(t,u)=\sum_{k=0}^{n}\psi^{k}(t,P_{n}u)e_{k},\]

for some functions \(\psi^{k}:[0,T]\times S_{n}\to\mathbb{R}\). However, in practice the dimension \(n\in\mathbb{N}\) of the ansatz space need to be fixed
a priori and one is interested in how close one can get to the optimal cost. Our second main result, Theorem 5, provides explicit convergence rates, however, we need to strengthen the assumption on our ansatz spaces. In the second part of this section we will explain how to construct for any \(n\in\mathbb{N}\) a sequence of ansatz spaces \((\mathbb{U}^{n,m})_{m\in\mathbb{N}}\subset U_{\mathrm{ad}}\) that satisfies the following approximation property with respect to the optimal feedback \(G^{*}\):

**Definition 3**: _Let \(G:[0,T]\times L^{2}(\Lambda)\to L^{2}(\Lambda)\) and \(n\in\mathbb{N}\). We say that a sequence of subsets \((\mathbb{U}^{n,m})_{m\in\mathbb{N}}\subset U_{\mathrm{ad}}\) satisfies the Lipschitz approximation property with respect to \(G\) in dimension \(n\), if there exists a sequence of Lipschitz continuous controls \((G^{n,m})_{m\in\mathbb{N}}\) with Lipschitz constants independent of \(m\), such that \(G^{n,m}\in\mathbb{U}^{n,m}\), and a sequence of radii \((R^{n}_{m})_{m\in\mathbb{N}}\) with \(\lim_{m\to\infty}R^{n}_{m}=\infty\), such that_

\[\begin{split}\varepsilon^{n}_{m}&:=\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,R^{n}_{m})}\|P_{n}(G^{n}(t,u)-G^{n,m}( t,u))\|^{2}_{L^{2}(\Lambda)}\\ &\to 0,\end{split} \tag{10}\]

_as \(m\to\infty\)._

For this type of ansatz spaces, Theorem 5 provides error estimates for

\[|\inf_{g\in\mathbb{U}^{n,m}}J(g)-\inf_{g\in\Lambda}J(g)|\]

in terms of \(\varepsilon^{n}_{m}\) and the projection error \(\gamma_{n}\) (11).

### Universal Ansatz Spaces

We start by constructing an ansatz space that satisfies a universal approximation property with respect to \(G^{*}\). For \(n\in\mathbb{N}\), we consider the function \(g^{n}:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}\) given by

\[g^{n}_{k}(t,u_{1},\ldots,u_{n})=\langle G^{*}(t,\sum_{j=1}^{n}u_{j}e_{j}),e_{k }\rangle_{L^{2}(\Lambda)},\quad k=1,\ldots,n.\]

Since \(G^{*}\) is continuous, the functions \((g^{n})_{n\in\mathbb{N}}\) are also continuous. In particular, it is possible to approximate these functions by simpler functions that can be treated numerically, e.g., artificial neural networks. In the following we consider for any \(n\in\mathbb{N}\) a set \(\mathcal{N}^{n}\) of Lipschitz continuous approximations, such that for all \(R>0\) there exists a sequence \((\psi^{n}_{m})_{m\in\mathbb{N}}\in\mathcal{N}^{n}\) with

\[\sup_{(t,x)\in[0,T]\times\mathcal{B}_{\mathrm{fa}}(0,R)}|\psi^{n}_{m}(t,x)-g^ {n}(t,x)|^{2}\to 0,\quad m\to\infty,\]

where

\[\mathcal{B}_{\mathrm{fa}}(0,R):=\{x\in\mathbb{R}^{n}||x|\leq R\}.\]

For a particular choice of \(\mathcal{N}^{n}\) we refer to our examples in Section VI. Then we define the ansatz space

\[\mathbb{U}:=\Big{\{}G(t,\sum_{j=0}^{\infty}u_{j}e_{j}):=\sum_{k=0 }^{n}\psi^{k}(t,\eta^{l}(u_{1},\ldots,u_{n}))e_{k}\\ \Big{|}\psi\in\mathcal{N}^{n},n,l\in\mathbb{N}\Big{\}},\]

where \(\eta^{l}:\mathbb{R}^{n}\to\mathbb{R}^{n}\)

\[\eta^{l}(x)=\begin{cases}x&|x|\leq l\\ I\frac{x}{|x|}&|x|>l\end{cases}\]

is a smooth cutoff function. It is not difficult to see that for any \(n\in\mathbb{N}\) and \(R>0\) there exists a sequence \((G^{n,m})_{m\in\mathbb{N}}\subset\mathbb{U}\) that satisfies a linear growth condition of the type

\[\|G^{n,m}(t,u)\|_{L^{2}(\Lambda)}\leq C_{n}(1+\|u\|_{L^{2}(\Lambda)}),\]

such that for any \(n\in\mathbb{N}\)

\[\lim_{m\to\infty}\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,R)} \|G^{n}(t,u)-G^{n,m}(t,u)\|^{2}_{L^{2}(\Lambda)}=0,\]

where

\[\mathcal{B}_{L^{2}(\Lambda)}(0,R):=\{u\in L^{2}(\Lambda)\|u\|_{L^{2}(\Lambda )}\leq R\}.\]

In particular \(\mathbb{U}\) satisfies a universal approximation property with respect to \(G^{*}\). We will give a short proof for this in Section VI where we consider artificial neural networks, mapping from \(\mathbb{R}^{n+1}\) to \(\mathbb{R}^{n}\), as an explicit example for Lipschitz continuous approximations \(\mathcal{N}^{n}\).

### Finite Dimensional Ansatz Spaces

Our second main result, Theorem 5, requires that the controls of our ansatz space take values in the Sobolev space \(H^{1}(\Lambda)\). Therefore we strengthen our assumption on the finite dimensional subspaces and assume that

\[S_{n}\subset H^{1}(\Lambda).\]

In order to obtain convergence rates, we also need to specify the rate of convergence of the orthogonal projections. Therefore, we assume that

\[\|P_{n}u-u\|_{L^{2}(\Lambda)}\leq\gamma_{n}\|u\|_{H^{1}(\Lambda)}, \tag{11}\]

for some \(\gamma_{n}\to 0\), as \(n\to\infty\). Furthermore, we assume that there exists an optimal control \(\mathfrak{g}^{*}\) in feedback form with a Lipschitz continuous feedback function \(G^{*}\). A simple example for such a situation is the linear quadratic case discussed in Example 1.

Since \(G^{*}\) is Lipschitz continuous, the functions \(g^{n}\) are also Lipschitz continuous. As we will see in the examples in Section VI, it is therefore possible to approximate \(g^{n}\) by artificial neural networks that have uniformly bounded Lipschitz constants. For any \(n\in\mathbb{N}\), let \((\mathcal{N}^{n,m})_{m\in\mathbb{N}}\) be a sequence of sets of Lipschitz continuous approximations, such that for all \(m\in\mathbb{N}\) there exists a \(\psi^{n,m}\in\mathcal{N}^{n,m}\) with Lipschitz constant independent of \(m\) and

\[\sup_{(t,x)\in[0,T]\times\mathcal{B}_{\mathrm{fa}}(0,R^{n}_{m})}|\psi^{n,m}(t, x)-g^{n}(t,x)|^{2}=:\varepsilon^{n}_{m}\to 0,\]

as \(m\to\infty\), for some sequence \((R^{n}_{m})_{m\in\mathbb{N}}\) of radii with \(R^{n}_{m}\to\infty\), as \(m\to\infty\). Then we define the sequence of ansatz spaces of
dimension \(n\in\mathbb{N}\) by

\[\mathbb{U}^{n,m}:=\Big{\{}G(t,\sum_{j=0}^{\infty}u_{j}e_{j}):=\sum_{k =0}^{n}\psi_{k}(t,u_{1},\ldots,u_{n})e_{k}\\ \Big{|}\psi\in\mathcal{N}^{n,m}\Big{\}}.\]

It is not difficult to observe that for any \(m\in\mathbb{N}\), there exists a Lipschitz continuous control \(G^{n,m}\in\mathbb{U}^{n,m}\) with Lipschitz constant independent of \(m\), such that

\[\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,R^{m}_{m})}\|P_{n}(G^{ n,m}(t,u)-G^{n}(t,u))\|_{L^{2}(\Lambda)}^{2}\to 0.\]

In particular, for any \(n\in\mathbb{N}\), the sequence \((\mathbb{U}^{n,m})_{m\in\mathbb{N}}\) satisfies the Lipschitz approximation property with respect to \(G^{*}\) in dimension \(n\).

## IV Main approximation results

Our first main result, Theorem 4, provides an approximation result for universal ansatz spaces of the type constructed in Section III. In the second main result, Theorem 5, we provide error estimates for ansatz spaces of fixed dimension.

### Universal Approximation

The following result shows that we can reach the optimal cost of the control problem when we consider ansatz spaces of the type constructed in Section III.

**Theorem 4**: _Assume that \(\mathbb{U}\subset U_{ad}\) satisfies the universal approximation property with respect to \(G^{*}\). Then_

\[\inf_{\mathfrak{g}\in\mathbb{A}}J(\mathfrak{g})=\inf_{G\in\mathbb{U}}J(G).\]

### Approximation with Ansatz Spaces of Fixed Dimensions

In Section VI, we construct as a particular example our ansatz space using artificial neural networks. By Theorem 4 we can get arbitrarily close to the optimal cost of the original control problem. However, at this point we do not know how close we can get to the optimal cost using ansatz spaces of a fixed size. Our second main result, Theorem 5, provides error estimates for the convergence of the cost of our approximation and the optimal cost. Similar to Section III.2, we consider finite dimensional subspaces

\[S_{n}\subset H^{1}(\Lambda).\]

and assume that for any \(u\in H^{1}(\Lambda)\) the orthogonal projections satisfy

\[\|P_{n}u-u\|_{L^{2}(\Lambda)}\leq\gamma_{n}\|u\|_{H^{1}(\Lambda)},\]

for some \(\gamma_{n}\to 0\), as \(n\to\infty\). Furthermore we assume that there exists an optimal control \(\mathfrak{g}^{*}\) in feedback form with a Lipschitz continuous feedback function \(G^{*}\), as already mentioned in Section III.2.

**Theorem 5**: _Let \(n\in\mathbb{N}\). Let \((\mathbb{U}^{n,m})_{m\in\mathbb{N}}\) be a sequence of subsets of \(U_{ad}\) that satisfies the Lipschitz approximation property with respect to \(G^{*}\) in dimension \(n\). Then it holds_

\[\inf_{G\in\mathbb{U}^{n,m}} J(G)-\inf_{\mathfrak{g}\in\mathbb{A}}J(\mathfrak{g})\] \[\leq C\left(1+\sqrt{\mathbb{E}\left[\int_{0}^{T}\|\mathfrak{g}^{* }_{H^{1}(\Lambda)}\mathrm{d}t\right]}\right)\gamma_{n}+C_{n}\sqrt{\varepsilon ^{n}_{m}},\]

_for some universal constant \(C_{n}\) independent of \(n\) and \(m\) and some constant \(C_{n}\) which is independent of \(m\). Here \(\varepsilon^{n}_{m}\) is given by (10)._

## V Gradient descent algorithm

In this section, we describe our gradient descent algorithm. Theorem 4 enables us to consider the approximating optimal control problem of minimizing optimal costs on a finite dimensional ansatz space \(\mathbb{U}^{n,m}\). We assume that

\[\mathbb{U}^{n,m}=\Big{\{}\Phi(\cdot,\cdot,\alpha)\Big{|}\alpha\in\mathbb{R}^{d _{m}}\Big{\}}\]

for a parametrization \(\Phi:[0,T]\times L^{2}(\Lambda)\times\mathbb{R}^{d_{m}}\to L^{2}(\Lambda)\). Replacing \(G(t,u)=\Phi(t,u,\alpha)\)

for some given \(\alpha\in\mathbb{R}^{d_{m}}\) leads to the state equation

\[\begin{cases}\mathrm{d}u^{\alpha}_{t}=[\Delta u^{\alpha}_{t}+\mathcal{F}(u^{ \alpha}_{t})+\Phi(t,u^{\alpha}_{t},\alpha)]\mathrm{d}t+\sigma\mathrm{d}W_{t} \\ u^{\alpha}_{0}=u\in L^{2}(\Lambda),\end{cases} \tag{12}\]

and the cost functional \(J:\mathbb{R}^{d_{m}}\to\mathbb{R}\),

\[J(\alpha)=\mathbb{E}\bigg{[}\int_{0}^{T}\int_{\Lambda}l(t,x,u^{ \alpha}_{t}(x))\mathrm{d}x+\frac{\nu}{2}\|\Phi(t,u^{\alpha}_{t},\alpha)\|_{L^{ 2}(\Lambda)}^{2}\mathrm{d}t\\ +\int_{\Lambda}m(x,u^{\alpha}_{T}(x))\mathrm{d}x\bigg{]}.\]

Using this parametrization, we obtain under suitable regularity assumptions on \(\Phi\) the following representation of the gradient of the cost functional:

\[\nabla J(\alpha)\] \[=\mathbb{E}\left[\int_{0}^{T}\nu\Phi^{*}_{\alpha}(t,u^{\alpha}_{ t},\alpha)\Phi(t,u^{\alpha}_{t},\alpha)+\Phi^{*}_{\alpha}(t,u^{\alpha}_{t}, \alpha)p_{t}\mathrm{d}t\right], \tag{13}\]

where \(p\) is the solution of the so-called adjoint equation

\[\begin{cases}\mathrm{d}p_{t}=-[(\Delta+\mathcal{F}^{*}(u^{\alpha}_{t})+\Phi^{ *}_{u}(t,u^{\alpha}_{t},\alpha))p_{t}+\mathcal{L}^{\prime}(t,u^{\alpha}_{t})\\ +\nu\Phi^{*}_{u}(t,u^{\alpha}_{t},\alpha)\Phi(t,u^{\alpha}_{t},\alpha)]\mathrm{d} t\\ p_{T}=\mathcal{M}^{\prime}(u^{\alpha}_{T}).\end{cases} \tag{14}\]
Here

\[\mathcal{L}(t,u)(x):=l(t,x,u(x)),\quad u\in L^{2}(\Lambda),(t,x)\in[0,T]\times\Lambda\]

denotes the Nemytskii operator associated with \(l\), and

\[\mathcal{M}(u)(x):=m(x,u(x)),\quad u\in L^{2}(\Lambda),x\in\Lambda\]

denotes the Nemytskii operator associated with \(m\). Furthermore, \(\Phi_{\alpha}(t,u,\alpha):\mathbb{R}^{d_{m}}\to L^{2}(\Lambda)\) (resp. \(\Phi_{u}(t,u,\alpha):L^{2}(\Lambda)\to L^{2}(\Lambda)\)) denotes the derivative of \(\Phi\) with respect to \(\alpha\) (resp. \(u\)), and \(\Phi^{\alpha}_{u}(t,u,\alpha)\) (resp. \(\Phi^{\alpha}_{u}(t,u,\alpha)\)) denotes the adjoint of \(\Phi_{\alpha}(t,u,\alpha)\) (resp. \(\Phi_{u}(t,u,\alpha)\)). For a derivation of the gradient, see the supplemental material.

Note that equation (14) is a linear backward PDE with random coefficients which are given by the state equation (12). For details concerning the numerical implementation, see the supplemental material.

**Example 6** (Artificial Neural Network): _In the case of an artificial neural network with activator function \(\theta\), the controls in the ansatz space \(\mathbb{V}^{n,m}\) can be parametrized as_

\[\Phi(t,u,\alpha)=\sum_{i=1}^{n}\left\langle C\theta\left(A\left(\frac{t}{ \pi_{n}u}\right)+a\right),e_{i}\right\rangle e_{i},\]

_where \(\alpha=(A,a,C)\) consists of \(A\in\mathbb{R}^{k\times(n+1)}\), \(a\in\mathbb{R}^{k}\), and \(C\in\mathbb{R}^{n\times k}\), for respective dimensions \(n=n_{m}\) and \(k=k_{m}\), and \(\pi_{n}u=(\langle u,e_{1}\rangle,\ldots,\langle u,e_{n}\rangle)^{\top}\)._

Based on this representation, we implement the following algorithm:

**Algorithm 7**: _Fix an initial control \(\alpha_{0}\), a stopping criterion \(\rho>0\), and a step size \(s>0\)._

1. _Solve the state equation (_12_) for one realization of the noise._
2. _Solve the adjoint equation (_14_) with the data given by the sample calculated in Step 1._
3. _Repeat Step 1 and Step 2 to approximate the gradient (_13_) using Monte Carlo approximation._
4. _Compute new control via_ \(\alpha_{n+1}=\alpha_{n}-s\nabla J(\alpha_{n})\)_._
5. _Stop if_ \(\|\nabla J(\alpha_{n})\|<\rho\)_._

## VI Artificial Neural Networks as Ansatz Spaces

In this section, we discuss explicit examples for ansatz spaces \(\mathbb{U}\subset U_{\text{ad}}\) that satisfy the assumptions of our main results, Theorem 4 and Theorem 5, and are suitable for the numerical implementation of our gradient descent algorithm. In the first part of this section, we focus on universal spaces and Theorem 4. The second part is devoted to ansatz spaces of fixed size and Theorem 5 with corresponding convergence rates. In the whole section, we consider the finite dimensional subspaces

\[S_{n}:=\text{span}\left\{1,\sqrt{2}\cos(k\pi\cdot)\Big{|}\,k=1,\ldots,n\right\} \subset H^{1}(0,1),\]

with orthonormal basis in \(L^{2}(\Lambda)\)

\[e_{0}=1,\quad e_{k}=\sqrt{2}\cos(k\pi\cdot),\quad k=1,\ldots,n.\]

In particular we have for any \(u\in H^{1}(0,1)\)

\[\|P_{n}u-u\|_{L^{2}(0,1)}^{2} =\sum_{k=n+1}^{\infty}|\langle u,\sqrt{2}\cos(k\pi\cdot)\rangle_ {L^{2}(0,1)}|^{2}\] \[\leq\frac{1}{n^{2}\pi^{2}}\sum_{k=n+1}^{\infty}|\langle\Delta^{1 }/2u,\sqrt{2}\cos(k\pi\cdot)\rangle_{L^{2}(0,1)}|^{2}\] \[\leq\frac{1}{n^{2}\pi^{2}}\|u\|_{H^{1}(0,1)}^{2}=:\gamma_{n}\|u \|_{H^{1}(0,1)}^{2},\]

where we used that the \(k\)-th eigenvalue of the Neumann Laplace operator with respect to the eigenfunction \(e_{k}\) is given by \(-\frac{1}{k^{2}\pi^{2}}\).

### Neural Network Approximation

Regarding Theorem 4, we will show in our first example that it is indeed sufficient to consider the type of ansatz space constructed in the first part of Section III, using 1-layer artificial neural networks for the approximating sets, to get arbitrarily close to the optimal cost. More precisely, we show that the set

\[\mathbb{U}:=\left\{G(t,\sum_{i=1}^{\infty}u_{i}e_{i})(x)=\sum_{i =1}^{n}\psi_{i}(t,\eta^{l}(u_{1},\ldots,u_{n}))e_{i}(x)\right.\\ \left.\left|\psi\in\mathcal{N}^{n},n,l\in\mathbb{N}\right\}\right.\]

satisfies the universal approximation property with respect to \(G^{*}\), where

\[\mathcal{N}^{n}:=\bigcup_{k=1}^{\infty}\mathcal{N}_{k}^{n}\]

and

\[\mathcal{N}_{k}^{n}:=\left\{\psi(x)=C\theta(Ax+a)\right.\\ \left.\big{|}\,A\in\mathbb{R}^{k\times n},a\in\mathbb{R}^{k},C \in\mathbb{R}^{n\times k}\right\}\]

denotes the set of all 1-layer artificial neural networks from \(\mathbb{R}^{n}\) to \(\mathbb{R}^{n}\) with \(k\) neurons, for a given non-polynomial, Lipschitz continuous activator function \(\theta\).

To this end, we recall the following classical universal approximation result by [35]:
**Theorem 8**: _Let \(\theta\in\mathcal{C}(\mathbb{R})\), then we define for \(u=(u_{1},\ldots,u_{d})\in\mathbb{R}^{d}\)_

\[\theta(u)^{i}:=\theta(u_{i}).\]

_If \(\theta\) is not polynomial, then for any \(n,m\in\mathbb{N}\), compact set \(K\subset\mathbb{R}^{n}\), \(h\in\mathcal{C}(K,\mathbb{R}^{n})\) and \(\varepsilon>0\), there exists \(k\in\mathbb{N},A\in\mathbb{R}^{k\times n},a\in\mathbb{R}^{k},C\in\mathbb{R}^ {m\times k}\), such that_

\[\sup_{u\in K}|h(u)-\psi(u)|<\varepsilon,\]

_where \(\psi\) is the \(1\)-layer artificial neural network_

\[\psi(u):=C\theta(Au+a).\]

Let \(n\in\mathbb{N}\) and recall from Section III the finitely based approximation of \(G^{*}\)

\[g^{n}:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}\] \[g^{n}_{i}(t,u):=\langle G^{*}(t,\sum_{j=1}^{n}u_{j}e_{j}),e_{i} \rangle,\quad i=1,\ldots,n.\]

Since \(g^{n}\) is continuous, there exists for any \(m\in\mathbb{N}\) a \(1\)-layer artificial neural network \(\psi^{n,m}:[0,T]\times\mathbb{R}^{n}\to\mathbb{R}^{n}\), such that

\[\sup_{(u,u)\in[0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)}|g^{n}(t,u)-\psi^{ n,m}(t,u)|<\frac{1}{m}.\]

Since \(G^{*}\) satisfies a linear growth condition, i.e.,

\[\|G^{*}(t,u)\|_{L^{2}(\Lambda)}\leq C(1+\|u\|_{L^{2}(\Lambda)}),\]

for some \(C>0\), we have

\[|g^{n}(t,u)| =\|P_{n}G^{*}(t,\sum_{i=1}^{n}u_{i}e_{i})\|_{L^{2}(\Lambda)}\] \[\leq\|G^{*}(t,\sum_{i=1}^{n}u_{i}e_{i})\|_{L^{2}(\Lambda)}\] \[\leq C(1+|u|),\]

where \(u=(u_{1},\ldots,u_{n})\). Hence, if we consider the continuous function \(\eta^{m}:\mathbb{R}^{n}\to\mathbb{R}^{n}\)

\[\eta^{m}(x)=\begin{cases}x&|x|\leq m\\ m\frac{x}{|x|}&|x|>m\end{cases}\]

and define

\[\psi^{n,m}(t,u):=\psi^{n,m}(t,\eta(u)),\]

then clearly \(\tilde{\psi}^{n,m}=\psi^{n,m}\) on \([0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)\) and for any \((t,u)\in[0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)\)

\[|\psi^{n,m}(t,u)| \leq|g^{n}(t,u)|+|\psi^{n,m}(t,u)-g^{n}(t,u)|\] \[\leq C(1+|u|)+1.\]

Furthermore, on \([0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)^{c}\) we have

\[|\tilde{\psi}^{n,m}(t,u)| =|\psi^{n,m}(t,m\frac{u}{|u|})|\] \[\leq|g^{n}(t,m\frac{u}{|u|})|+|\psi^{n,m}(t,m\frac{u}{|u|})-g^{n} (t,m\frac{u}{|u|})|\] \[\leq C(1+m)+1\] \[\leq C(1+|u|)+1.\]

Therefore \(\tilde{\psi}^{n,m}\) satisfies a linear growth condition with some constant \(C>0\) independent of \(m\). Now, we define

\[G^{n,m}(t,u):=\sum_{i=1}^{n}\tilde{\psi}^{n,m}_{i}(t,\langle u,e_{1}\rangle_{ L^{2}(\Lambda)},\ldots,\langle u,e_{n}\rangle_{L^{2}(\Lambda)}))e_{i}.\]

One can easily check that all the assumptions of Theorem 5 are satisfied for \(\mathbb{U}\). Indeed, due to the Lipschitz continuity of the elements in \(\mathcal{N}_{k}^{n}\), any control in \(\mathbb{U}\) is indeed admissible, i.e., \(\mathbb{U}\subset U_{\text{ad}}\). Furthermore, \((G^{n,m})_{m\in\mathbb{N}}\) is a sequence in \(\mathbb{U}\) that satisfies a linear growth condition with some constant \(C>0\) independent of \(m\) and approximates any finitely based \(G^{n}\). Indeed, for any \(R>0\) and any \(\varepsilon>0\) there exists an \(M\in\mathbb{N}\), such that \(\mathcal{B}_{L^{2}(\Lambda)}(0,R)\subset\mathcal{B}_{L^{2}(\Lambda)}(0,m)\) and \(\frac{1}{m}<\varepsilon\) for every \(m\geq M\). Therefore, we have for any \(m\geq M\)

\[\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,R)}\|G^{n, m}(t,u)-G^{n}(t,u)\|^{2}\] \[\leq\sup_{(t,u)\in[0,T]\times\mathcal{B}_{L^{2}(\Lambda)}(0,m)}\| \tilde{\psi}^{n,m}(t,u)-g^{n}(t,u)\|^{2}\] \[\leq\sup_{(t,u)\in[0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)}| \psi^{n,m}(t,u)-g^{n}(t,u)|^{2}\] \[\leq\sup_{(t,u)\in[0,T]\times\mathcal{B}_{\mathbb{R}^{n}}(0,m)}| \psi^{n,m}(t,u)-g^{n}(t,u)|^{2}\] \[<\frac{1}{m}<\varepsilon.\]

In the case of bounded controls, for example if \(\mathcal{U}=\mathcal{B}_{L^{2}(\Lambda)}(0,R)\), we could consider the ansatz space

\[\mathbb{U}:=\bigg{\{}G(t,\sum_{i=1}^{n}u_{i}e_{i})(x)=\sum_{i=1}^{ n}\psi_{i}(t,\eta^{l}(u_{1},\ldots,u_{n}))e_{i}(x)\] \[\qquad\qquad\qquad\left\|G(t,u)\right\|_{L^{2}(\Lambda)}\leq R \text{ where }\psi\in\mathcal{N}^{n},n,l\in\mathbb{N}\bigg{\}}\]

and the sequence

\[G^{n,m}(t,u)\] \[:=(1-\frac{1}{m(R+1)})\sum_{i=1}^{n}\tilde{\psi}^{n,m}_{i}(t, \langle u,e_{1}\rangle_{L^{2}(\Lambda)},\ldots,\langle u,e_{n}\rangle_{L^{2}( \Lambda)}))e_{i}.\]
### Convergence Rates

Next, we provide explicit convergence rates for the sequence of approximating spaces

\[\mathbb{U}^{n,m}:=\Big{\{}G(t,\sum_{i=1}^{\infty}u_{i}e_{i})(x)=\sum _{i=1}^{n}\psi_{i}(t,(u_{1},\ldots,u_{n}))e_{i}(x)\\ \Big{|}\psi\in\mathcal{N}_{m}^{n}\Big{\}}\]

using Theorem 5. We will mainly follow the ideas of [36]. In the following we consider a \(2\pi\)-periodic activator function \(\mathbf{\theta}:\mathbb{R}\to\mathbb{R}\) that satisfies \(\hat{\mathbf{\theta}}:=\int_{-\pi}^{\pi}\mathbf{\theta}(x)e^{-\mathrm{i}\mathrm{d}x}\neq 0\). Recall the following result from [36]:

**Proposition 9**: _Let \(K,L_{1},L_{2}>0\) and \(h:[0,T]\times\mathscr{B}_{\mathbb{R}^{n+1}}(0,R)\to\mathbb{R}^{n}\) be Lipschitz continuous in \((t,x)\) with \(\|h\|_{\mathcal{G}^{0}(0,T]\times\mathscr{B}_{\mathbb{R}^{n+1}}(0,R)}\leq K\) and Lipschitz constant bounded by \(L_{1}\). Furthermore we assume that \(h\) is twice differentiable in \(x\) and \(\partial_{x_{i}}h\) is Lipschitz continuous with Lipschitz constant bounded by \(L_{2}\). Then there exists a constant \(C>0\) depending only on the above constants, on \(n,T\) and on the activator function \(\mathbf{\theta}\) through \(\hat{\mathbf{\theta}}\), \(\|\mathbf{\theta}^{\prime}\|_{\mathcal{G}^{0}}\), \(\|\mathbf{\theta}^{\prime\prime}\|_{\mathcal{G}^{0}}\) and \(\|\mathbf{\theta}^{\prime\prime}\|_{\mathcal{G}^{0}}\), and there exists a constant \(m_{0}\) depending only on \(n\) with the following property. For every \(R>0\) and every \(m_{n}>m_{0}\), there exists a one-hidden layer artificial neural network \(\psi_{h}\in\mathcal{N}_{m_{n}}^{n}\) such that_

\[\|h-\psi_{h}\|_{\mathcal{G}^{0}([0,T]\times\mathscr{B}_{\mathbb{R}^{n}}(0,R); \mathbb{R}^{n})}\leq C(1+R)m_{in}^{-1/(2(n+1))}\]

_and such that the Lipschitz constants of \(\psi_{h},\partial_{x}\psi_{h}\) are at most \(C(1+Rm_{in}^{-1/(2(n+1))})\)._

If we now set \(R^{n}(m):=m^{1/(3(n+1))}\), then by Proposition 9 there exists for any sufficiently large \(m\in\mathbb{N}\) a network \(\psi^{n,m}\in\mathcal{N}_{m}^{n}\), such that for

\[G^{n,m}(t,\sum_{i=1}^{\infty}u_{i}e_{i})(x)=\sum_{i=1}^{n}\psi_{i}^{n,m}(t,(u_ {1},\ldots,u_{n}))e_{i}(x),\]

it holds

\[\sup_{(t,x)\in[0,T]\times\mathscr{B}_{L^{2}(\Lambda)}(0,R^{n}(m)) }\|P_{n}(G^{n,m}(t,x)-G^{n}(t,x))\|_{L^{2}(\Lambda)}^{2}\\ \leq m^{-1/(3(n+1))}\to 0.\]

Furthermore \(G^{n,m}\in\mathbb{U}^{n,m}\) is Lipschitz continuous with some Lipschitz constant independent of \(n\). Therefore, applying Theorem 5 yields

\[\inf_{G\in\mathbb{U}^{n,m}}J(G)-\inf_{\mathbf{\theta}\in\Lambda}J( \mathfrak{g})\\ \leq C\left(1+\sqrt{\mathbb{E}\left[\int_{0}^{T}\|\mathbf{\phi}_{t}^{* }\|_{H^{1}(\Lambda)}^{2}\mathrm{d}t\right]}\right)\frac{1}{n^{2}\pi^{2}}+C_{n }\sqrt{m^{-1/(3(n+1))}}.\]

## VII Numerical discretization of the control problem

In order to implement a numerical algorithm, the control problem needs to be discretized. In this section, we introduce the discretized version of our control problem and provide a bound for the error resulting from the numerical discretization.

**Problem 10** (Discretized Control Problem): _Let \(n\in\mathbb{N}\). Minimize_

\[J^{n}(G):=\mathbb{E}\left[\int_{0}^{T}\int_{\Lambda}l(t,x,u_{t}^{G,n}(x)) \mathrm{d}x+\frac{\nu}{2}\|P_{n}G(t,u_{t}^{G,n})\|_{L^{2}(\Lambda)}^{2} \mathrm{d}t\right]\\ +\mathbb{E}\left[\int_{\Lambda}m(x,u_{T}^{G,n}(x))\mathrm{d}x\right]\]

_over the set \(U_{\text{ad}}\) subject to the discretized SPDE_

\[\begin{cases}\mathrm{d}u_{t}^{G,n}=[-\Delta_{n}u_{t}^{G,n}+P_{n}\mathcal{F}(u_ {t}^{G,n})+P_{n}G(t,u_{t}^{G,n})]\mathrm{d}t+\sigma P_{n}\mathrm{d}W_{t}\\ u_{0}^{G,n}=P_{n}u\in S_{n},\end{cases}\]

_where \(\Delta_{n}x_{n}\) is defined as the unique element in \(S_{n}\) with_

\[\left\langle\nabla x_{n},\nabla y_{n}\right\rangle_{L^{2}(\Lambda)}=\left\langle \Delta_{n}x_{n},y_{n}\right\rangle_{L^{2}(\Lambda)}\quad\forall y_{n}\in S_{n}.\]

**Proposition 11**: _Under the same assumptions as in Theorem 5, it holds_

\[\inf_{G\in\mathbb{U}^{n,m}} J^{n}(G)-\inf_{\mathfrak{g}\in\mathbb{A}}J(\mathfrak{g})\] \[\leq C\left(1+\sqrt{\mathbb{E}\left[\int_{0}^{T}\|\mathbf{\phi}_{t}^{* }\|_{H^{1}(\Lambda)}^{2}\mathrm{d}t\right]}\right)\gamma_{n}+C_{n}\sqrt{\mathbf{ \varepsilon}_{m}^{n}},\]

_for some constant \(C_{n}>0\) that only depends on \(n\in\mathbb{N}\) and on the Lipschitz constant of \(G^{*}\)._

**Remark 12**: _Under additional assumptions, in particular convexity of the Hamiltonian, one can show that for any \(\mathbb{U}\subset U_{\text{ad}}\), it holds_

\[\inf_{\mathfrak{g}\in\mathbb{A}}J(\mathfrak{g})-\inf_{G\in\mathbb{U}}J^{n}(G) \leq C\gamma_{n},\]

_for some constant \(C>0\) which is independent of \(n\)._

## VIII Simulations

In this whole section, we consider equation (2) on some interval \(\Lambda=(0,L)\) of length \(L\). For the approximation we consider the Galerkin finite dimensional subspace

\[S_{n}:=\text{span}\left\{1,\sqrt{\frac{2}{L}}\cos\left(\frac{k}{L}\mathbf{\pi} \cdot\right)\middle|k=1,\ldots,n\right\}\subset H^{1}(0,L),\]

with orthonormal basis

\[e_{0}=\sqrt{\frac{1}{L}},\quad e_{k}=\sqrt{\frac{2}{L}}\cos\left(\frac{k}{L}\mathbf{ \pi}\cdot\right),\quad k=1,\ldots,n,\]
or the finite element subspace

\[\overline{S}_{n}:=\text{span}\left\{\overline{e}_{k}|\,k=1,\ldots,n\right\}\subset H ^{1}(0,L),\]

with basis

\[\overline{e}_{k}(x)=\begin{cases}n\left(x-\frac{k-1}{n}L\right)&\text{if }x\in\left[\frac{k-1}{n}L,\frac{k}{n}L\right]\\ n\left(\frac{k+1}{n}L-x\right)&\text{if }x\in\left[\frac{k}{n}L,\frac{k+1}{n}L \right]\\ 0&\text{otherwise}\end{cases}\]

for \(k=1,\ldots,n\).

### Heat Equation

In this subsection, we consider the controlled stochastic heat equation in order to validate our algorithm by comparing with the optimal feedback control obtained from the associated Riccati equation, see Example 1. The controlled state equation is given by

\[\begin{cases}\text{d}u_{0}^{\theta}=[\Delta u_{t}^{\theta}+\mathfrak{g}_{t}] \text{d}t+0.05\text{d}W_{t},\quad t\in[0,20]\\ u_{0}^{\theta}=u\in L^{2}(0,20),\end{cases} \tag{15}\]

where \(u=\mathbf{I}_{[20/3,40/3]}\). We consider the problem of steering the solution of the stochastic heat equation into the constant zero profile. To this end, we introduce the cost functional

\[J(\mathfrak{g})=\frac{1}{2}\mathbb{E}\left[\int_{0}^{20}\|u_{t}^{\theta}\|_{L ^{2}(0,20)}^{2}+\|\mathfrak{g}_{t}\|_{L^{2}(0,20)}^{2}\text{d}t\right].\]

Note that the second term is a regularization, which is necessary in linear quadratic control theory. We approximate the Riccati equation (8) numerically based on 400 Fourier coefficients to obtain the approximated optimal feedback control

\[\mathfrak{g}_{t}^{\text{Ric}}=P^{n}(t)u_{t}^{\theta^{\text{Ric}}}\]

and use this approximation as a benchmark for our gradient descent algorithm.

For the approximation of the optimal control, we use the ansatz space

\[\mathbb{U}^{n,k_{1}+k_{2}}:=\bigg{\{}G(t,\sum_{i=1}^{\infty}u_{i} e_{i})(x)=\sum_{i=1}^{n}\psi_{t}(t,(u_{1},\ldots,u_{n}))e_{i}(x)\\ \bigg{|}\psi\in\mathcal{N}_{k_{1},k_{2}}^{n}\bigg{\}}, \tag{16}\]

for \(k=k_{1}=k_{2}=100\), where

\[\mathcal{N}_{k,k}^{n}:=\bigg{\{}\psi(t,u)=C\theta\left(B\theta \left(A\begin{pmatrix}t\\ u\end{pmatrix}+a\right)+b\right)\\ \bigg{|}A\in\mathbb{R}^{k\times(n+1)},B\in\mathbb{R}^{k\times k },C\in\mathbb{R}^{(n+1)\times k},a,b\in\mathbb{R}^{k}\bigg{\}}\]

denotes the set of all 2-layer neural networks with \(k\) neurons in the first and second layer, and ReLU activator function \(\theta\). After about \(20,000\) iterations of our stochastic gradient descent algorithm, we end up with an approximated cost of \(J\approx 5.3\), and approximated \(L^{2}\)-distance of our approximation \(\mathfrak{g}_{t}^{\text{approx}}=G^{\text{approx}}(t,u_{t}^{G^{\text{approx}}})\) to the optimal control \(\mathfrak{g}^{\text{Ric}}\) given by

\[\mathbb{E}\left[\int_{0}^{20}\|\mathfrak{g}_{t}^{\text{Ric}}-\mathfrak{g}_{t} ^{\text{approx}}\|_{L^{2}(\Lambda)}^{2}\text{d}t\right]\approx 0.05.\]

Below, we display our simulation results. Figure 1 displays one realization of the uncontrolled stochastic heat equation. Figure 2 displays our neural network approximation of the optimal feedback control, and Figure 3 shows its impact when applied to the system (15). Figure 4 displays the optimal control obtained using the Riccati equation, and shows that our approximated feedback control is indeed qualitatively close.
### \(L^{2}\)-Feedback Control of the Nagumo Equation

In this example, we apply our algorithm to the controlled stochastic Nagumo equation

\[\begin{cases}\mathrm{d}u_{t}^{G}=\big{[}\Delta u_{t}^{G}-u_{t}^{G}(u_{t}^{G}- \frac{1}{2})(u_{t}^{G}-1)+G(t,u_{t}^{G})\big{]}\mathrm{d}t+0.05\mathrm{d}W_{t} \\ u_{0}^{G}=u\in L^{2}(0,20),\end{cases} \tag{17}\]

where \(u=\mathbf{1}_{[\bar{5},15]}\). We consider the problem of stabilizing a bump profile given by the solution to the uncontrolled deterministic Nagumo equation, i.e., equation (17) with \(G\equiv 0\) and without noise, see Figure 5. To this end, we introduce the cost functional

\[J(G) =\mathbb{E}\left[\int_{0}^{100}\|u_{t}^{G}-u_{t}^{0}\|_{L^{2}(0, 20)}^{2}+\frac{1}{2}\|G(t,u_{t}^{G})\|_{L^{2}(0,20)}^{2}\mathrm{d}t\right]\] \[\quad+\mathbb{E}\left[\|u_{100}^{G}-u_{100}^{0}\|_{L^{2}(0,20)}^ {2}\right]. \tag{18}\]

We consider again the ansatz space (16) with the same parameters as in the previous example. In this case, our optimal control achieves an approximated cost of \(J\approx 8.1\).

Figures 6 and 7 display two realizations of the uncontrolled stochastic Nagumo equation (17), i.e., \(G\equiv 0\). Without control, the bump is unstable and the noise pushes the solution to one of the stable steady states \(u\equiv 0\) or \(u\equiv 1\). Figure 8 displays one realization of the approximated optimal control and shows that the control mostly acts on the interface, but also reacts to the noise in the system. Figure 9 illustrates the impact of the approximated optimal control when applied to the system (17); it shows that our feedback control indeed stabilizes the bump.
### Nemytskii Feedback Control of the Nagumo Equation

In our final example, we consider again the controlled stochastic Nagumo equation (17) with the same cost functional (18). However, now we only consider feedback controls of Nemytskii-type, i.e., the control \(G\) is a Nemytskii operator

\[\mathcal{G}(t,u)(x)=g(t,x,u(x))\]

for some function \(g:[0,100]\times[0,20]\times\mathbb{R}\to\mathbb{R}\). This means that the control at point \(x\in\Lambda\) depends on \(u(x)\), the value of the solution at point \(x\), but not on the whole function \(u\in L^{2}(\Lambda)\) as in the previous example. This restriction to feedback controls of Nemytskii-type significantly reduces the computational complexity and therefore leads to a more efficient approximation.

For the approximation of the optimal control, we consider the following ansatz space of Gaussian radial basis function neural networks with \(m=40\) neurons:

\[\mathbb{U}^{n,m}\] \[:=\bigg{\{}G(t,u)(x)=\sum_{i=1}^{n}\sum_{j=1}^{m}\sum_{k=1}^{r} \alpha_{ijk}\mathds{1}_{j_{k-1}j_{k}}(t)e^{-\kappa|u(x)-\overline{u}_{j}|^{2} }\overline{e}_{i}(x)\] \[\bigg{|}\alpha_{ijk}\in\mathbb{R},\overline{u}_{j}\in\mathbb{R} \bigg{\}},\]

where \(\{0=t_{0}<t_{1}<\ldots<t_{r}=100\}\) and \(\kappa=6\). In this case, our optimal control achieves an approximated cost of \(J\approx 1.25\). Similar to the approximated control of Section VIII.2, Figure 10 shows that the control again mostly acts on the interface and also reacts to the noise in the system (compare with Figure 8). Figure 11 shows that the Nemytskii feedback control achieves a better result than the \(L^{2}\)-feedback control from Subsection VIII.2 (compare with Figure 9). Figure 12 displays the approximation of the feedback function \((x,u)\mapsto g(t,x,u)\) at time \(t=40\). Observe that the feedback function is only trained for profiles arising in simulations. In particular, the feedback function is not trained in the middle of the bump (\(x\approx 10\)) for \(u\approx 0\).
## Author Declarations

### Conflict of Interest

The authors have no conflicts to disclose.

### Author Contributions

Wilhelm Stannat: Mathematical theory (equal). Alexander Vogler: Mathematical theory (equal); Software. Lukas Wessels: Mathematical theory (equal).

## Data Availability

Software and simulation output freely available with publication on GitHub at [https://github.com/AVoglerTu/SFB91OFeedback](https://github.com/AVoglerTu/SFB91OFeedback).

## Appendix A Derivation of the Gradient of the Cost Functional

As discussed in Section V, we now consider the following approximating optimal control problem: Minimize

\[J(\alpha)=\mathbb{E}\Big{[}\int_{0}^{T}\int_{\Lambda}l(t,x,u_{t} ^{\alpha}(x))\mathrm{d}x+\frac{\nu}{2}\|\Phi(t,u_{t}^{\alpha},\alpha)\|_{L^{2} (\Lambda)}^{2}\mathrm{d}t\\ +\int_{\Lambda}m(x,u_{T}^{\alpha}(x))\mathrm{d}x\Big{]}.\] (A.1)

subject to

\[\begin{cases}\mathrm{d}u_{t}^{\alpha}=[\Delta u_{t}^{\alpha}+\mathcal{F}(u_{t }^{\alpha})+\Phi(t,u_{t}^{\alpha},\alpha)]\mathrm{d}t+\sigma\mathrm{d}W_{t} \\ u_{0}^{\alpha}=u\in L^{2}(\Lambda).\end{cases}\] (A.2)

In the following derivation of the gradient of the cost functional, we assume that \(l\) and \(m\) are differentiable with respect to \(u\), \(\mathcal{F}\) is differentiable, and \(\Phi\) is differentiable with respect to \(u\) and \(\alpha\). The derivation of the gradient of the cost functional follows along the same lines as the derivation in [32], however, in the present setting, the control enters the state equation (A.1) in a nonlinear fashion, which requires slight modifications of the arguments. We begin by differentiating the cost functional in some direction \(\beta\in\mathbb{R}^{d_{m}}\) which yields

\[\frac{\partial J(\alpha)}{\partial\beta}\] (A.3) \[=\mathbb{E}\Big{[}\int_{0}^{T}\langle\mathcal{L}^{\prime}(t,u_{t} ^{\alpha}),y_{t}^{\beta}\rangle+\nu\langle\Phi(t,u_{t}^{\alpha},\alpha),\Phi_{ \alpha}(t,u_{t}^{\alpha},\alpha)\beta\rangle\] \[+\nu\langle\Phi(t,u_{t}^{\alpha},\alpha),\Phi_{u}(t,u_{t}^{\alpha },\alpha)y_{t}^{\beta}\rangle\mathrm{d}t+\langle\mathcal{M}^{\prime}(u_{t}^{ \alpha}),y_{T}^{\beta}\rangle\Big{]},\]

where \(y^{\beta}\) is the solution of the linearized state equation

\[\begin{cases}\mathrm{d}y_{t}^{\beta}=[\Delta y_{t}^{\beta}+ \mathcal{F}^{\prime}(u_{t}^{\alpha})y_{t}^{\beta}+\Phi_{u}(t,u^{\alpha}, \alpha)y_{t}^{\beta}\\ \hskip 113.811024pt+\Phi_{\alpha}(t,u_{t}^{\alpha},\alpha)\beta]\mathrm{d}t\\ y_{0}^{\beta}=0.\end{cases}\]

Next, we introduce the adjoint state \(p\) as the solution of the adjoint equation

\[\begin{cases}\mathrm{d}p_{t}=-[(\Delta+\mathcal{F}^{\prime}(u_{t}^{\alpha})+ \Phi_{u}^{*}(t,u_{t}^{\alpha},\alpha))p_{t}+\mathcal{L}^{\prime}(t,u_{t}^{ \alpha})\\ +\nu\Phi_{u}^{*}(t,u_{t}^{\alpha},\alpha)\Phi(t,u_{t}^{\alpha},\alpha)] \mathrm{d}t\\ p_{T}=\mathcal{M}^{\prime}(u_{T}^{\alpha}).\end{cases}\]

A straightforward computation leads to the following adjoint state property:

\[\begin{split}\mathrm{d}(y_{t}^{\beta},p_{t})&=\langle y_{t }^{\beta},\mathrm{d}p_{t}\rangle+\langle p_{t},\mathrm{d}y_{t}^{\beta}\rangle \\ &=\big{[}-\nu\langle\Phi_{u}^{*}(t,u_{t}^{\alpha},\alpha)\Phi(t,u_{t }^{\alpha},\alpha),y_{t}^{\beta}\rangle-\langle\mathcal{L}^{\prime}(t,u_{t}^{ \alpha}),y_{t}^{\beta}\rangle\\ &\hskip 113.811024pt+\langle\Phi_{\alpha}(t,u_{t}^{\alpha}, \alpha)\beta,p_{t}\rangle\big{]}\mathrm{d}t.\end{split}\]

Integrating over \([0,T]\) and taking expectations, we obtain from equation (A.3) the desired representation of the gradient of the cost functional:

\[\begin{split}&\nabla J(\alpha)\\ &=\mathbb{E}\left[\int_{0}^{T}\nu\Phi_{\alpha}^{*}(t,u_{t}^{\alpha}, \alpha)\Phi(t,u_{t}^{\alpha},\alpha)+\Phi_{\alpha}^{*}(t,u_{t}^{\alpha},\alpha )p_{t}\mathrm{d}t\right].\end{split}\]
