Newton's methods for solving linear inverse problems with neural network
  coders
Neural networks functions are supposed to be able to encode the desired
solution of an inverse problem very efficiently. In this paper, we consider the
problem of solving linear inverse problems with neural network coders. First we
establish some correspondences of this formulation with existing concepts in
regularization theory, in particular with state space regularization, operator
decomposition and iterative regularization methods. A Gauss-Newton's method is
suitable for solving encoded linear inverse problems, which is supported by a
local convergence result. The convergence studies, however, are not complete,
and are based on a conjecture on linear independence of activation functions
and its derivatives.
Otmar Scherzer, Bernd Hofmann, Zuhair Nashed
# Newton's methods for solving linear inverse problems with neural network coders

###### Abstract

Neural networks functions are supposed to be able to encode the desired solution of an inverse problem very efficiently. In this paper, we consider the problem of solving linear inverse problems with neural network coders. First we establish some correspondences of this formulation with existing concepts in regularization theory, in particular with state space regularization, operator decomposition and iterative regularization methods. A Gauss-Newton's method is suitable for solving encoded linear inverse problems, which is supported by a local convergence result. The convergence studies, however, are not complete, and are based on a conjecture on linear independence of activation functions and its derivatives.
that the solution of Equation 1.2 is a natural image, or in other words that it can be represented as a combination of neural network functions, we get the operator equation

\[N(\vec{p}\,)=F\Psi(\vec{p}\,)=\mathbf{y}, \tag{1.3}\]

where \(\Psi:\vec{P}\to\mathbf{X}\) is the before mentioned nonlinear operator that maps neural network parameters to image functions. We call

* \(\mathbf{X}\) the _image space_ and
* \(\mathbf{Y}\) the _data space_, in accordance with a terminology, which we generated in [2, 1].
* \(\vec{P}\) is called the _parameter space_. We make a different notation for \(\vec{P}\), because it represents parametrizations, and is often considered a space of vectors below.

The advantage of this ansatz is that the solution of Equation 1.2 is sparsely coded. However, the price to pay is that the reformulated Equation 1.3 is nonlinear. Operator equations of the form of Equation 1.3 are not new: They have been studied in abstract settings for instance in the context of

* _state space regularization_[8] and
* in the context of the _degree of ill-posedness_[15, 20, 21, 24, 25] as well as of the _degree of nonlinearity_[24] of nonlinear ill-posed operator equations.
* Another related approach is _finite dimensional approximation_ of regularization in Banach spaces (see for instance [42]). Particularly, finite dimensional approximations of regularization methods with neural network functions (in the context of frames and _deep synthesis regularization_) have been studied in [38].

In this paper we aim to link the general regularization of the degree of ill-posedness and nonlinearity with coding theory. We investigate generalized Gauss-Newton's methods for solving Equation 1.3; Such methods replace the inverse of standard Newton's method by approximations of outer inverses (see [37]).

The outline of the paper is as follows: In Section 2 we review two decomposition cases as stated in [21] first, one of them is Equation 1.3. The study of decomposition cases follows the work on classifying inverse problems and regularization (see [33]). For operators associated to Equation 1.3, Newton's methods seem to be better suited than gradient descent methods, which we support by a convergence analysis (see Section 3). Section 3.4 is devoted to solving Equation 1.3, where \(\Psi\) is a shallow neural network synthesis operator.

## 2. Decomposition cases

We start with a definition for nonlinear operator equations possessing forward operators that are compositions of a linear and a nonlinear operator. Precisely, we distinguish between a first decomposition case (i), where the linear operator is the inner operator and the nonlinear is the outer one, and a second decomposition case (ii), where the nonlinear operator is the inner operator and the linear is the outer operator.

**Definition 2.1** (Decomposition cases): Let \(\vec{P},\mathbf{X},\mathbf{Y}\) be Hilbert-spaces.

1. An operator \(N\) is said to satisfy the _first decomposition case_ in an open, non-empty neighborhood \(\mathcal{B}(\vec{p}^{\,\dagger};\rho)\subseteq\vec{P}\) of some point \(\vec{p}^{\,\dagger}\) if there exists a linear operator \(F:\vec{P}\to\mathbf{X}\) and a nonlinear operator \(\Psi:\mathbf{X}\to\mathbf{Y}\) such that \[N(\vec{p}\,)=\Psi(F\vec{p}\,)\text{ for }\vec{p}\,\in\mathcal{B}(\vec{p}^{\, \dagger};\rho).\]
2. \(N\) is said to satisfy the _second decomposition case_ in a neighborhood \(\mathcal{B}(\vec{p}^{\,\dagger};\rho)\subseteq\vec{P}\) of some point \(\vec{p}^{\,\dagger}\) if there exists a linear operator \(F:\mathbf{X}\to\mathbf{Y}\) and a nonlinear operator \(\Psi:\vec{P}\to\mathbf{X}\) such that \[N(\vec{p}\,)=F\Psi(\vec{p}\,)\text{ for }\vec{p}\,\in\mathcal{B}(\vec{p}^{\, \dagger};\rho).\] (2.1) Typically it is assumed that the nonlinear operator \(\Psi\) is well-posed.
**Remark 2.2** (First decomposition case): In [21], this decomposition case has been studied under structural conditions, relating the second derivative of \(N\) with the first derivative. Under such assumptions convergence rates conditions (see [21, Lemma 4.1]) could be proven. The first decomposition case also arises in inverse option pricing problems in math finance (see [19] and [23, Sect.4]), where the ill-posed compact linear integration operator occurs as inner operator and a well-posed Nemytskii operator as outer operator.

**Remark 2.3** (Second decomposition case): Regularization methods for solving operator equations with operators satisfying the second order decomposition case, see Equation 2.1, were probably first analyzed in [8] under the name of _state space regularization_. They considered for instance Tikhonov-type regularization methods, consisting in minimization of

\[J_{\lambda}(\vec{p}\,)=\left\|F\Psi(\vec{p}\,)-\mathbf{y}\right\|_{\mathbf{Y}} ^{2}+\lambda\left\|\Psi(\vec{p}\,)-\tilde{\mathbf{x}}\right\|_{\mathbf{X}}^{ 2}, \tag{2.2}\]

where \(\tilde{\mathbf{x}}\) is a prior and \(\lambda>0\) is a regularization parameter. In [8] they derived estimates for the second derivative of \(J_{\lambda}(\vec{p}_{\lambda})(\mathbf{h},\mathbf{h})\), where \(\mathbf{h}\in\vec{P}\), that is for the curvature of \(J_{\lambda}\). If the curvature can be bounded from below by some terms \(\left\|\mathbf{h}\right\|_{\vec{p}}^{2}\) then, for instance, a locally unique minimizer of \(J_{\lambda}\) can be guaranteed and also domains can be specified where the functional is convex. Conditions, which guarantee convexity are called _curvature to size conditions_. Subsequently, these decomposition cases have been studied exemplarily in [21]. The theory developed there directly applies to Equation 1.3.

Instead of \(J_{\lambda}\) researchers often study direct regularization with respect to \(\vec{p}\,\). For instance in [13] functionals of the form

\[J_{\lambda}(\vec{p}\,)=\left\|F\Psi(\vec{p}\,)-\mathbf{y}\right\|_{\mathbf{Y} }^{2}+\lambda\mathcal{L}(\vec{p}\,), \tag{2.3}\]

where \(\mathcal{L}\) is some functional directly regularizing the parameter space. Typically \(\mathcal{L}\) is chosen to penalize for sparsity of parameters. The main difference between Equation 2.2 and Equation 2.3 is that in the prior regularization is performed with respect to the image space \(\mathbf{X}\) and in the later with respect to the parameter space \(\vec{P}\). Well-posedness of the functional \(J_{\lambda}\) in Equation 2.2 follows if \(F\circ\Psi\) is lower-semicontinuous, which in turn follows if \(\Psi\) is invertible.

In the following we study the solution of decomposable operator equations, such as Equation 1.3, with Gauss-Newton's methods. Decomposition cases have been used in the analysis of iterative regularization methods as well (see [29]):

**Definition 2.4** (Strong tangential cone condition): Let \(N:\mathcal{D}(N)\subset\vec{P}\to\mathbf{Y}\) with \(\mathcal{D}(N)\) its domain be a nonlinear operator.

* Then \(N\) is said to satisfy the strong tangential cone condition, originally introduced in [17], if \[N^{\prime}(\vec{p_{2}}\,)=R_{\vec{p}_{2},\vec{p}_{1}}N^{\prime}(\vec{p}_{1}) \text{ for all }\vec{p}_{1},\vec{p}_{2}\in\mathcal{D}(N).\] (2.4) where \[\left\|R_{\vec{p}_{2},\vec{p}_{1}}-I\right\|\leq C_{T}\left\|\vec{p}_{2}-\vec {p}_{1}\right\|_{\vec{P}}.\] (2.5)
* In [5] the _order reversed_ tangential cone condition, \[N^{\prime}(\vec{p}_{2}\,)=N^{\prime}(\vec{p}_{1})R_{\vec{p}_{2},\vec{p}_{1}} \text{ for all }\vec{p}_{1},\vec{p}_{2}\in\mathcal{D}(N),\] (2.6) together with Equation 2.5, has been introduced.

**Remark 2.5**: Equation 2.4 has been used for analyzing _gradient descent methods_ (see for instance [17, 29]). For the analysis of Newton's methods Equation 2.6 has been used (see [5, 29]).

The relation to the decomposition cases is as follows:

**Lemma 2.6**: _Let \(N:\mathcal{D}(N)\subseteq\vec{P}\to\mathbf{Y}\) with \(\mathcal{D}(N)=\mathcal{B}(\vec{p}^{\uparrow};\rho)\) satisfy the second decomposition case and assume that \(\Psi^{\prime}(\vec{p}\,)\) is invertible for \(\vec{p}\,\in\mathcal{D}(N)\). Then \(N\) satisfies Equation 2.6._

_Proof:_ If \(N\) satisfies the second decomposition case, Equation 2.1, then \(N^{\prime}(\vec{p}\,)=F\Psi^{\prime}(\vec{p}\,)\) for all \(\vec{p}\,\in\mathcal{D}(N)\). Note, that because \(F\) is defined on the whole space \(\mathbf{X}\), \(\mathcal{D}(N)=\mathcal{D}(\Psi)\). By using the invertability assumption on \(\Psi\) we get

\[N^{\prime}(\vec{p}_{2}\,)=F\Psi^{\prime}(\vec{p}_{2}\,)=F\Psi^{\prime}(\vec{p} _{1}\,)\underbrace{\Psi^{\prime}(\vec{p}_{1}\,)^{-1}\Psi^{\prime}(\vec{p}_{2}\,) }_{=:R_{\vec{p}_{2},\vec{p}_{1}}}=N^{\prime}(\vec{p}_{1})R_{\vec{p}_{2},\vec{p} _{1}},\]

which gives the assertion.
As we have shown, decomposition cases have been extensively studied in the regularization literature. One conclusion out of these studies is that the order reversed tangential cone condition Equation 2.6 is suitable for analyzing Newton's methods [5, 29] and thus in turn for the coded linear operator Equation 1.3 because of Lemma 2.6. The standard tool for analyzing Newton's methods is the Newton-Mysovskii condition as discussed below.

## 3 The Newton-Mysovskii Conditions

In this section we put abstract convergence conditions for Newton type methods in context with decoding. We consider first Newton's methods for solving the _general_ operator Equation 1.1. Decomposition cases of the operator \(N\) will be considered afterwards.

### Newton's method with invertible linearizations

For Newton's methods _local convergence_ is guaranteed under _Newton-Mysovskii_ conditions. For comparison reasons, we first recall a simple Newton's method analysis in finite dimensional spaces if the nonlinear operator has derivatives which are invertible. The proof of more general results, such as Theorem 3.6 below, applies here as well, and thus here the proof is omitted. Several variants of Newton-Mysovskii conditions have been proposed in the literature (see for instance [11, 12, 37]). Analysis of Newton's method was an active research area in the last century, see for instance [39, 46].

**Theorem 3.1** (Finite dimensional Newton's method): _Let \(N:\mathcal{D}(N)\subseteq\mathds{R}^{n}\to\mathds{R}^{n}\) be continuously Frechet-differentiable on a non-empty, open and convex set \(\mathcal{D}(N)\). Let \(\vec{p}^{\,\dagger}\in\mathcal{D}(N)\) be a solution of Equation 1.1. Moreover, we assume that_

1. \(N^{\prime}(\vec{p}\,)\) _is invertible for all_ \(\vec{p}\,\in\mathcal{D}(N)\) _and that_
2. _the_ Newton-Mysovskii condition _holds: That is, there exist some_ \(C_{N}>0\) _such that_ \[\begin{split}\big{\|}N^{\prime}(\vec{q}\,)^{-1}(N^{\prime}(\vec{p }+s(\vec{q}\,-\vec{p}\,))-N^{\prime}(\vec{p}\,))(\vec{q}\,-\vec{p}\,)\big{\|}_ {\vec{p}}\leq sC_{N}\,\|\vec{p}\,-\vec{q}\,\|_{\vec{P}}^{2}\\ \text{for all }\vec{p}\,,\vec{q}\,\in\mathcal{D}(N),s\in[0,1]. \end{split}\] (3.1)

_Let \(\vec{p}^{\,0}\in\mathcal{D}(N)\) which satisfies_

\[\overline{\mathcal{B}(\vec{p}^{\,0};\rho)}\subseteq\mathcal{D}(N)\text{ with }\rho:=\big{\|}\vec{p}^{\,\dagger}-\vec{p}^{\,0}\big{\|}_{\vec{P}}\text{ and }h:=\frac{\rho C_{I}C_{L}}{2}<1. \tag{3.2}\]

_Then the Newton's iteration with starting point \(\vec{p}^{\,0}\),_

\[\vec{p}^{\,k+1}=\vec{p}^{\,k}-N^{\prime}(\vec{p}^{\,k})^{-1}(N(\vec{p}^{\,k} )-\mathbf{y})\quad k\in\mathds{N}_{0}, \tag{3.3}\]

_satisfies that the iterates \(\big{\{}\vec{p}^{\,k}:k=0,1,2,\dots\big{\}}\) belong to \(\overline{\mathcal{B}(\vec{p}^{\,0};\rho)}\) and converge quadratically to \(\vec{p}^{\,\dagger}\in\overline{\mathcal{B}(\vec{p}^{\,0},\rho)}\)._

Now, we turn to the case that \(N\) is a decomposition operator.

### Newton-Mysovskii conditions with composed operator

Now, we study the case of convergence of Gauss-Newton's methods where \(N:\vec{P}\to\mathbf{Y}\) with \(\vec{P}=\mathds{R}^{n_{*}}\) and \(\mathbf{Y}\) is an infinite dimensional Hilbert space, where \(F:\mathbf{X}\to\mathbf{Y}\) is linear and bounded and \(\Psi:\vec{P}=\mathds{R}^{n_{*}}\to\mathbf{X}\). In this case the Moore-Penrose inverse, or even more general the outer inverse, replaces the inverse in a classical Newton's method (see Equation 3.3), because linearizations of \(N\) will not be invertible anymore as a simple count of dimensions show. We refer now to Gauss-Newton's methods if the linearizations might not be invertible to distinguish between classical Newton's methods also by name.

Before we phrase a convergence result for Gauss-Newton's methods we recall and introduce some definitions:

**Notation 3.2** (Inner, outer and Moore-Penrose inverse): (see [36, 34]) Let \(L:\vec{P}\to\mathbf{Y}\) be a linear and bounded operator mapping between two vector spaces \(\vec{P}\) and \(\mathbf{Y}\). Then
1. the operator \(B:\mathbf{Y}\to\vec{P}\) is called a _left inverse_ to \(L\) if \[BL=I\;.\]
2. \(B:\mathbf{Y}\to\vec{P}\) is called a _right inverse_ to \(L\) if \[LB=I\;.\] Left and right inverses are used in different context: * For a left inverse the nullspace of \(L\) has to be trivial, in contrast to \(B\). * For a right inverse the nullspace of \(B\) has to be trivial.
3. \(B:\vec{P}\to\vec{P}\) is called a _inverse_ to \(L\) if \(B\) is a right and a left inverse.
4. \(B:\vec{P}\to\mathbf{Y}\) is an _outer inverse_ to \(L\) if \[BLB=B.\] (3.4)
5. Let \(\vec{P}\) and \(\mathbf{Y}\) be Hilbert-spaces, \(L:\vec{P}\to\mathbf{Y}\) be a linear bounded operator. We denote the orthogonal projections \(P\) and \(Q\) onto \(\mathcal{N}(L)\), the nullspace of \(L\) (which is closed), and \(\overline{\mathcal{R}(L)}\), the closure of the range of \(L\): That is for all \(\vec{p}\in\vec{P}\) and \(\mathbf{y}\in\mathbf{Y}\) we have \[P\vec{p}=\operatorname{argmin}\left\{\|\vec{p}_{1}-\vec{p}\|_{\vec{p}}:\vec{ p}_{1}\in\mathcal{N}(L)\right\}\text{ and }Q\mathbf{y}=\operatorname{argmin}\left\{\|\mathbf{y}_{1}-\mathbf{y}\|_{ \mathbf{Y}}:\mathbf{y}_{1}\in\overline{\mathcal{R}(L)}\right\}.\] (3.5) We therefore have \[P:\vec{P} \to\mathcal{N}(L)\dot{+}\mathcal{N}(L)^{\perp}\] and \[Q:\mathbf{Y}\to\overline{\mathcal{R}(L)}\dot{+}\mathcal{R}(L)^{\perp}.\] \[\vec{p} \mapsto P\vec{p}+0 \mathbf{y}\to Q\mathbf{y}+0\] \[B:\mathcal{D}(B)\subseteq\mathbf{Y}\to\vec{P}\text{ with }\mathcal{D}(B):=\mathcal{R}(L)\dot{+} \mathcal{R}(L)^{\perp}\text{ is called the {\it Moore-Penrose inverse} of }L\text{ if the following identities hold }\] \[LBL =L,\] \[BL =B,\] (3.6) \[BL =I-P,\] \[LB =Q|_{\mathcal{D}(B)}.\]

In coding theory it is often stated that the range of a neural network operator \(\Psi\) forms a manifold in \(\mathbf{X}\), a space, which contains the natural images. This is the basis of the following definition making use of the Moore-Penrose inverse.

**Definition 3.3** (Lipschitz-differentiable immersion): Let \(\Psi:\mathcal{D}(\Psi)\subseteq\vec{P}=\mathds{R}^{n_{*}}\to\mathbf{X}\) where \(\mathcal{D}(\Psi)\) is open, non-empty, convex and \(\mathbf{X}\) is a seperable (potentially infinite dimensional) Hilbert-space.

1. We assume that \(\mathcal{M}:=\Psi(\mathcal{D}(\Psi))\) is a \(n_{*}\)-dimensional _submanifold_ in \(\mathbf{X}\): * Let for all \(\vec{p}=(p_{i})_{i=1}^{n_{*}}\in\mathcal{D}(\Psi)\) denote with \(\Psi^{\prime}(\vec{p}\,)\) the Frechet-derivative of \(\Psi\): \[\Psi^{\prime}(\vec{p}\,):\vec{P} \to\mathbf{X},\] \[\vec{q} =(q_{i})_{i=1}^{n_{*}} \mapsto\left(\partial_{p_{i}}\Psi(\vec{p}\,)\right)_{i=1,\dots,n_{ *}}\vec{q}\;.\] Here \(\left(\partial_{p_{i}}\Psi(\vec{p}\,)\right)_{i=1,\dots,n_{*}}\) denotes the vector of functions consisting of all partial derivatives with respect to \(\vec{p}\,\). In differential geometry notation this coincides with the _tangential mapping_\(T_{\vec{p}}\Psi\). However, the situation is slightly different here because \(\mathbf{X}\) can be infinite dimensional. * The _representation mapping_ of the derivative \[\Psi^{\prime}:\mathcal{D}(\Psi) \to\mathbf{X}^{n_{*}},\] \[\vec{p} \mapsto\left(\partial_{p_{i}}\Psi(\vec{p}\,)\right)_{i=1,\dots,n_{ *}}.\] has always the same rank \(n_{*}\) in \(\mathcal{D}(\Psi)\), meaning that all elements of \(\partial_{\vec{p}}\Psi(\vec{p}\,)\) are linearly independent. This assumption means, in particular, that \(\Psi\) is an _immersion_ and \(\mathcal{M}\) is a submanifold.
_._
2. _We define_ \[\begin{array}{c}P_{\vec{p}}:\mathbf{X}\rightarrow\mathbf{X}_{\vec{p}}:=\mbox{ span}\left\{\partial_{p_{i}}\Psi(\vec{p}\,):i=1,\ldots,n_{*}\right\},\\ \mathbf{x}\mapsto P_{\vec{p}}\,\mathbf{x}:=\mbox{argmin}\left\{\left\|\mathbf{ x}_{1}-\mathbf{x}\right\|_{\mathbf{X}}:\mathbf{x}_{1}\in\mathbf{X}_{\vec{p}} \right\}\end{array}\] (3.7) _as the projection from_ \[\mathbf{X}=\mathbf{X}_{\vec{p}}\,\dot{+}\mathbf{X}_{\vec{p}}^{\bot}\] _onto_ \(\mathbf{X}_{\vec{p}}\)_, which is well-defined by the closedness of the finite dimensional subspace_ \(\mathbf{X}_{\vec{p}}\)_._ _Next we define the inverse of_ \(\Psi^{\prime}(\vec{p}\,)\) _on_ \(\mathbf{X}_{\vec{p}}\)_:_ \[\begin{array}{c}\Psi^{\prime}(\vec{p}\,)^{-1}:\mbox{span}\left\{\partial_{p_ {i}}\Psi(\vec{p}\,):i=1,\ldots,n_{*}\right\}\rightarrow\vec{P},\\ \mathbf{x}=\sum_{i=1}^{n_{*}}x_{i}\partial_{p_{i}}\Psi(\vec{p}\,) \mapsto(x_{i})_{i=1}^{n_{*}}\end{array}\] _and consequently on_ \(\mathbf{X}\)__ \[\begin{array}{c}\Psi^{\prime}(\vec{p}\,)^{\dagger}:\mathbf{X}=\mathbf{X}_{ \vec{p}}\,\dot{+}\mathbf{X}_{\vec{p}}^{\bot}\rightarrow\vec{P},\\ \mathbf{x}=(\mathbf{x}_{1},\mathbf{x}_{2})\mapsto\Psi^{\prime}(\vec{p}\,)^{-1 }\mathbf{x}_{1}\end{array}\] (3.8) _which are both well-defined because we assume that_ \(\Psi\) _is an immersion. Note that_ \(x_{i}\)_,_ \(i=1,\ldots,n_{*}\) _are not necessarily the coordinates with respect to an orthonormal system in_ \(\mbox{span}\left\{\partial_{p_{i}}\Psi(\vec{p}\,):i=1,\ldots,n_{*}\right\}\)_._
3. _Finally, we assume that the operators_ \(\Psi^{\prime}(\vec{p}\,)\) _are locally bounded and locally Lipschitz-continuous in_ \(\mathcal{D}(\Psi)\)_. That is_ \[\begin{array}{c}\left\|\Psi^{\prime}(\vec{p}\,)-\Psi^{\prime}(\vec{q}\,) \right\|_{\vec{P}\rightarrow\mathbf{X}}\leq C_{L}\left\|\vec{p}-\vec{q}\, \right\|_{\vec{P}}\quad\left\|\Psi^{\prime}(\vec{p}\,)\right\|_{\vec{P} \rightarrow\mathbf{X}}\leq C_{I}\mbox{ for }\vec{p}\,,\vec{q}\in\mathcal{D}(\Psi). \end{array}\] (3.9) _If_ \(\Psi\) _satisfies these three properties we call it a Lipschitz-differentiable immersion._

The following lemma is proved by standard means:

**Lemma 3.4**: _For a Lipschitz-differentiable immersion_

* _the function_ \(\Psi^{\prime}(\vec{p}\,)^{\dagger}:\mathbf{X}\rightarrow\vec{P}\) _is in fact the Moore-Penrose inverse of_ \(\Psi^{\prime}(\vec{p}\,)\) _and_
* _for every point_ \(\vec{p}\,\in\mathcal{D}(\Psi)\subseteq\vec{P}\) _there exists a non-empty closed neighborhood where_ \(\Psi^{\prime}(\vec{p}\,)^{\dagger}\) _is uniformly bounded and it is Lipschitz-continuous; That is_ \[\left\|\Psi^{\prime}(\vec{p}\,)^{\dagger}-\Psi^{\prime}(\vec{q}\,)^{\dagger} \right\|_{\mathbf{X}\rightarrow\vec{P}}\leq C_{L}\left\|\vec{p}-\vec{q}\, \right\|_{\vec{P}},\quad\left\|\Psi^{\prime}(\vec{p}\,)^{\dagger}\right\|_{ \mathbf{X}\rightarrow\vec{P}}\leq C_{I}\mbox{ for }\vec{p}\,,\vec{q}\in \mathcal{D}(\Psi).\] (3.10)
* _Moreover, the operator_ \(P_{\vec{p}}\,\) _from Equation_ 3.7 _is bounded._

* We verify the four conditions Equation 3.6 with
* \(L=\Psi^{\prime}(\vec{p}\,):\vec{P}=\mathds{R}^{n_{*}}\rightarrow\mathbf{X}\), \(B=\Psi^{\prime}(\vec{p}\,)^{\dagger}:\mathbf{X}\rightarrow\vec{P}\), with \(\mathcal{D}(B)=\mathcal{D}(\Psi^{\prime}(\vec{p}\,)^{\dagger})=\mathbf{X}\) and
* \(P:\vec{P}\rightarrow\vec{P}\) the zero-operator and \(Q=P_{\vec{p}}:\mathbf{X}\rightarrow\mathbf{X}_{\vec{p}}\,\), the projection operator onto \(\mathbf{X}_{\vec{p}}\,\) (see Equation 3.7).
* First we prove the third identity with \(P=0\) in Equation 3.6: This follows from the fact that for all \(\vec{q}\,=(q_{i})_{i=1}^{n_{*}}\in\vec{P}\) we have \[\Psi^{\prime}(\vec{p}\,)^{\dagger}\Psi^{\prime}(\vec{p}\,)\vec{q}=\Psi^{\prime }(\vec{p}\,)^{-1}\left(\sum_{i=1}^{n_{*}}q_{i}\partial_{p_{i}}\Psi(\vec{p}\,) \right)=(q_{i})_{i=1}^{n_{*}}=\vec{q}\,.\] (3.11)
* For the forth identity we see that for all \(\mathbf{x}=(\mathbf{x}_{1},\mathbf{x}_{2})\in\mathbf{X}\) there exists \(x_{i}\), \(i=1,\ldots,n_{*}\) (because \(\partial_{p_{i}}\Psi(\vec{p}\,)\), \(i=1,\ldots,n_{*}\) is a basis) such that \[\mathbf{x}=\sum_{i=1}^{n_{*}}x_{i}\partial_{p_{i}}\Psi(\vec{p}\,)+\mathbf{x}_{2 }\mbox{ with }\mathbf{x}_{2}\in\mathbf{X}_{\vec{p}}^{\bot}\] and thus \[P_{\vec{p}}\,\mathbf{x}=\sum_{i=1}^{n_{*}}x_{i}\partial_{p_{i}}\Psi(\vec{p}\,)\] and therefore \[\Psi(\vec{p}\,)^{\dagger}\mathbf{x}=(x_{i})_{i=1}^{n_{*}}=\vec{x}.\] (3.12) Consequently, we have \[\Psi^{\prime}(\vec{p}\,)\Psi^{\prime}(\vec{p}\,)^{\dagger}\mathbf{x}=\Psi^{ \prime}(\vec{p}\,)\vec{x}=P_{\vec{p}}\,\mathbf{x}.\] (3.13)
* For the second identity we use that for all \(\mathbf{x}\in\mathbf{X}\) \[\Psi^{\prime}(\vec{p}\,)^{\dagger}\Psi^{\prime}(\vec{p}\,)\Psi^{\prime}(\vec{p} \,)^{\dagger}\mathbf{x}\underset{Equation~{}\ref{eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eqeq:eqeq:eqeq:eq:eq:eq:eqeq:eqeq:eqeq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eq:eqeq:eqeq:eqeq:eq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeqeq:eqeqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eqeqeq:eqeq:eqeqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eqeqeq:eqeqeq:eqeq:eqeqeq:eqeqeq:eqeqeq:eqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeq:
* _Decomposition property of the Moore-Penrose inverse:_ \[N^{\prime}(\vec{p}\,)^{\dagger}\mathbf{z}=\Psi^{\prime}(\vec{p}\,)^{\dagger}F^{- 1}\mathbf{z}\text{ for all }\vec{p}\,\in\mathcal{D}(N),\mathbf{z}\in\mathcal{R}(F)\subseteq \mathbf{Y}.\] (3.16) _In particular this means that_ \[N^{\prime}(\vec{p}\,)^{\dagger}N^{\prime}(\vec{p}\,)=I\text{ on }\mathds{R}^{n_{\star}}\text{ and }N^{\prime}(\vec{p}\,)N^{\prime}(\vec{p}\,)^{\dagger}=Q |_{\mathcal{R}(FP_{\vec{p}})},\] (3.17) _where_ \(I\) _denotes the identity operator on_ \(\mathds{R}^{n_{\star}}\) _and_ \(Q:\mathbf{Y}\rightarrow\overline{\mathcal{R}(FP_{\vec{p}})}^{\perp}\hat{+} \mathcal{R}(FP_{\vec{p}})^{\perp}\)_, respectively._
* _Generalized Newton-Mysovskii condition:_ \[\begin{split}\big{\|}N^{\prime}(\vec{p}\,)^{\dagger}(N^{\prime}( \vec{q}+s(\vec{p}-\vec{q}\,)-N^{\prime}(\vec{q}\,))(\vec{p}\,-\vec{q}\,) \big{\|}_{\vec{p}}\leq& sC_{I}C_{L}\,\|\vec{p}-\vec{q}\,\|_{ \vec{p}}^{2}\\ \vec{p}\,,\vec{q}\,\in&\mathcal{D}(N),s\in[0,1]\;. \end{split}\] (3.18) _We recall that the Lipschitz-constants_ \(C_{I}\) _and_ \(C_{L}\) _are defined in Equation_ 3.9_._

_Proof:_ First of all, we note that

\[N^{\prime}(\vec{p}\,)=F\Psi^{\prime}(\vec{p}\,)\text{ on }\mathcal{D}(\Psi)= \mathcal{D}(N).\]

To prove Equation 3.16 we have to verify Equation 3.6 with

\[L:=N^{\prime}(\vec{p}\,)=F\psi^{\prime}(\vec{p}\,):\vec{P}\rightarrow\mathbf{ Y}\text{ and }B:=\Psi^{\prime}(\vec{p}\,)^{\dagger}F^{-1}:\mathcal{R}(F)\subseteq \mathbf{Y}\rightarrow\vec{P}.\]

Note that since we assume that \(F\) has dense range we do not need to define and consider \(B\) on \(\mathcal{R}(F)\hat{+}\underbrace{\mathcal{R}(F)^{\perp}}_{=\{0\}}\).

Let us first state that with the notation of Equation 3.6 we have for fixed \(\vec{p}\,\):

\[\mathcal{D}(B)=\mathcal{D}(\Psi^{\prime}(\vec{p}\,)^{\dagger}F^{-1})= \mathcal{R}(F)\text{ and }\mathcal{R}(L)=\{F\Psi^{\prime}(\vec{p}\,)\vec{q}:\vec{q}\in \mathds{R}^{n_{\star}}\}=\mathcal{R}(FP_{\vec{p}}).\]

We use \(P\equiv 0\) in Equation 3.6.

In particular the first item shows that for \(\mathbf{z}=F\mathbf{x}=FP_{\vec{p}}\mathbf{x}+F(I-P_{\vec{p}})\mathbf{x}\) we have

\[Q\mathbf{z}=Q(FP_{\vec{p}}\mathbf{x}+F(I-P_{\vec{p}}\,)(\mathbf{x}))=FP_{\vec {p}}\mathbf{x}. \tag{3.19}\]

Applying Lemma 3.4 and the invertability of \(F\) on the range of \(F\) shows that

\[LBL=F\Psi^{\prime}(\vec{p}\,)\Psi^{\prime}(\vec{p}\,)^{\dagger}F^{-1}F\Psi^{ \prime}(\vec{p}\,) =F\Psi^{\prime}(\vec{p}\,)\Psi^{\prime}(\vec{p}\,)^{\dagger}\Psi^{ \prime}(\vec{p}\,)\underbrace{\rule{0.0pt}{12.9pt}}_{\text{Equation \ref{eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:
We have now all ingredients to prove a local convergence rates result for a Gauss-Newton's method, where the operator \(N\) is the composition of a linear bounded operator and a Lipschitz-differentiable immersions:

**Theorem 3.6**: _Let \(F:\mathbf{X}\to\mathbf{Y}\) be linear, bounded, with trivial nullspace and dense range. Moreover, let \(\Psi:\mathcal{D}(\Psi)\subseteq\vec{P}\to\mathbf{X}\) be a Lipschitz-differentiable immersion with \(\mathcal{D}(\Psi)\) open, non-empty, and convex. Moreover, \(N=F\circ\Psi:\mathcal{D}(\Psi)\to\mathbf{Y}\). We assume that there exist \(\vec{p}^{\,\dagger}\in\mathcal{D}(\Psi)\) that satisfies_

\[N(\vec{p}^{\,\dagger})=\mathbf{y}. \tag{3.20}\]

_Moreover, we assume that there exists \(\vec{p}^{\,0}\in\mathcal{D}(\Psi)\), which satisfies Equation 3.2. Then, the iterates of the Gauss-Newton's iteration,_

\[\vec{p}^{\,k+1}=\vec{p}^{\,k}-N^{\prime}(\vec{p}^{\,k})^{\dagger}(N(\vec{p}^{\, k})-\mathbf{y})\quad k\in\mathds{N}_{0} \tag{3.21}\]

_are well-defined elements in \(\overline{\mathcal{B}(\vec{p}^{\,0},\rho)}\) and converge quadratically to \(\vec{p}^{\,\dagger}\)._

_Proof:_ First of all note, that \(\mathcal{D}(\Psi)=\mathcal{D}(N)\) since \(F\) is defined all over \(\mathbf{X}\).

Let \(\rho=\left\|\vec{p}^{\,\dagger}-\vec{p}^{\,0}\right\|_{\vec{P}}\): We prove by induction that \(\vec{p}^{\,k}\in\overline{\mathcal{B}(\vec{p}^{\,\dagger};\rho)}\) for all \(k\in\mathds{N}_{0}\).

* For \(k=0\) the assertion is satisfied by assumption Equation 3.2.
* Let \(\vec{p}^{\,k}\in\overline{\mathcal{B}(\vec{p}^{\,\dagger};\rho)}\). Using the first condition of Equation 3.6, which a Moore-Penrose inverse satisfies, we see that \[N^{\prime}(\vec{p}^{\,k})N^{\prime}(\vec{p}^{\,k})^{\dagger}N^{\prime}(\vec{p} ^{\,k})(\vec{p}^{\,k+1}-\vec{p}^{\,\dagger})=N^{\prime}(\vec{p}^{\,k})(\vec{p }^{\,k+1}-\vec{p}^{\,\dagger}).\] The definition of Gauss-Newton's method, Equation 3.21, and Equation 3.20 then imply that \[N^{\prime}(\vec{p}^{\,k})(\vec{p}^{\,k+1}-\vec{p}^{\,\dagger})=N^{\prime}(\vec {p}^{\,k})N^{\prime}(\vec{p}^{\,k})^{\dagger}(N(\vec{p}^{\,\dagger})-N(\vec{p }^{\,k})-N^{\prime}(\vec{p}^{\,k})(\vec{p}^{\,\dagger}-\vec{p}^{\,k})),\] and consequently, using the third identity of Equation 3.6 (note that under the assumptions of this theorem \(P=0\), see the proof prior to Equation 3.11), the second identity of Equation 3.6 and that \(F\) is injective, we get \[\vec{p}^{\,k+1}-\vec{p}^{\,\dagger}=N^{\prime}(\vec{p}^{\,k})^{ \dagger}N^{\prime}(\vec{p}^{\,k})(\vec{p}^{\,k+1}-\vec{p}^{\,\dagger}) =N^{\prime}(\vec{p}^{\,k})^{\dagger}(N(\vec{p}^{\,\dagger})-N( \vec{p}^{\,k})-N^{\prime}(\vec{p}^{\,k})(\vec{p}^{\,\dagger}-\vec{p}^{\,k}))\] \[=\Psi^{\prime}(\vec{p}^{\,k})^{\dagger}(\Psi(\vec{p}^{\,\dagger} )-\Psi(\vec{p}^{\,k})-\Psi^{\prime}(\vec{p}^{\,k})(\vec{p}^{\,\dagger}-\vec{p} ^{\,k})).\] From the Newton-Mysovskii condition Equation 3.18 and Equation 3.2 it then follows that \[\begin{split}\left\|\vec{p}^{\,k+1}-\vec{p}^{\,\dagger}\right\|_{ \vec{P}}\leq\frac{C_{I}C_{L}}{2}\left\|\vec{p}^{\,k}-\vec{p}^{\,\dagger}\right\| _{\vec{P}}^{2}&\leq\frac{C_{I}C_{L}\rho}{2}\left\|\vec{p}^{\,k}- \vec{p}^{\,\dagger}\right\|_{\vec{P}}<\left\|\vec{p}^{\,k}-\vec{p}^{\,\dagger} \right\|_{\vec{P}}\\ &\text{ or }\left\|\vec{p}^{\,k+1}-\vec{p}^{\,\dagger}\right\|_{ \vec{P}}=\left\|\vec{p}^{\,k}-\vec{p}^{\,\dagger}\right\|_{\vec{P}}=0.\end{split}\] (3.22) This, in particular shows that \(\vec{p}^{\,k+1}\in\mathcal{B}(\vec{p}^{\,\dagger};\rho)\), thus the well-definedness of the Gauss-Newton's iterations in the closed ball.
* Using Equation 3.22 we then get, since \(h=C_{I}C_{L}\rho/2<1\), that \[\left\|\vec{p}^{\,k+1}-\vec{p}^{\,\dagger}\right\|_{\vec{P}}\leq h^{k+1}\left\| \vec{p}^{\,0}-\vec{p}^{\,\dagger}\right\|_{\vec{P}}\leq h^{k+1}\rho,\] which converges to \(0\) for \(k\to\infty\).
* Convergence and the first inequality of Equation 3.22 imply quadratic convergence. \(\Box\)

**Remark 3.7**: Based on the assumption of an immersion we have shown in Lemma 3.5 that \(\Psi(\vec{p}^{\,\dagger})^{\dagger}F^{-1}\) is the Moore-Penrose inverse of \(N=F\Psi(\vec{p}^{\,\prime})\). In order to prove (quadratic) convergence of Gauss-Newton's methods one only requires an _outer inverse_ (see Notation 3.2). Following [37] (see also [18]) the analysis of Gauss-Newton's method could be based on _outer inverses_, which is more general than for the Moore-Penrose inverse (compare Equation 3.4 and Equation 3.6). However, it is nice to actually see that \(N^{\prime}(\vec{p}^{\,\prime})^{\dagger}\) is a Moore-Penrose inverse, which is the novelty compared to the analysis of [37]. For excellent expositions on Kantorovich and Mysovskii theory see [30, 40, 46] - here we replace the Newton-Mysovskii conditions by properties of an immersion. For aspects related to Newton's methods for singular points see [10, 16]. For applications of generalized inverses in nonlinear analysis see [34, 35].
### Neural networks

We want to apply the decomposition theory to Gauss-Newton's methods for solving Equation 1.3, where \(\Psi\) is a _shallow neural network operator_.

**Definition 3.8** (Shallow neural network operator): Let \(N\in\mathds{N}\) be fixed. We consider the operator

\[\begin{split}\Psi:\vec{P}:=\mathds{R}^{N}\times\mathds{R}^{n \times N}\times\mathds{R}^{N}&\to C^{1}([0,1]^{n})\subseteq \mathbf{X}:=L^{2}([0,1]^{n}),\\ (\vec{\alpha},\mathbf{w},\vec{\theta})&\mapsto \left(\vec{x}\rightarrow\sum_{j=1}^{N}\alpha_{j}\sigma\left(\mathbf{w}_{j}^{T} \vec{x}+\theta_{j}\right)\right)\\ \text{where }\alpha_{j},\theta_{j}\in\mathds{R}\text{ and }\vec{x}, \mathbf{w}_{j}\in\mathds{R}^{n}.\end{split} \tag{3.23}\]

Note, that with our previous notation, for instance in Definition 3.3, we have \(n_{*}=(n+2)*N\).

We summarize the notation, because it is quite heavy:

1. [label=()]
2. \(\vec{\ }\) denotes a vector in \(\mathds{R}^{n}\) or \(\mathds{R}^{N}\),
3. \(\mathbf{w}\) denotes a matrix: The only exception is Definition 3.10, where it is a tensor. \(\mathbf{w}_{j}\) denotes a vector, aside from Definition 3.10, where it is again a tensor.

**Example 3.9** (Examples of activation functions): \(\sigma\) is called the _activation function_, such as

* the _sigmoid function_, defined by \[\sigma(t)=\frac{1}{1+\mathrm{e}^{-\frac{1}{2}t}}\text{ for all }t\in \mathds{R}.\] (3.24) Note, we omit the \(\varepsilon\) dependence for notational convenience.
* The _hyperbolic tangent_ \[t\rightarrow\tanh(t)=\frac{\mathrm{e}^{2t}-1}{\mathrm{e}^{2t}+1}.\] (3.25)
* The _ReLU_ activation function, \[\sigma(t)=\max\left\{0,t\right\}\text{ for all }t\in\mathds{R}.\] (3.26)
* The _step function_, which is the pointwise limit of the sigmoid function, with respect to \(\varepsilon\to 0\), \[\sigma(t)=\left\{\begin{array}{ll}0&\text{for }t<0,\\ \frac{1}{2}&\text{for }t=0,\text{ }t\in\mathds{R}.\\ 1&\text{for }t>0.\end{array}\right.\] (3.27)

We only consider shallow neural networks in contrast to _deep neural networks_, which consist of several layers of shallow neural networks (see for instance [26]):

Figure 1: Three different activation functions: Sigmoid, tanh and ReLU. O-th derivative green, first derivative pink with circles and with pink \(\times\) is the derivative multiplied by \(x\), 2nd derivative blue. The ReLU function is scaled by a factor \(1/10\) and the derivative is plotted in original form, derivative times \(x\) is again scaled by \(1/10\) for visualization purposes.
**Definition 3.10** (Deep neural networks): Let

\[\vec{P}_{l}:=\mathds{R}^{N_{l}}\times\mathds{R}^{n\times N_{l}}\times\mathds{R}^{ N_{l}}\text{ for }l=1,\ldots,L\text{ and }\vec{P}:=\prod_{l=1}^{L}\vec{P}_{l}.\]

Then a deep neural network consisting of \(L\) layers is written as

\[\begin{split}\Psi:\vec{P}&\to L^{2}([0,1]^{n}),\\ (\vec{\alpha}_{l},\mathbf{w}_{l},\vec{\theta}_{l})_{l=1}^{L}& \mapsto\left(\vec{x}\rightarrow\sum_{j_{L}=1}^{N_{L}}\alpha_{j_{L}}^{L} \sigma_{\varepsilon_{L}}^{L}\left(p_{j_{L},L}\left(\sum_{j_{L-1}=1}^{N_{L-1}} \cdots\left(\sum_{j_{1}=1}^{N_{1}}\alpha_{j_{1},1}\sigma_{\varepsilon_{1}}^{1} \left(p_{j_{1}}^{1}(\vec{x})\right)\right)\right)\right)\right),\end{split} \tag{3.28}\]

where

\[p_{j}^{i}(\vec{x})=\mathbf{w}_{j,i}^{T}\vec{x}+\theta_{j}^{i}\text{ with }\alpha_{j}^{i},\theta_{j}^{i}\in\mathds{R}\text{ and }\vec{x},\mathbf{w}_{j,i}\in\mathds{R}^{n}\text{ for all }i=1,\ldots,L.\]

Note that the values \(\varepsilon_{k}\), \(k=1,\ldots,L\) can be chosen differently for activation functions at different levels (cf. Equation 3.24).

The success of neural network is due to the universal approximation properties, proven for the first time in [9, 27]. The universal approximation result states that shallow neural networks are universal, that is, that each continuous function can be approximated arbitrarily well by a neural network function. We review this result now.

**Theorem 3.11** ([26]): _In dependence of the smoothness of the activation function \(\sigma\) there exist two classes of results._

* _Theorem 2 from_ _[_26_]__: Let_ \(\sigma:\mathds{R}\rightarrow\mathds{R}^{+}\) _be a_ **continuous, bounded and nonconstant** _function_. Then, for every function_ \(g\in C(\mathds{R}^{n})\) _and every_ \(\nu>0\)_, there exists a function_ \[\vec{x}\to G(\vec{x})=\sum_{j=1}^{N}\alpha_{j}\sigma(\mathbf{w}_{j}^{T} \vec{x}+\theta_{j})\qquad\text{ with }N\in\mathds{N},\alpha_{j},\theta_{j}\in \mathds{R},\mathbf{w}_{j}\in\mathds{R}^{n},\] (3.29) _satisfying_ \[|G(\vec{x})-g(\vec{x})|<\nu\text{ uniformly for all compact subsets }K\subseteq\mathds{R}^{n}.\]
* _Theorem 1 from_ _[_26_]__: Let_ \(\sigma:\mathds{R}\rightarrow\mathds{R}^{+}\) _be_ **unbounded and nonconstant**_. Then for every measure_ \(\mu\) _on_ \(\mathds{R}^{n}\) _and every constant_ \(\nu>0\) _and_ \(p\geq 1\) _there exists a function_ \(G\) _of the form Equation_ 3.29 _that satisfies_ \[\int_{\mathds{R}^{n}}|G(\vec{x})-g(\vec{x})|^{p}d\vec{x}<\nu.\]

The first result applies for instance to the sigmoid and hyperbolic tangent function (see Equation 3.24 and Equation 3.25). The second result applies to the ReLU function (see Equation 3.26). In particular all approximation properties also hold on the compact set \([0,1]^{n}\), which we are considering.

### Newton-Mysovskii condition with neural networks

In the following we verify Newton-Mysovskii conditions for \(\Psi\) being the encoder of Equation 3.23. First we calculate the first and second derivatives of \(\Psi\) with respect to \(\vec{\alpha},\mathbf{w}\) and \(\vec{\theta}\). The computations can be performed for deep neural network encoders as defined in Equation 3.28 in principle analogously, but are technically and notationally more complicated. To make the notation consistent we define

\[\vec{p}:=(\vec{\alpha},\mathbf{w},\vec{\theta})\in\mathds{R}^{N}\times \mathds{R}^{n\times N}\times\mathds{R}^{N}=\mathds{R}^{n_{*}}.\]

**Lemma 3.12**: _Let \(\sigma:\mathds{R}\rightarrow\mathds{R}^{+}\) be a two times differentiable function with uniformly bounded function values and first, second order derivatives, such as the sigmoid, hyperbolic tangent functions (see Figure 1)1. Then, the derivatives of \(\Psi\) with respect to the coefficients \(\vec{p}\) are given by the following formulas:_

Footnote 1: This assumption is actually too restrictive, and only used to see that \(\Psi\in L^{2}([0,1]^{n})\).
* _Derivative with respect to_ \(\alpha_{s}\)_,_ \(s=1,\ldots,N\)_:_ \[\frac{\partial\Psi}{\partial\alpha_{s}}[\vec{p}\,](\vec{x})=\sigma\left(\sum_{i= 1}^{n}w_{s}^{i}x_{i}+\theta_{s}\right)\text{ for }s=1,\ldots,N.\] (3.30)
* _Derivative with respect to_ \(w_{s}^{t}\) _where_ \(s=1,\ldots,N\)_,_ \(t=1,\ldots,n\)_:_ \[\frac{\partial\Psi}{\partial w_{s}^{t}}[\vec{p}\,](\vec{x})=\sum_{j=1}^{N} \alpha_{j}\sigma^{\prime}\left(\sum_{i=1}^{n}w_{j}^{i}x_{i}+\theta_{j}\right) \delta_{s=j}x_{t}=\alpha_{s}\sigma^{\prime}\left(\sum_{i=1}^{n}w_{s}^{i}x_{i}+ \theta_{s}\right)x_{t}\] (3.31)
* _Derivative with respect to_ \(\theta_{s}\) _where_ \(s=1,\ldots,N\)_:_ \[\frac{\partial\Psi}{\partial\theta_{s}}[\vec{p}\,](\vec{x})=\sum_{j=1}^{N} \alpha_{j}\sigma^{\prime}\left(\sum_{i=1}^{n}w_{j}^{i}x_{i}+\theta_{j}\right) \delta_{s=j}=\alpha_{s}\sigma^{\prime}\left(\sum_{i=1}^{n}w_{s}^{i}x_{i}+ \theta_{s}\right).\] (3.32)

_Note, that all the derivatives above are functions in \(\mathbf{X}=L^{2}([0,1]^{n})\). In particular, maybe in a more intuitive way, we have_

\[D\Psi[\vec{p}\,](\vec{x})\vec{h}=\left(\tfrac{\partial\Psi}{\partial\vec{a}} [\vec{p}\,](\vec{x})\quad\tfrac{\partial\Psi}{\partial\mathbf{w}}[\vec{p}\,]( \vec{x})\quad\tfrac{\partial\Phi}{\partial\theta}[\vec{p}\,](\vec{x})\right)^ {T}\vec{h}\text{ for all }\vec{h}=\left(\begin{matrix}\vec{h}_{\vec{a}}\\ \mathbf{h}_{\mathbf{w}}\\ \vec{h}_{\vec{b}}\end{matrix}\right)\in\mathds{R}^{n_{*}}\text{ and }\vec{x}\in\mathds{R}^{n}. \tag{3.33}\]

_Moreover, let \(s_{1},s_{2}=1,\ldots,N\), \(t_{1},t_{2}=1,\ldots,n\), then we have in a formal way:_

\[\frac{\partial^{2}\Psi}{\partial\alpha_{s_{1}}\partial\alpha_{s_{ 2}}}(\vec{x}) =0, \tag{3.34}\] \[\frac{\partial^{2}\Psi}{\partial\alpha_{s_{1}}\partial w_{s_{2}} ^{t_{1}}}(\vec{x}) =\sigma^{\prime}\left(\sum_{i=1}^{n}w_{s_{1}}^{i}x_{i}+\theta_{s_ {1}}\right)x_{t_{1}}\delta_{s_{1}=s_{2}},\] \[\frac{\partial^{2}\Psi}{\partial\alpha_{s_{1}}\partial\theta_{s_ {2}}}(\vec{x}) =\sigma^{\prime}\left(\sum_{i=1}^{n}w_{s_{1}}^{i}x_{i}+\theta_{s_ {1}}\right)\delta_{s_{1}=s_{2}},\] \[\frac{\partial^{2}\Psi}{\partial w_{s_{1}}^{t_{1}}\partial\theta_{ s_{2}}}(\vec{x}) =\alpha_{s_{1}}\sigma^{\prime\prime}\left(\sum_{i=1}^{n}w_{s_{1}}^{i}x_{i}+ \theta_{s_{1}}\right)x_{t_{1}}\delta_{s_{1}=s_{2}},\] \[\frac{\partial^{2}\Psi}{\partial\theta_{s_{1}}\partial\theta_{s_ {2}}}(\vec{x}) =\alpha_{s_{1}}\sigma^{\prime\prime}\left(\sum_{i=1}^{n}w_{s_{1}}^{i }x_{i}+\theta_{s_{1}}\right)\delta_{s_{1}=s_{2}},\]

_where \(\delta_{a=b}=1\) if \(a=b\) and \(0\) else, that is the Kronecker-delta._

The notation of directional derivatives with respect to parameters might be confusing. Note, that for instance \(\tfrac{\partial\Psi}{\partial\theta_{s}}[\vec{p}\,](\vec{x})\) denotes a direction derivative of the functional \(\Psi\) with respect to the variable \(\theta_{s}\) and this derivative is a function, which depends on \(\vec{x}\). The argument, where the derivative is evaluated is a vector. So in such a formula \(\theta_{s}\) has two different meanings. Notationaly differentiating between them would be exact but becomes quite unreadable.

**Remark 3.13**:
* In particular Equation 3.34 shows that \[\left(\vec{h}_{\vec{a}}\quad\mathbf{h}_{\mathbf{w}}\quad\vec{h}_{\vec{g}} \right)D^{2}\Psi[\vec{p}\,](\vec{x})\begin{pmatrix}\vec{h}_{\vec{a}}\\ \mathbf{h}_{\vec{b}}\\ \end{pmatrix}\text{ is continuous (for fixed }\vec{x})\text{ with respect to }\vec{p}\,.\] (3.35)
* We emphasize that under the assumptions of Lemma 3.12 the linear space (for fixed \(\vec{p}\,\)) \[\mathcal{R}(D\Psi[\vec{p}\,])=\left\{D\Psi[\vec{p}\,]\vec{h}:\vec{h}=(\vec{h} _{\vec{a}},\mathbf{h}_{\mathbf{w}},\vec{h}_{\vec{g}})\in\mathds{R}^{N\times(n+ 2)}\right\}\subseteq L^{2}([0,1]^{n}).\]
* In order to prove convergence of the Gauss-Newton's method, Equation 3.21, by applying Theorem 3.1, we have to prove that \(\Psi\) is a Lipschitz-continuous immersion. Below we lack proving one important property so far, namely, that \[\partial_{k}\Psi[\vec{p}\,],\quad k=1,\ldots,n_{*}=N(n+2)\] (3.36) are linearly independent functions. In this paper, this will remain open as a conjecture, and the following statements are valid modulo this conjecture.

In the following we survey some results on linear independence with respect to the coefficients \(\vec{\alpha},\mathbf{w},\vec{\theta}\) of the functions \(\vec{x}\to\sigma\left(\sum_{i=1}^{n}w_{s}^{i}x_{i}+\theta_{s}\right)\), which match the functions \(\vec{x}\to\frac{\partial\Psi}{\partial\alpha_{s}}[\vec{p}\,](\vec{x})\), that is with respect to the first \(N\) variables.

### Linear independence of activation functions and its derivatives

The universal approximation results from for instance [9, 27, 26] do not allow to conclude that neural networks function as in Equation 3.23 are linearly independent. Linear independence is a non-trivial research question: We recall a result from [31] from which linear independence of a shallow neural network operator, as defined in Equation 3.23, can be deduced for a variety of activator functions. Similar results on linear independence of shallow network functions based on sigmoid activation functions have been stated in [47, 28], but the discussion in [31] raises questions on the completeness of the proofs. In [31] it is stated that all activation functions from the _Pytorch library_[41] are linearly independent with respect to almost all parameters \(\mathbf{w}\) and \(\theta\).

**Theorem 3.14** ([31]): _For all activation functions_ HardShrink, HardSigmoid, HardTanh, HardSwish, LeakyReLU, PReLU, ReLU, ReLU6, RReLU, SoftShrink, Threshold_,_ LogSigmoid, Sigmoid, SoftPlus, Tanh, and TanShrink _and the_ PyTorch _functions_ CELU, ELU, SELU _the shallow neural network functions Equation 3.23 formed by randomly generated vectors \((\mathbf{w},\vec{\theta})\) are linearly independent._

**Remark 3.15**:
* Theorem 3.14 states that the functions \(\frac{\partial\Psi}{\partial\alpha_{s}}\) (taking into account Equation 3.30) are linearly independent for _almost all_ parameters \((\mathbf{w},\vec{\theta})\in\mathds{R}^{n\times N}\times\mathds{R}^{N}\). In other words, the first block of the matrix is \(D\Psi\) in Equation 3.33 consists of functions, which are linearly independent for almost all parameters \((\mathbf{w},\vec{\theta})\). For our results to hold we need on top that the functions \(\frac{\partial\Psi}{\partial w_{s}^{2}}\) and \(\frac{\partial\Psi}{\partial\theta_{s}}\) from the second and third block (see Equation 3.34) are linearly independent within the blocks, respectively, and also across the blocks. So far this has not been proven but can be conjectured already from Figure 1.
* For the sigmoid function we have _obvious symmetries_ because \[\sigma^{\prime}\left(\mathbf{w}_{j}^{T}\vec{x}+\theta_{j}\right)=\sigma^{ \prime}\left(-\mathbf{w}_{j}^{T}\vec{x}-\theta_{j}\right)\text{ for every }\mathbf{w}_{j}\in\mathds{R}^{n},\vec{ \theta}\in\mathds{R}^{N},\] (3.37) or in other words for the function \(\Psi\) from Equation 3.23 we have according to Equation 3.32 that \[\frac{\partial\Psi}{\partial\theta_{s}}[\vec{\alpha},\mathbf{w},\vec{\theta} ](\vec{x})=\alpha_{s}\sigma^{\prime}(\mathbf{w}_{j}^{T}\vec{x}+\theta_{j})= \alpha_{s}\sigma^{\prime}(-\mathbf{w}_{j}^{T}\vec{x}-\theta_{j})=\frac{ \partial\Psi}{\partial\theta_{s}}[\vec{\alpha},-\mathbf{w},-\vec{\theta}]( \vec{x})\] (3.38) or in other words \(\frac{\partial\Psi}{\partial\theta_{s}}[\vec{\alpha},\mathbf{w},-\vec{ \theta}]\) and \(\frac{\partial\Psi}{\partial\theta_{s}}[\vec{\alpha},-\mathbf{w},-\vec{ \theta}]\) are linearly dependent.

**Conjecture 3.16**: We define by \(\mathcal{D}(\Psi)\) a _maximal set of vectors_\((\vec{\alpha},\mathbf{w},\vec{\theta})\) such that the \(n_{*}=N\times(n+2)\) functions in \(\vec{x}\)

\[\vec{x}\to\frac{\partial\Psi}{\partial\alpha_{s}}[\vec{\alpha},\mathbf{w},\vec {\theta}](\vec{x}),\quad\vec{x}\to\frac{\partial\Psi}{\partial w_{s}^{t}}[ \vec{\alpha},\mathbf{w},\vec{\theta}](\vec{x}),\quad\vec{x}\to\frac{\partial \Psi}{\partial\theta_{s}}[\vec{\alpha},\mathbf{w},\vec{\theta}](\vec{x}),\quad s =1,\ldots,N,t=1,\ldots,n,\]

are linearly independent. We assume that \(\mathcal{D}(\Psi)\) is open and dense in \(\mathbf{X}\in L^{2}([0,1])^{2}\) The later is guaranteed by Theorem 3.11. Recall the discussion above: The differentiation variables and the arguments coincide notationally, but are different objects.

**Remark 3.17**:
* It can be conjectured that for every element from \(\mathcal{D}(\Psi)\) only one element in \(\mathds{R}^{n_{*}}\) exists, which satisfies _obvious symmetries_ such as formulated in Equation 3.38. These "mirrored" elements are a set of measure zero in \(\vec{P}\). We conjecture that this corresponds to the set of measure zero as stated in [31], which is derived with Fourier methods.
* Equation 3.34 requires that all components of the vector \(\vec{\alpha}\) are non-zero. This means in particular that for "sparse solutions", with less that \(n_{*}=N*(n+2)\) coefficients, convergence is not guaranteed, because of a locally degenerating submanifold. We consider the manifold given by the function \[F:\mathds{R}^{2} \to\mathds{R}^{2}.\] \[\begin{pmatrix}x\\ y\end{pmatrix} \mapsto\begin{pmatrix}xy\\ x^{2}+y^{2}\end{pmatrix}\] (3.39) Then \[\nabla F(x,y)=\begin{pmatrix}y&x\\ 2x&2y\end{pmatrix}.\] We have \(\det\nabla F(x,y)=2(y^{2}-x^{2})\), which vanishes along the diagonals in \((x,y)\)-space. That is of the diagonals the function is locally a submanifold (see Figure 2): 

### Local convergence of Gauss-Newton's method with coding networks

In the following we prove a local convergence result for a Gauss-Newton's method, for solving operator equations Equation 1.3 where \(F\) is complemented by a shallow neural network coder \(\Psi\). In order to apply Theorem 3.1 we have to verify that the shallow neural network operator (see Equation 3.23) is a Lipschitz-differentiable immersion.

**Lemma 3.18**: _Let \(F:\mathbf{X}=L^{2}([0,1]^{n})\to\mathbf{Y}\) be linear, bounded, with trivial nullspace and closed range, and let \(\sigma\) be strictly monotonic (like sigmoid or hyperbolic tangent) and satisfy the assumptions of Lemma 3.12. Moreover, assume that Conjecture 3.16 holds. Then_

* _For every element_ \(\vec{p}\,=(\vec{\alpha},\mathbf{w},\vec{\theta})\in\mathds{R}^{n_{*}}\) _in the maximal set_ \(\mathcal{D}(\Psi)\) _(see Conjecture_ 3.16_),_ \(\mathcal{R}(D\Psi[\vec{p}\,])\) _is a linear subspace of the space_ \(\mathbf{X}\) _of dimension_ \(n_{*}=N\times(n+2)\)_._
* _There exists an open neighborhood_ \(\mathcal{U}\subseteq\mathds{R}^{N\times(n+2)}\) _of vectors_ \((\vec{\alpha},\mathbf{w},\vec{\theta})\) _such that_ \(\Psi\) _is a Lipschitz-differentiable immersion in_ \(\mathcal{U}\)_._

_Proof:_ * It is clear that for each fixed \(\vec{p}\,,\,D\Psi[\vec{p}\,]\in L^{2}([0,1]^{n})\) because of the differentiability assumptions of \(\sigma\), see Equation 3.35. Conjecture 3.16 implies that \(\mathcal{R}(D\Psi[\vec{p}\,])\) is a linear subspaces of \(\mathbf{X}\) of dimension \(N\times(n+2)\) (note the elements are functions).
* \(D^{2}\Psi[\vec{p}\,]:\mathds{R}^{N\times(n+2)}\to L^{2}([0,1]^{n})\) is continuous (see Equation 3.35) since we assume that the activation function \(\sigma\) is twice differentiable. Now we consider a non-empty open neighborhood \(\mathcal{U}\) of a vector \(\vec{p}\), with a compact closure.

Figure 2: The function \(F\) from Equation 3.39. We have plotted \(F(x,y)\) via its polar coordinates, I.e. \(r=|F(x,y)|\) and \(\theta=\tan^{-1}\left(\frac{xy}{x^{2}+y^{2}}\right)\). The colors correspond to identical angles.
Then, from the continuity of \(D^{2}\Psi\) with respect to \(\vec{p}\), it follows that \(D\Psi\) is a Frechet-differentiable with Lipschitz-continuous derivative on \(\mathcal{U}\). In particular this means that item _(i)_ in Definition 3.3 holds. Moreover, Equation 3.9 holds for \(\Psi^{\prime}\). That is, there exists constants \(C_{L}\) and \(C_{I}\) such that

\[\|\Psi^{\prime}(\vec{p}\,)-\Psi^{\prime}(\vec{q}\,)\|_{\vec{P}\to\mathbf{Y}} \leq C_{L}\,\|\vec{p}\,-\vec{q}\,\|_{\vec{P}}\,\,\,\text{and}\,\,\,\|\Psi^{ \prime}(\vec{p}\,)\|_{\vec{P}\to\mathbf{Y}}\leq C_{I}\,\,\text{for}\,\,\vec{p} \,,\vec{q}\,\in\mathcal{D}(\Psi). \tag{3.40}\]

Note that \(\Psi^{\prime}(p)^{\dagger}\) as defined in Equation 3.8 is also uniformly bounded and Lipschitz-continuous as a consequence of Lemma 3.4.

**Theorem 3.19** (Local convergence of Gauss-Newton's method): _Let \(F:\mathbf{X}=L^{2}([0,1]^{n})\to\mathbf{Y}\) be a linear, bounded operator with trivial nullspace and dense range and let \(N=F\circ\Psi\), where \(\Psi:\mathcal{D}(\Psi)\subseteq\mathds{R}^{N\times(n+2)}\to\mathbf{X}\) is a shallow neural network operator generated by an activation function \(\sigma\) which satisfies the assumptions of Lemma 3.18 and Conjecture 3.16. Let \(\vec{p}^{\,0}\in\mathcal{D}(\Psi)\) be the starting point of the Gauss-Newton's iteration Equation 3.21 and let \(\vec{p}^{\,\dagger}\in\mathcal{D}(\Psi)\) be a solution of Equation 3.20, which satisfy Equation 3.2. Then the Gauss-Newton's iterations are locally, that is if \(\vec{p}^{\,0}\) is sufficiently close to \(\vec{p}^{\,\dagger}\), and quadratically converging._

_Proof:_ The proof is an immediate application of Lemma 3.18 to Theorem 3.6. \(\Box\)

**Remark 3.20**: We have shown that a nonlinear operator equation, where the operator is a composition of a linear compact operator and a shallow neural network operator, can be solved with a Gauss-Newton's method with guaranteed local convergence in the parameter space.

**Conclusion.** We have shown that Gauss-Newton's methods are efficient algorithms for solving linear inverse problems, where the solution can be encoded with a neural network. The convergence studies, however, are not complete, and are based on a conjecture on linear independence of activation functions and its derivatives.

**Acknowledgements.** This research was funded in whole, or in part, by the Austrian Science Fund (FWF) P 34981 - New Inverse Problems of Super-Resolved Microscopy (NIPSUM). For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. Moreover, OS is supported by the Austrian Science Fund (FWF), with SFB F68 "Tomography Across the Scales", project F6807-N36 (Tomography with Uncertainties). The financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development and the Christian Doppler Research Association is gratefully acknowledged. BH is supported by the German Science Foundation (DFG) under the grant HO 1454/13-1 (Project No. 453804957).
