Complex Critical Points of Deep Linear Neural Networks
We extend the work of Mehta, Chen, Tang, and Hauenstein on computing the
complex critical points of the loss function of deep linear neutral networks
when the activation function is the identity function. For networks with a
single hidden layer trained on a single data point we give an improved bound on
the number of complex critical points of the loss function. We show that for
any number of hidden layers complex critical points with zero coordinates arise
in certain patterns which we completely classify for networks with one hidden
layer. We report our results of computational experiments with varying network
architectures defining small deep linear networks using
HomotopyContinuation.jl.
Ayush Bharadwaj, Serkan Hoşten
# Complex critical points of deep linear neural networks

###### Abstract.

We extend the work of Mehta, Chen, Tang, and Hauenstein on computing the complex critical points of the loss function of deep linear neutral networks when the activation function is the identity function. For networks with a single hidden layer trained on a single data point we give an improved bound on the number of complex critical points of the loss function. We show that for any number of hidden layers complex critical points with zero coordinates arise in certain patterns which we completely classify for networks with one hidden layer. We report our results of computational experiments with varying network architectures defining small deep linear networks using _HomotopyContinuation.jl_.

## 1. Introduction

Machine learning applications through deep learning techniques have been enormously successful in areas ranging from natural language processing and object recognition to drug discovery [4, 11]. These techniques involve optimizing a non-convex loss (cost) function such as the mean squared error between the observed and predicted data from the underlying neural network. Typically, this is an NP-hard problem [6] where the loss function has numerous local minima. Though methods such as stochastic gradient descent are empirically observed to close in to _good enough_ local minima (see [1, 9, 17]), a theoretical understanding for the success of deep learning is incomplete.

A deep _linear_ neural network is a simpler model on which detailed analysis is possible [1]. They differ from more general deep neural networks in the activation function used: linear neural networks employ linear activation functions instead of typical nonlinear ones such as ReLU. Nevertheless, the loss function of the deep linear networks is non-convex, and these networks exhibit many characteristics found in deep nonlinear networks. They have been used as a testing ground for ideas in general artificial neural networks [2].

Our starting point is the work of Mehta, Chen, Tang, and Hauenstein [14] in which the authors employ techniques from (numerical)
Introduction

The study of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of the local structure of local structure of the local structure of the local structure of the local structure of local structure of the local structure
minimize the loss function:

\[\mathcal{L}(W)=\frac{1}{2}\sum_{i=1}^{m}\lVert W_{H+1}W_{H}\cdots W_{1}x_{i}-y_{i }\rVert_{2}^{2}.\]

We will follow [14] to regularize the loss function by using regularization matrices \(\Lambda_{1},\ldots,\Lambda_{H+1}\) where each \(\Lambda_{i}\) is the same size as \(W_{i}\). We arrive at the regularized loss function

\[\mathcal{L}^{\Lambda}(W)=\mathcal{L}(W)+\frac{1}{2}\left(\lVert\Lambda_{1} \circ W_{1}\rVert_{F}^{2}+\cdots+\lVert\Lambda_{H+1}\circ W_{H+1}\rVert_{F}^{ 2}\right) \tag{1}\]

where \(\Lambda_{i}\circ W_{i}\) denotes the entrywise (Hadamard) product of \(\Lambda_{i}\) and \(W_{i}\). This choice of regularization is justified by the results (see Theorem 1 and Theorem 2 in [14]) that for almost all choices of the regularization matrices the solutions to the gradient system \(\nabla\mathcal{L}^{\Lambda}=0\) are isolated and nondegenerate.

### Our contributions

This paper is about the complex critical points of \(\mathcal{L}^{\Lambda}\), namely, the solutions to \(\nabla\mathcal{L}^{\Lambda}=0\). This gradient system is a system of \(N\) polynomial equations in \(N\) variables (weights) where each polynomial has degree \(2H+1\). This set up opens the door to using (numerical) algebraic geometry to bound the number of critical points as well as to compute them.

In Section 2 we will summarize the gradient equations for the regularized loss function of deep linear neural networks. In Section 3 we will prove a bound on the number of complex critical points of the loss function of a linear neural network with one hidden layer trained on a single data point. We achieve this bound by rewriting the gradient equations. Our new bound is much better than the classic Bezout bound as well as the BKK bound previously observed [14, Table I] and confirmed and expanded by us (see Table 1 in the Appendix).

Section 4 is devoted to understanding critical points where certain weights are zero. We show that, for any deep linear network trained on a single data point, if a weight is zero, then the entire row or column containing that weight must be zero. We also prove that row \(k\) of the weight matrix \(W_{i}\) is entirely zero if and only if the column \(k\) of the weight matrix \(W_{i+1}\) is entirely zero. As a corollary, we completely classify the zero patterns of complex critical points for linear networks with one hidden layer trained on a single data point.

In Section 5 we turn to our computational experiments. Using _HomotopyContinuation.jl_[7] we solve the gradient equations for different parameter values of \(d_{x},d_{y}\) and \(d_{i}\) when \(H,m\in\{1,2\}\). These extend the computational horizon beyond the one set in [14]. We believe the extended data we provide will inspire further results.
## 2. Gradient Equations for the Loss Function

The critical points of the regularized loss function \(\mathcal{L}^{\Lambda}(W)\) are the solutions to \(\nabla\mathcal{L}^{\Lambda}=0\), the gradient equations. We derive the gradient equations in a compact form.

Let \(W=W_{H+1}W_{H}\cdots W_{1}\), and let \(U_{i}^{T}=\prod_{j=i+1}^{H+1}W_{j}^{T}\) and \(V_{i}^{T}=\prod_{j=1}^{i-1}W_{j}^{T}\). Then

\[\frac{\partial\mathcal{L}^{\Lambda}}{\partial W_{i}}=U_{i}^{T}\left(W\left( \sum_{k=1}^{m}x_{k}x_{k}^{T}\right)-\left(\sum_{k=1}^{m}y_{k}x_{k}^{T}\right) \right)V_{i}^{T}+\Lambda_{i}\circ W_{i} \tag{2}\]

where the \((j,k)\) entry of the matrix \(\frac{\partial\mathcal{L}^{\Lambda}}{\partial W_{i}}\) is the partial derivative of \(\mathcal{L}^{\Lambda}\) with respect to the \((j,k)\) entry of \(W_{i}\).

It is easy to observe that the resulting system consists of \(N\) polynomials in \(N\) variables. The constant term in each polynomial is zero, therefore \(0\in\mathbb{C}^{N}\) is always a critical point. The terms of highest degree in every polynomial have degree \(2H+1\). This leads to an explicit classical Bezout bound (CBB).

**Proposition 1**.: _[_14_, Proposition 3]_ _The regularized loss function \(\mathcal{L}^{\Lambda}\) has at most \((2H+1)^{N}\) complex critical points._

**Example 2**.: Let us consider a 2-layer network with \(W_{1}=[\alpha_{1}\ \alpha_{2}]\), \(W_{2}=\left[\begin{smallmatrix}\beta_{1}\\ \beta_{2}\end{smallmatrix}\right]\), \(X=\left[\begin{smallmatrix}1&2\\ 3&4\end{smallmatrix}\right]\), \(Y=\left[\begin{smallmatrix}1&3\\ 2&4\end{smallmatrix}\right]\), \(\Lambda_{1}=\left[\begin{smallmatrix}4&-3\end{smallmatrix}\right]\), and \(\Lambda_{2}=\left[\begin{smallmatrix}-2\\ 5\end{smallmatrix}\right]\). Using these in (2), we get the gradient system

\[5\alpha_{1}\beta_{1}^{2}+5\alpha_{1}\beta_{2}^{2}+11\alpha_{2} \beta_{1}^{2}+11\alpha_{2}\beta_{2}^{2}-7\beta_{1}-10\beta_{2}+4\alpha_{1} =0\] \[11\alpha_{1}\beta_{1}^{2}+11\alpha_{1}\beta_{2}^{2}+25\alpha_{2} \beta_{1}^{2}+25\alpha_{2}\beta_{2}^{2}-15\beta_{1}-22\beta_{2}-3\alpha_{2} =0\] \[5\alpha_{1}^{2}\beta_{1}+22\alpha_{1}\alpha_{2}\beta_{1}+25 \alpha_{2}^{2}\beta_{1}-7\alpha_{1}-15\alpha_{2}-2\beta_{1} =0\] \[5\alpha_{1}^{2}\beta_{2}+22\alpha_{1}\alpha_{2}\beta_{2}+25 \alpha_{2}^{2}\beta_{2}-10\alpha_{1}-22\alpha_{2}+5\beta_{2} =0\]

As the above example indicates the gradient equations are sparse. This allows the use of the BKK bound for the number of critical points in \((\mathbb{C}^{*})^{N}\) as well as a modified BKK bound for the number of critical points in \(\mathbb{C}^{N}\)[12, 15]. For instance, the BKK bound for the solutions in \(\mathbb{C}^{4}\) for the gradient equations in Example 2 is 33. The actual number of solutions in \((\mathbb{C}^{*})^{4}\) and \(\mathbb{C}^{4}\) are 16 and 17, respectively. The Bezout bound gives \(3^{4}=81\). Our computational experiments in Section 5 (also see [14]) will show that generally all of these bounds are far from the actual number of critical points. Starting from the next section we take a closer look into improving the BKK bound. We close this section with the following result.
**Proposition 3**.: _Each polynomial in the gradient equations (2) has the same monomial support for generic input and output vectors as \(m\), the number of these vectors, varies while the rest of the parameters, namely, \(H\), \(d_{x}\), \(d_{y}\), and \(d_{i}\), \(i=1,\ldots,H\) stay constant. In particular, BKK bounds on the number of complex critical points for these systems in \((\mathbb{C}^{*})^{N}\) and \((\mathbb{C})^{N}\) are the same bounds for any \(m\)._

Proof.: First, we replace in (2) the matrix \(\sum_{k=1}^{m}x_{k}x_{k}^{T}\) with the \(d_{x}\times d_{x}\) symmetric matrix of indeterminates \(Z=(z_{ij})\), and the matrix \(\sum_{k=1}^{m}y_{k}x_{k}^{T}\) with the \(d_{y}\times d_{x}\) matrix of indeterminates \(T=(t_{ij})\). Multiplying out the matrices in (2) given polynomials in the entries of \(W_{1},\ldots,W_{H+1}\) with coefficients in \(z_{ij}\) and \(t_{ij}\). These polynomials have terms only in degrees \(2H+1\), \(H\), and \(1\). The terms of degree \(2H+1\) that actually appear in a polynomial have coefficients that are linear in the entries of \(Z\), and the terms of degree \(H\) that appear have coefficients that are linear in the entries of \(T\). Finally, the single monomial of degree one that appears in a polynomial has an entry of some \(\Lambda_{i}\) as a coefficient. We claim that for generic \(x_{1},\ldots,x_{m}\) and \(y_{1},\ldots,y_{m}\) the linear coefficient polynomials in \(Z\) or \(T\) do not vanish. This follows from the fact that determinantal varieties of \(d_{x}\times d_{x}\) symmetric matrices and determinantal varieties of \(d_{y}\times d_{x}\) matrices of rank \(\leq r\) are not contained in linear subspaces. Genericity of \(\Lambda_{i}\) guarantees that the required monomial of degree one is also present. 

## 3. Networks with one hidden layer trained on one data point

This section provides evidence that, by carefully looking at the gradient equations, one can gain insight into the number of critical points of the regularized loss function. We will treat linear neural networks with one hidden layer trained on a single data point (\(H=1\) and \(m=1\)) with no restrictions on the dimensions of the input and output vectors (\(d_{x}=n\) and \(d_{y}=p\)) as well as on the number of neurons in the hidden layer (\(d_{1}=d\)). Although this may be viewed unrealistic for real world applications it is meant to be a point of departure for more work from symbolic and numerical algebraic geometry.

To simplify the exposition we let \(A=W_{1}\) and \(B=W_{2}\). We also use \(\Lambda=\Lambda_{1}\) and \(\Sigma=\Lambda_{2}\). If we declare \(S_{i}=\sum_{i=1}^{n}a_{ij}x_{j}\) for \(i=1,\ldots,d\) and \(R_{k}=\sum_{i=1}^{d}b_{ki}S_{i}-y_{k}\) for \(k=1,\ldots,p\), then

\[Wxx^{T}-yx^{T}=\begin{pmatrix}R_{1}x_{1}&\cdots&R_{1}x_{n}\\ \vdots&&\vdots\\ R_{p}x_{1}&\cdots&R_{p}x_{n}\end{pmatrix}.\]
Now \(B^{T}(Wxx^{T}-yx^{T})+\Lambda\circ A=0\) yields

\[\left(\sum_{k=1}^{p}b_{k1}R_{k}\right)x_{j} =-\lambda_{1j}a_{1j}\qquad j=1,\ldots,n\] \[\vdots\] \[\left(\sum_{k=1}^{p}b_{kd}R_{k}\right)x_{j} =-\lambda_{dj}a_{dj}\qquad j=1,\ldots,n \tag{3}\]

Similarly, \((Wxx^{T}-yx^{T})A^{T}+\Sigma\circ B=0\) gives rise to

\[R_{k}S_{i}=-\sigma_{ki}b_{ki}\quad\ i=1,\ldots,d\quad\ k=1,\ldots,p. \tag{4}\]

**Lemma 4**.: _For generic input and output vector \(x\) and \(y\) and generic regularization matrices \(\Lambda\) and \(\Sigma\) we obtain_

\[a_{1j} =\left(\frac{x_{j}}{\lambda_{1j}}\frac{\lambda_{11}}{x_{1}} \right)a_{11}\qquad j=1,\ldots,n\] \[\vdots\] \[a_{dj} =\left(\frac{x_{j}}{\lambda_{dj}}\frac{\lambda_{d1}}{x_{1}} \right)a_{d1}\qquad j=1,\ldots,n \tag{5}\]

_and_

\[S_{i}=\left(\sum_{j=1}^{n}\frac{x_{j}^{2}}{\lambda_{ij}}\right)\frac{\lambda _{i1}}{x_{1}}a_{i1}\quad\ i=1,\ldots,d \tag{6}\]

Proof.: For generic data, the equation \(B^{T}(Wxx^{T}-yx^{T})+\Lambda\circ A=0\) implies that \(\Lambda\circ A\) is a matrix of rank at most one since \(xx^{T}\) and \(yx^{T}\) are matrices of rank one. The identities in (5) are an explicit version of this observation. They follow from the fact that \((\sum_{k=1}^{p}b_{k1}R_{k})\), i.e. the coefficient of \(x_{j}\) in the first row of (3), is a constant complex number that does not depend on \(j\). Therefore \(\frac{-\lambda_{1j}a_{1j}}{x_{j}}\) is a constant. Hence, we have \(\frac{-\lambda_{11}a_{11}}{x_{1}}=\frac{-\lambda_{12}a_{12}}{x_{2}}=\cdots= \frac{-\lambda_{1n}a_{1n}}{x_{n}}\). Equivalently, \(a_{1j}=\left(\frac{x_{j}}{\lambda_{1j}}\frac{\lambda_{11}}{x_{1}}\right)a_{11}\) for all \(j=1,\ldots,n\). Repeating the same for all equations in (3) gives (5). Substituting these into \(S_{i}=\sum_{i=1}^{n}a_{ij}x_{j}\) results in (6). 

The next result is the main theorem of this section.

**Theorem 5**.: _Consider a linear network where \(H=1\), \(m=1\), \(d_{x}=n\), \(d_{y}=p\) and \(d_{1}=d\) with generic input/output vectors and generic regularization matrices. Then, there are at most_

\[\mathcal{B}_{\mathbb{C}^{*}}=(4p)^{d}\]
solutions of (2) for which \(a_{11},\dots,a_{dn}\in\mathbb{C}^{*}\)._

Proof.: The overall strategy for this proof is to eliminate \(b_{ki}\) using (4), and then substitute the expressions for the \(b_{ki}\) variables so obtained into (3). That will give us a polynomial system in just the \(a_{ij}\) variables. We will then compute the BKK bound for this "reduced" system. That will give us an upper bound on the number of solutions in \((\mathbb{C}^{*})^{d\times n}\).

We first express \(b_{k2}\),..., \(b_{kd}\) in terms of \(b_{k1}\). By the equations in (6) each \(S_{i}\) is equal to \(a_{i1}\) scaled by some constant. Since each \(a_{ij}\) is non-zero by assumption, each \(S_{i}\) is non-zero. Then (4) implies

\[\frac{b_{k1}\sigma_{k1}}{S_{1}}=\frac{b_{k2}\sigma_{k2}}{S_{2}}=\dots=\frac{b_ {kd}\sigma_{kd}}{S_{d}}.\]

Hence, \(b_{k2}=b_{k1}\left(\frac{\sigma_{k1}}{\sigma_{k2}}\frac{S_{2}}{S_{1}}\right)\),..., \(b_{kd}=b_{k1}\left(\frac{\sigma_{k1}}{\sigma_{k,d}}\frac{S_{d}}{S_{1}}\right)\). Substituting the preceding expressions for \(b_{k2}\) through \(b_{kd}\) into \(R_{k}\), we get:

\[R_{k}=b_{k1}\left[S_{1}+\left(\frac{\sigma_{k1}}{\sigma_{k2}}\frac{S_{2}^{2}}{ S_{1}}\right)+\dots+\left(\frac{\sigma_{k1}}{\sigma_{kd}}\frac{S_{d}^{2}}{S_{1}} \right)\right]-y_{k}.\]

Using this expression in (4) and with a bit of algebra we obtain

\[b_{ki}=\frac{y_{k}S_{i}}{\sigma_{ki}}\frac{1}{(1+T_{k})}\]

where \(T_{k}=\frac{S_{1}^{2}}{\sigma_{k1}}+\frac{S_{2}^{2}}{\sigma_{k2}}+\dots+\frac {S_{d}^{2}}{\sigma_{kd}}\). Note that the right hand side of this identity is a rational function of \(a_{ij}\) only. We substitute these expressions for \(b_{ki}\) into the first equation of (3) to result in

\[-\frac{\lambda_{1j}a_{1j}}{x_{j}}=-\sum_{k=1}^{p}\frac{y_{k}^{2}S_{1}}{\sigma _{k1}(1+T_{k})^{2}}.\]

Now, (5) implies \(-\frac{\lambda_{1j}a_{1j}}{x_{j}}=-\frac{\lambda_{11}a_{11}}{x_{1}}\) and using (6) for \(S_{1}\), the above equation gives

\[\frac{1}{\left(\sum_{j=1}^{n}\frac{x_{j}^{2}}{\lambda_{1j}}\right)}-\left[ \frac{y_{1}^{2}}{\sigma_{11}(1+T_{1})^{2}}+\frac{y_{2}^{2}}{\sigma_{21}(1+T_{ 2})^{2}}+\dots+\frac{y_{p}^{2}}{\sigma_{p1}(1+T_{p})^{2}}\right]=0.\]

Clearing denominators we arrive at

\[\frac{(1+T_{1})^{2}(1+T_{2})^{2}\cdot\cdot\cdot(1+T_{p})^{2}}{\left(\sum_{j=1 }^{n}\frac{x_{j}^{2}}{\lambda_{1j}}\right)}-\sum_{k=1}^{p}\frac{y_{k}^{2}}{ \sigma_{k1}}\prod_{i\neq k}^{p}(1+T_{i})^{2}=0. \tag{7}\]

Each \(T_{k}\) is a polynomial in the variables \(a_{11},\dots,a_{d1}\). Hence, (7) is a polynomial equation in these variables only. A total of \(d\) such polynomials comprises a system of \(d\) polynomial equations in \(d\) variables. The solution set contains the solutions to (2) when \(a_{ij}\in\mathbb{C}^{*}\), but it also
contains components with positive dimension, for instance, the codimension two algebraic sets given by \(1+T_{i}=1+T_{j}=0\) for \(i\neq j\). We proceed to introduce regularization for this system as well. To equation \(i=1,\ldots,d\) we add the regularization term \(\mu_{i}a_{di}\) where \(\mu_{i}\) is a regularization parameter. A standard argument using the generalized Sard's theorem and the implicit function theorem guarantees that, for almost all choices of \(\mu_{i}\), \(i=1,\ldots,d\), all solutions to the regularized system are isolated and nondegenerate, and for sufficiently small \(\mu_{i}\), as they shrink to \(0\) uniformly, the solutions to the regularized system either converge to components of the solution set of the non-regularized system (including the solutions we want to count) or diverge to infinity. For an excellent discussion of regularization in the context of solving systems of polynomials we refer to Section 3.2 in [14]. The Newton polytope of (7) and its regularization is \((4p)\Delta_{d}=\operatorname{conv}\{0,4p\cdot e_{1},...,4p\cdot e_{d}\}\) where \(e_{i}\) is the \(i\)-th standard basis vector of \(\mathbb{R}^{d}\). Thus, we obtain a zero-dimensional polynomial system with \(d\) equations in \(d\) variables where the Newton polytope of each equation is \((4p)\Delta_{d}\). Therefore the BKK bound for this system is given by the normalized volume of \((4p)\Delta_{d}\) which is equal to \((4p)^{d}\). This completes the proof. 

**Remark 6**.: Theorem 5 gives a bound on the number of critical points of the regularized loss function when the weights in \(A=W_{1}\) are all nonzero. In the following section we will relax this requirement after treating patterns of zeros for the \(H=1\) and \(m=1\) case. We also note that in this theorem we have not made any assumptions about the weights in \(B=W_{2}\).

**Remark 7**.: It is an open question to find a reduced system for \(m>1\) even when \(H=1\) that will yield a better bound on the number of critical points. In particular, is there a systematic way to eliminate the weights in \(B=W_{2}\)?

## 4. Patterns of Zeros

This section is about patterns we observe in the critical points of the regularized loss function of a deep linear neural network. All of our results are about networks trained on a single data point (\(m=1\)). However, we believe the phenomena we observe are more general and persist for arbitrary \(m\). To support this claim we provide computational evidence. We start with the observation that if in a critical point an entry of a weight matrix is zero then the entire row or column of that entry must also consist of zeros.
**Proposition 8**.: _Suppose an arbitrary deep linear neural network is trained on a single generic data point \(x\) with the corresponding generic output vector \(y\). Then, if in a critical point of \(\mathcal{L}^{\Lambda}\) the \((i,j)\) entry of a weight matrix is zero, then either the \(i\)th row or the \(j\)th column of the same weight matrix is zero._

Proof.: We let \(d_{x}=n\) and \(d_{y}=p\). Suppose the \((i,j)\) entry of \(Z=W_{k}\in\mathbb{C}^{r\times s}\) is zero. Then

\[\frac{\partial\mathcal{L}^{\Lambda}}{\partial Z}=U_{Z}^{T}\left(Wxx^{T}-yx^{T} \right)V_{Z}^{T}+\Lambda\circ Z=0\]

where \(U_{Z}=U_{k}\), \(V_{Z}=V_{k}\) and \(\Lambda=\Lambda_{k}\) as in (2). It follows that

\[\begin{pmatrix}D_{1}E_{1}&\cdots&D_{1}E_{s}\\ \vdots&&\vdots\\ D_{r}E_{1}&\cdots&D_{r}E_{s}\end{pmatrix}=-\begin{pmatrix}\lambda_{11}z_{11}& \cdots&\lambda_{1s}z_{1s}\\ \vdots&&\vdots\\ \lambda_{r1}z_{r1}&\cdots&\lambda_{rs}z_{rs}\end{pmatrix} \tag{8}\]

where

\[D_{i}=(U_{Z}^{T})_{i1}R_{1}+\cdots+(U_{Z}^{T})_{ip}R_{p}\]

\[E_{j}=x_{1}(V_{Z}^{T})_{1j}+\cdots+x_{n}(V_{Z}^{T})_{nj}\]

and \(R_{k}=(Wx-y)_{k}\). If \(z_{ij}=0\) then \(D_{i}=0\) or \(E_{j}=0\) and the result follows. 

It is worth noting that if row \(i\) of the weight matrix \(W_{k}\) is \(0\), it means that the \(i\)th neuron on layer \(k\) can be removed, and the column \(i\) of \(W_{k+1}\) can be deleted. Conversely, if column \(j\) of the weight matrix \(W_{k}\) is \(0\), then this column can be deleted and the \(j\)th neuron on layer \(k-1\) can be removed. These statements have the following stronger counterpart.

**Theorem 9**.: _Suppose an arbitrary deep linear neural network is trained on a single generic data point \(x\) with the corresponding generic output vector \(y\). Then in a critical point of \(\mathcal{L}^{\Lambda}\) the row \(i\) of \(W_{k-1}\) is zero if and only if the \(i\)th column of \(W_{k}\) is zero._

Proof.: We retain our notation from the previous proof, and let \(P=W_{k-1}\) and \(Z=W_{k}\). Then \(V_{Z}=P\cdots W_{2}W_{1}\), and if row \(i\) of \(P\) is zero, then row \(i\) of \(V_{Z}\) is zero. It follows that \(E_{i}=x_{1}(V_{Z}^{T})_{1i}+\cdots+x_{n}(V_{Z}^{T})_{ni}=0\) and from (8) we conclude that the \(i\)th column of \(Z\) is zero. Conversely, suppose the \(i\)th column of \(Z\) is zero. Since \((U_{P})^{T}=Z^{T}\cdots W_{H}^{T}W_{H+1}^{T}\), then row \(i\) of \((U_{P})^{T}\) is zero. It follows that \(D_{i}=(U_{P}^{T})_{i1}R_{1}+\cdots+(U_{P}^{T})_{ip}R_{p}=0\) and the result follows again from (8). 

For the weight matrices \(W_{1}\) and \(W_{H+1}\), Proposition 8 has stronger corollaries.
**Corollary 10**.: _Suppose an arbitrary deep linear neural network is trained on a single generic data point \(x\) with the corresponding generic output vector \(y\). Then, if in a critical point of \(\mathcal{L}^{\Lambda}\) the \((i,j)\) entry of \(W_{1}\) is zero, then the \(i\)th row of \(W_{1}\) must be zero._

Proof.: This follows from equations (5). 

**Corollary 11**.: _Suppose an arbitrary deep linear neural network is trained on a single generic data point \(x\) with the corresponding generic output vector \(y\). Then, if in a critical point of \(\mathcal{L}^{\Lambda}\) the \((i,j)\) entry of \(W_{H+1}\) is zero then the \(j\)th column of \(W_{H+1}\) must be zero._

Proof.: Let \(Z=W_{H+1}\) and suppose \(z_{ij}=0\). Then \(D_{i}E_{j}=0\) as in the proof of Proposition 8. We observe that \(U_{Z}\) is the identity matrix and therefore \(D_{i}=R_{i}=(Wx-y)_{i}\). If column \(j\) of \(Z\) is not zero, then \(E_{j}\neq 0\) and therefore \(D_{i}=0\). Hence \(R_{i}=0\) and this means that \((Wx)_{i}=y_{i}\). But this also means that \(i\)th row of \(Z\) is zero, and therefore the \(i\)th row of \(W\) is zero. But then \(y_{i}=0\) which contradicts the genericity of \(y\). We conclude that the \(j\)th column of \(Z=W_{H+1}\) must be zero. 

For a network with one hidden layer trained on a single data point we gave a bound on the number of critical points in which the entries of the first weight matrix are in \(\mathbb{C}^{*}\); see Theorem 5. Now we expand this count to all complex critical points.

**Corollary 12**.: _Consider a linear network with \(H=1\), \(m=1\), \(d_{x}=n\), \(d_{y}=p\) and \(d_{1}=d\) with generic input/output vectors and generic regularization matrices. Then, there are at most_

\[\mathcal{B}_{\mathbb{C}}=(1+4p)^{d}\]

_complex solutions of (2)._

Proof.: By the above results the possible patterns of zeros in critical points are completely determined by the rows of \(W_{1}\) that are zero. Since the corresponding columns of \(W_{2}\) must be zero we can reduce the neural network by removing the neurons matching the zero rows of \(W_{1}\). If \(r\) rows of \(W_{1}\) are zero, Theorem 5 implies that there are at most \((4p)^{d-r}\) critical points where the remaining rows of \(W_{1}\) are in \(\mathbb{C}^{*}\). Therefore the number of all complex critical points for this network is

\[\mathcal{B}_{\mathbb{C}}=\sum_{r=0}^{d}\binom{d}{r}(4p)^{d-r}=(1+4p)^{d}.\]
**Example 13**.: Consider a linear neural network with one hidden layer trained on a single input vector where \(d_{x}=d_{y}=d_{1}=2\). The classic Bezout bound on the number of complex critical points is \(3^{8}=6561\). The BKK bound yields \(1089\). According to Theorem 5 the number of critical points in \((\mathbb{C}^{*})^{8}\) is bounded by \(\mathcal{B}_{\mathbb{C}^{*}}=64\), and by the above corollary the number of critical points in \(\mathbb{C}^{8}\) is bounded by \(\mathcal{B}_{\mathbb{C}}=81\). These solutions are expected to come in four flavors:

\[\begin{array}{ccccc}W_{2}&&W_{1}&&W_{2}&&W_{1}\\ \left(\begin{array}{ccccc}*&*&*&*\\ *&*&*&*\end{array}\right)&\left(\begin{array}{ccccc}0&*&0&0\\ 0&*&*&*\end{array}\right)\\ \\ \left(\begin{array}{ccccc}*&0&*&*\\ *&0&0&0\end{array}\right)&\left(\begin{array}{ccccc}0&0&0&0\\ 0&0&0&0\end{array}\right)\end{array}\]

The first type with full support corresponds to critical points in \((\mathbb{C}^{*})^{8}\), and there are at most \(64\) of them. The next two types have zeros in the first or second row of \(W_{1}\), and there are at most \(8\) such solutions for each type. Finally, there is the last type which corresponds to the origin. Using _HomotopyContinuation.jl_ we see that there are actually \(33\) critical points in \(\mathbb{C}^{8}\) of which \(16\) are in \((\mathbb{C}^{*})^{8}\). \(\triangle\)

**Example 14**.: Not every possible pattern of zeros is realized. A simple example comes from the case with \(d_{x}=d_{y}=1\) and \(d_{1}=2\) when \(H=m=1\). Again one expects four types of critical points, but there are no complex critical points with only nonzero coordinates. The total number of critical points is \(9\). A more interesting example arises from the case when \(H=d_{x}=d_{y}=d_{1}=2\) with \(m=1\). The total number of possible types of critical points is ten. However, the following two types of solutions do not appear:

\[\begin{array}{ccccc}W_{3}&&W_{2}&&W_{1}\\ 0&*&0&0&*&*\\ 0&*&*&*&*&*\\ *&0&*&*&*&*\\ *&0&0&0&*&*\end{array}\]

\(\triangle\)

**Conjecture 15**.: All our results in this section are for linear neural networks trained on a single data point. However, in our computations we observe that Proposition 8, Theorem 9, Corollary 10, and Corollary 11 hold for \(m=2\). We conjecture these results to be true for any \(m\).
## 5. Computations and Conclusions

In this section we present results showing the number of critical points for different network architectures, i.e. different values of \(H,m,d_{i},d_{x}\) and \(d_{y}\). When \(H=2\) we use \(d_{1}=d_{2}\) and denote it by \(d_{i}\). The results are summarized in Tables 1 - 4 which can be found in the Appendix. Each case yields a polynomial system whose coefficients are determined by the entries of the data matrices \(X\in\mathbb{R}^{d_{x}\times m}\) and \(Y\in\mathbb{R}^{d_{y}\times m}\), and the regularization constants from \(\Lambda_{i}\). The entries of \(X\) and \(Y\) are drawn i.i.d. from a Gaussian distribution with mean 0 and variance 1. The entries of \(\Lambda_{i}\) are drawn i.i.d. from the uniform distribution between 0 and 1. For each case, all isolated solutions to each of the 100 samples are computed using _HomotopyContinuation.jl_ on a single Amazon EC2 instance. Instances used were of type _c5.4xlarge_ (16 vCPUs, 8 cores, 32 GB memory) or _c5.9xlarge_ (36 vCPUs, 18 cores, 72 GB memory), depending on the size of the polynomial system. Most systems took less than an hour of wall-clock time to solve.

Table 1 through Table 4 expand the results for \(H,m\in\{1,2\}\) reported in [14] where the authors used _Bertini_. We report the classical Bezout bound, the BKK bound, the number of complex critical points (\(N_{\mathbb{C}}\)), the number of complex critical points in the algebraic torus (\(N_{\mathbb{C}^{*}}\)), and the maximum number of real critical points observed (\(\max\{N_{\mathbb{R}}\}\)). Table 1 contains two additional columns reporting \(\mathcal{B}_{\mathbb{C}^{*}}\) and \(\mathcal{B}_{\mathbb{C}}\) of Theorem 5 and Corollary 12, respectively.

We observe that while the BKK bound is much better than the Bezout bound, even the BKK bound is far from the actual number of complex critical points. The number of real critical points is even lower. The gradient equations (2) yield very structured and very sparse polynomials. Though the computed BKK bounds are not tight enough, it would be interesting to study the Newton polytopes of these equations in the hope of finding formulas for their mixed volumes. Theorem 5 provides much tighter bounds for the case of \(H=1\) and \(m=1\). We believe it is worthwhile to attempt to generalize this theorem. However, the real benefit of this theorem or its possible generalizations is in creating custom-made homotopies that will track only \(\mathcal{B}_{\mathbb{C}}\) paths. This will require constructing appropriate initial systems.

We would like to also note that while some of our results explain some of the numbers in the tables, there are other patterns waiting to be explained. Proposition 3 tells us that the BKK bounds in Table 1 and 2, and those in Table 3 and 4 for the same \(d_{i},d_{x},d_{y}\) must match. From the limited computations we have, it appears that the BKK bounds for the identical parameters except \(d_{x}\) and \(d_{y}\) values switched (for instance
\(d_{x}=2\) and \(d_{y}=3\) versus \(d_{x}=3\) and \(d_{y}=3\)) coincide. We invite the reader to unearth more patterns.
