Linear Unit*]{} ([*ReLU*]{},  denotes the set of real numbers).

The deep nonlinear neural network described in (Kawaguchi, 2016) consists of such neurons described above (Fig. \[fig1\]). The network has $H+2$ layers ($H$ is the number of hidden layers). The activation function $\psi$ of the neuron in the output layer is linear, i.e., $\psi(u)=u$ for any $u\in{\mbox{\boldmath $R$}}$. For any $0 \leq k \leq H+1$, let $d_k$ denote the number of neurons of the $k$-th layer, that is, the width of the $k$-th layer where the 0-th layer is the input layer and the $(H+1)$-th layer is the output layer. Let $d_x = d_0$ and $d_y = d_{H+1}$ for simplicity.

Let $({\mbox{\boldmath $X$}}, {\mbox{\boldmath $Y$}})$ be the training data where ${\mbox{\boldmath $X$}}\in {\mbox{\boldmath $R$}}^{d_x \times m}$ and ${\mbox{\boldmath $Y$}}\in {\mbox{\boldmath $R$}}^{d_y \times m}$ and where $m$ denotes the number of training patterns. We can rewrite the $m$ training data as $\{ ({\mbox{\boldmath $X$}}_i, {\mbox{\boldmath $Y$}}_i) \}_{i=1}^m$ where ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $R$}}^{d_x}$ is the $i$-th input training pattern and ${\mbox{\boldmath $Y$}}_i \in {\mbox{\boldmath $R$}}^{d_y}$ is the $i$-th output training pattern. Let ${\mbox{\boldmath $W$}}_k$ denote the weight matrix between the $(k-1)$-th layer and the $k$-th layer for any $1 \leq k \leq 