\Bigr]E\Bigl[ \Bigl( y^{(n)}_j-E[y^{(n)}_j] \Bigr)^2 \Bigr]}}.\end{aligned}$$ Here, $y^{(n)}_j$ is the value of the $j$-th output layer unit for the $n$-th input data sample.

We define a feature vector of each hidden layer unit based on the above definitions.

We define the feature vector of the $k$-th hidden layer unit as $v_k\equiv [v^{\mathrm{in}}_{1k}, \cdots , v^{\mathrm{in}}_{i_0 k}, v^{\mathrm{out}}_{k1}, \cdots , v^{\mathrm{out}}_{kj_0}]$. Here, $i_0$ and $j_0$, respectively, represent the dimensions of the input and output data. \[def:feature\]

#### Alignment of signs of feature vectors based on cosine similarity

The feature vectors of Definition \[def:feature\] represent the roles of the hidden layer units in terms of input-output mapping. When interpreting such roles of hidden layer units, it is natural to regard the roles of any pair of units $(k_1, k_2)$ as being the same iff they satisfy $v_{k_1}=v_{k_2}$ or $v_{k_1}=-v_{k_2}$. The latter condition corresponds to the case where the $k_1$-th and $k_2$-th units have the same correlations with input and output dimensions except that their signs are the opposite, as depicted in Figure \[fig:alignment\]. To regard the roles of unit pairs that satisfy one of the above conditions as the same, we propose an algorithm for aligning the signs of the feature vectors based 