on cosine similarity (Algorithm \[alg:alignment\]). By randomly selecting a feature vector and aligning its sign according to the sum of the cosine similarities with all the other feature vectors, the sum of the cosine similarities of all the pairs of feature vectors increases monotonically. We show experimentally the effect of this sign alignment algorithm in Appendix 2.

Let $v_k$ and $a_0$ respectively be the feature vector for the $k$-th hidden layer unit and the number of iterations. layer units. Randomly choose the $k$-th hidden layer unit according to the uniform distribution. $v_k \gets -v_k$.

![An example of two hidden layer units with the same function. The corresponding feature vectors are the same, except that their signs are opposite. []{data-label="fig:alignment"}](alignment.png){width="55mm"}

Hierarchical Clustering of Units in a Trained LNN
-------------------------------------------------

Once we have obtained the feature vectors of all the hidden layer units as described in section \[sec:features\], we can extract a hierarchical modular representation of an LNN by applying hierarchical clustering to the feature vectors. Among the several existing methods for such hierarchical clustering including single-link and complete-link, Ward’s method[@Ward1963] has been shown experimentally to be effective in terms of its classification sensitivity, so we employ this method in our experiments.

