centroid of the feature vectors of the units assigned to the cluster, which can be interpreted as a representative input-output mapping of the cluster.

Experiments {#sec:experiments}
===========

We apply our proposed method to two kinds of data sets to show its effectiveness in interpreting the mechanism of trained LNNs. The experimental settings are detailed in the Appendix 3. In Appendix 1, we provide a qualitative comparison with the previous method[@Watanabe2018arxiv2].

Experiment Using the MNIST Data Set {#sec:mnist}
-----------------------------------

First, we applied our proposed method to an LNN trained with the MNIST data set[@LeCun1998] to recognize $10$ types of digits from input images. Before the LNN training, we sharpened the top, bottom, left and right margins and then resized the images to $14\times 14$ pixels. Figure \[fig:mnist\_samples\] shows sample images for each class of digits. Although our proposed method provided us with a clustering result for all the possible resolutions or the numbers of clusters $c$, we have only plotted the results for $c=4,8,16$, for ease of visibility. Figures \[fig:mnist\_align\_lnn\] and \[fig:mnist\_roles\], respectively, show the hierarchical cluster structure extracted from the trained LNN and the roles or representative input-output mappings of the extracted clusters. From these figures, we can gain knowledge 