result.

![Representative input-output mappings of extracted clusters. []{data-label="fig:food_roles"}](food_align_lnn.png){width="110mm"}

\[fig:food\_align\_lnn\]

\

\

\

\
\
\
\

Discussion
==========

Here, we discuss our proposed method for obtaining a hierarchical modular representation from the perspectives of statistical evaluation and visualization.

Our proposed method provides us with a series of clustering results for an arbitrary cluster size, and the resulting structure does not change if we use the same criterion (e.g. error sum of squares for Ward’s method) for evaluating the similarity of the feature vectors. However, there is no way to determine which criterion yields the optimal clustering result to represent a trained LNN, due to the fact that interpretability of acquired knowledge cannot be formulated mathematically (although there has been an attempt to quantify the interpretability for a specific task, especially image recognition[@Bau2017]). This problem makes it impossible to compare different methods for interpreting LNNs quantitatively, as pointed out in the previous studies[@Lipton2016; @DoshiVelez2017]. Therefore, the provision of a statistical evaluation method as regards both interpretability and accuracy for the resulting cluster structure constitutes important future work.

Although we can apply our proposed method to an arbitrary network structure, as long as it contains a set of units that outputs some 