extensively studied in terms of which part of an input image affects the prediction result[@Ancona2018; @Bach2015; @Shrikumar2017; @Simonyan2014; @Smilkov2017; @Springenberg2015; @Sundararajan2017]. Approach (2) has been studied by several authors who examined the function of a given part of an LNN[@Alain2017; @Luo2016; @Raghu2017; @Zahavy2016]. There has also been an approach designed to automatically extract the cluster structure of a trained LNN[@Watanabe2017d; @Watanabe2017b; @Watanabe2018] based on network analysis.

Although the above studies have made it possible to provide us with an interpretable representation of an LNN function with a fixed resolution (or number of clusters), there is a problem in that we do not know in advance the optimal resolution for interpreting the original network. In the methods described in the previous studies[@Watanabe2017b; @Watanabe2017d; @Watanabe2018; @Watanabe2018arxiv; @Watanabe2018arxiv2], the unit clustering results may change greatly with the cluster size setting, and there is no criterion for determining the optimal cluster size. Another problem is that the previous studies could only provide us with information about the magnitude of the relationship between a cluster and each input or output dimension value, and we could not determine whether this relationship was positive or negative.

In this paper, we propose a method for extracting a hierarchical 