clustering results with every possible number of clusters. By determining the feature vectors of the hidden layer units based on their correlations with input and output dimension values, it also enabled us to know what range of input each cluster maps to what range of output. We showed the effectiveness of our proposed method experimentally by applying it to two kinds of practical data sets and by interpreting the resulting cluster structure.

[10]{}

G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. In [*ICLR 2017 Workshop*]{}, 2017.

M. Ancona, E. Ceolini, A. C. [Ö]{}ztireli, and M. Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In [*International Conference on Learning Representations*]{}, 2018.

S. Bach, A. Binder, G. Montavon, F. Klauschen, K-R. M[ü]{}ller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. , 10:1–46, 2015.

D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In [*Computer Vision and Pattern Recognition*]{}, 2017.

M. Craven and J. W. Shavlik. Extracting tree-structured representations of trained networks. In [*Advances in Neural Information Processing Systems 8*]{}, pages 24–30, 1996.

F. Doshi-Velez and B. Kim. Towards 