Wakamiya, Atsugi-shi, Kanagawa Pref. Japan\
    `watanabe.chihiro@lab.ntt.co.jp`
title: Interpreting Layered Neural Networks via Hierarchical Modular Representation
---

Introduction
============

To construct a method for interpreting the prediction mechanism of complex statistical models is currently one of the most important tasks in the machine learning field, especially with layered neural networks (or LNNs), which have achieved high predictive performance in various practical tasks. Due to their complex hierarchical structure and the nonlinear parameters that they use to process the input data, we cannot understand the function of a trained LNN as it is, and we need some kind of approximation method to convert the original function of an LNN into a simpler interpretable representation.

Recently, various methods have been proposed for interpreting the function of an LNN, and they can be roughly classified into (1) the approximation of an LNN with an interpretable model, and (2) the investigation of the roles of the partial structures constituting an LNN (e.g. units or layers). As for approach (1), various methods have been investigated for approximating an LNN with a linear model[@Lundberg2017; @Nagamine2017; @Ribeiro2016] or a decision tree[@Craven1996; @Johansson2009; @Krishnan1999; @Thiagarajan2016]. For image classification tasks in particular, methods for visualizing an LNN function have been 