 l is denoted R l = {R l,r = (D l,r, C l,r )} r, where D l,r and C l,r are sets of conjoined literals in the antecedents (conditions for satisfying the rule) and consequents (outcomes) of R l,r respectively. For example, D l,r = L i,l−1,3 ∧ ¬L i,l−1,6 ∧ L i,l−1,7 and C l,r = L i,l,2 ∧ L i,l,3 ∧ L i,l,5. C l,r may only contain positive literals as we assume default negation, i.e. by default all b i,l,k = −1 (¬L i,l,k ) unless some C l,r determines otherwise.

Inference
Inference is summarised in fig. 2. Eq. 1 and 2 formalise the process by which we infer a binary approximation b i,l,k for activation tensor A i,l,k for any kernel. An extracted program approximates convolutional layers after and including layer l LEP, at which point kernel activations are mapped to binary values via a quantisation function Q(A i,l,k, θ l,k ) so that these activations may be treated as the input to logic program M * (eq. 1). Q(A i,l,k, θ l,k ) is explained in detail later in subsection 3.3. The truths of all kernels in all following layers (b i,l,k for l > l LEP ) are derived through logical inference on the truths of binarised kernels from the previous layer b i,l−1 according to a set of layer-specific rules R l (eq. 2).

Rule extraction
Rule extraction is implemented as a pipeline of 5 modules ( fig. 1). First is the original model M for which we want to extract an approximation M *. We do not need to say much about this except that ERIC assumes M has already been trained. Next in the quantisation stage we obtain binarisations for all kernels after and including layer l LEP based on activations obtained for training data. We then extract rules which describe the relationship between kernels by reference to their binarisations. Then to interpret the meanings of individual kernels we first visualise each kernel as one or more images that represent what inputs the kernels strongly react to, before each kernel is assigned a label based on manual inspection, a process we plan to automate in future work.
Quantisation Our quantisation function Q is defined in eq. 3, where θ l,k is a kernel-specific threshold and norm function · is the l1 norm 1. Intuitively, we say that a kernel is active when its norm breaches a threshold specific to that kernel. Note that for the initial rule extraction process we quantise all extractable layers l ≥ l LEP but for inference we only need to quantise kernels at l LEP.
We define a kernel's threshold as the mean norm of its activations with respect to the training data x tr, as in eq 4. To this end we make a forward pass of x tr in order to obtain {A tr i,l,k } i,l,k, activations for each kernel 1 ≤ k ≤ K l at each layer l LEP ≤ l < l o for each input training sample 1 ≤ i ≤ n.
We can now use the quantisation function (eq. 3) to obtain binarisations of all kernel activations according to eq. 5. Where a convolutional layer outputs to a pooling layer, we take A i,l,k from the pooled output. As also shown in eq. 5, we also need to treat output neurons as kernels of dimension 1 × 1 so that b tr i,l o = M (x tr i ). This enables us to extract rules that map kernel activations at layer l o − 1 to the output classifications as inferred by M.
Rule extraction We now extract rules that describe the activation at each kernel at every layer l given activations at layer l − 1. Thus, the following is applied layer-wise from l o to l LEP. We use a tree-based extraction algorithm similar to the C4.5 algorithm [37] to extract rules which describe conditions for which each kernel evaluates as true. As we assume default negation, we do not need to extract rules that describe when a kernel is false. Let us denote the training data Z l = {(z i, t i ) | i = 1,..., n} where z i ∈ {T rue, F alse} 2K l−1 and t i ∈ {T rue, F alse}. Note that the length of z l is twice the number of kernels at layer l − 1 because each kernel has positive and negative literals. z l−1,k = T rue if it corresponds to a positive literal and its binary value is 1 or if it represents a negative literal and its binary value is -1. It is False otherwise. C4.5 generates a decision tree up to maximum depth d. Each path from the root of the tree to a leaf node represents a separate rule and nodes branch on rule conditions (i.e. antecedents). The maximum number of antecedents per rule is equal to d+1. C4.5 uses entropy as a branch selection criterion but based on a marginal improvement in fidelity observed in preliminary tests we chose to use gini index. Extraction can become intractable as more layers and therefore kernels are introduced due to combinatorial explosion. This can be moderated by reducing d or increasing another parameter α that we introduce for this purpose. Let P, Q represent sets of training instances that satisfy the path of conditions leading to a parent node and child node, respectively. We stop branching if |Q|/|P | < α. If a leaf node represents multiple outcomes, we set the consequence to the modal value of Q. Finally, we simplify extracted programs by merging rules with complementary literals (A ∧ B → C and A ∧ ¬B → C become A → C) and rules with identical antecedents but different consequents (A → B and A → C become A → B ∧ C).

Kernel visualisation and labelling
To visualise what a kernel sees, we select the m images from x tr which activate that kernel most strongly with respect to A tr i,l,k. We denote this visualisation asx m l,k. A label is assigned to a kernel based onx m l,k, which for the time being we do manually based on visual inspection but in future work plan to automate. For the time being, to defend the arguments of the paper, it is not so much the labels that are important as the distinction between the subsets of image that each kernel responds most strongly to.

Experiments
In sections 4.1 and 4.2 we outline the classification task and CNN configuration we use for our experiments. We then extract rules from a single convolutional layer in section 4.3 and then multiple convolutional layers in section 4.4. In section 4.5 we visualise and label kernels and in section 4.6 analyse some of the rules with these labels assigned to the antecedents.

Task
We chose to classify road scenes from the places365 dataset [36] for a number of reasons. First, we felt that a scene dataset was appropriate as scenes can easily be described by reference to symbolic entities within them which themselves could be described by a separate classifer (i.e. the kernel classifier) with a large vocabulary of labels. We selected a handful of 5 scenes to simplify the task given the complexity of rule extraction, and opted for roads in order to create a scenario where the distinction between scenes is particularly important (due to regulations, potential hazards, etc). We wanted to demonstrate ERIC on a multi-class task and on multiple combinations of class. 3 is the minimum required for multi-class case and gives us 5 3 = 10 combinations of scenes (table 1).

Network Model
For each combination of classes we train VGG16 (as defined for Tensorflow using Keras [38]) from scratch over 100 epochs using Stochastic Gradient Descent, a learning rate of 10 −5, categorical crossentropy and a batch size of 32.

Extraction from a single layer
We set α = 0.01 so that branching stops if a branch represents less than 1% of the samples represented by its parent node. We iterate the logical entry point l LEP ∈ [Conv8... Conv13] and tree depth d ∈ [1... 5] and observe the effects on accuracy, fidelity and the size of the extracted program. Size is measured as the sum length (number of entecedents) of all rules and is our metric for interpretability on the basis that larger programs take longer to read and understand. In all figures we average over all results except the variable we iterate over. Fig. 3 compares the average accuracy across all depths and layers for each class combination with the accuracy of the original model. The line of best fit shows that the accuracy of the extracted model is consistent with respect to that of the original model. In the validation set, accuracy drops by about 15% in each case. However average validation accuracy drops by an average of 10% for the optimal selection of depth and layer (table 1). In summary, the loss in accuracy can be moderated by adjusting extraction parameters. Fig. 4 shows how accuracy, fidelity and program size are affected as we adjust the extraction layer and tree depth. Accuracy and fidelity both improve as tree depth (and therefore rule length) is increased, demonstrating that extraction benefits from