A max-affine spline approximation of neural networks using the Legendre
  transform of a convex-concave representation
This work presents a novel algorithm for transforming a neural network into a
spline representation. Unlike previous work that required convex and
piecewise-affine network operators to create a max-affine spline alternate
form, this work relaxes this constraint. The only constraint is that the
function be bounded and possess a well-define second derivative, although this
was shown experimentally to not be strictly necessary. It can also be performed
over the whole network rather than on each layer independently. As in previous
work, this bridges the gap between neural networks and approximation theory but
also enables the visualisation of network feature maps. Mathematical proof and
experimental investigation of the technique is performed with approximation
error and feature maps being extracted from a range of architectures, including
convolutional neural networks.
Adam Perrett, Danny Wood, Gavin Brown
A max-affine spline approximation of neural networks using the Legendre transform of a convex-concave representation

###### Abstract

This work presents a novel algorithm for transforming a neural network into a spline representation. Unlike previous work that required convex and piecewise-affine network operators to create a max-affine spline alternate form, this work relaxes this constraint. The only constraint is that the function be bounded and possess a well-define second derivative, although this was shown experimentally to not be strictly necessary. It can also be performed over the whole network rather than on each layer independently. As in previous work, this bridges the gap between neural networks and approximation theory but also enables the visualisation of network feature maps. Mathematical proof and experimental investigation of the technique is performed with approximation error and feature maps being extracted from a range of architectures, including convolutional neural networks.

Machine Learning, Neural Networks, Neural Networks, Neural Networks, Neural Networks

## 1 Introduction

### Interpreting networks

Interpretability of neural networks is of increasing importance as they become embedded in more facets of everyday life. Applications such as medical diagnoses or self-driving cars would greatly benefit from easily applied inspection techniques. This would enable extracted network features to be validated and possibly used to make inferences about the data, such as knowing what features are indicative of certain diseases and responses. Being able to shine light into the black boxes that are neural networks facilitates verification beyond validation with hidden test data.

A common technique for understanding what input features correspond to particular outputs is to apply gradient ascent on the inputs whilst aiming to maximise a particular output's probability, also termed input optimisation (Erhan et al., 2009; Simonyan et al., 2013). As it relies on gradient techniques it can fall into common issues such as gradient saturation, initialisation sensitivity and local optima. There is also little guarantee about the extracted saliency map's relation to the trained features of the network. An alternative approach is the use of a complimentary deconvolutional network (Zeiler and Fergus, 2014; Linardatos et al., 2021). In effect the same filters of a convolutional layer are applied in reverse to extract the input saliency map. They have shown good performance in network interpretability but lack spatial precision because of the reverse convolution operation. They are also architecture dependant and therefore not a general purpose tool. Techniques are also available to understand what features were selected as pertinent in an image using gradient information (Zhou et al., 2016; Selvaraju et al., 2017), although this is on an image-by-image basis and does not extract general properties of the trained network.

### Network transformation

It was previously shown by Balestriero et al. that neural networks could be represented as a combination of Max-Affine Spline Operators (MASO) (Balestriero et al., 2018; Balestriero and Baraniuk, 2021). Splines are a way of approximating a continuous function with a combination of straight lines, or planes in high dimensions. This work provided a link between deep neural networks and approximation theory and was used to generate a regularisation technique which aided training. The contribution of Balestriero et al. was to show that neural network layers using convex operators, including convolution and pooling, can be represented as max-affine splines if they are piecewise-affine, such as ReLU, and approximated arbitrarily close if they are not piecewise. There is also discussion of how non-convex operators could be modelled as a combination of MASOs, although this is not explored experimentally. This technique is limited to creating a MASO of each layer individually and cannot be used to extract feature maps of the whole network.
The work of this paper presents a novel method for network transformation that converts all layers simultaneously into a pair of complementary Convex and Concave Splines (CCS). This alternate form allows feature maps to be extracted from the trained network to aid interpretability. There are no requirements of convexity, it only needs to be a bounded function with a well-defined second derivative, although it will be show experimentally how this constraint can be relaxed.

### Contributions

1. A novel method of splitting a neural network into complementary convex and concave components
2. The application and subsequent sampling of a Legendre transform of the convex and concave components to create a max-affine spline approximation of the original neural network that is embarrassingly parallel in operation
3. A novel method for feature visualisation and interpretation is proposed
4. The investigation of the CCS approximate form to evaluate network complexity

## 2 Background

Inspection of neural network behaviour is of increasing importance as they become crucial components of safety critical functions. There have been probing techniques which inspect the network (Zeiler and Fergus, 2014; Linardatos et al., 2021). Common techniques involve performing gradient ascent on the inputs to create representation that maximises the activation of hidden neurons or outputs. There have also been investigations into the theoretical bounds of neural network capacity, although these fall short of explaining a trained neural network's behaviour (Tishby and Zaslavsky, 2015; Achille and Soatto, 2018). An alternative neural network spline representation was presented by Balestriero et al. (Balestriero et al., 2018; Balestriero and Baraniuk, 2021) which showed that, by converting a network, properties can be inspected and understood using established mathematical techniques from approximation theory. Other authors have made the link between neural networks and splines but often only for single layers and in combination with piecewise linear (PWL) activations, such as ReLU (Parhi and Nowak, 2022; DeVore et al., 2021; Bunel et al., 2017, 2020; Misener and Floudas, 2010; Chu et al., 2018). Theoretical extensions are suggested for non-piecewise activation functions but to the authors knowledge there are no experimental examples as of writing.

### Affine (Linear) splines

Splines make an attractive alternative form for neural networks with PWL activations as it becomes possible to exactly match the original function with a finite set of splines and they also map \(\mathbb{R}^{D}\rightarrow\mathbb{R}\). In the domain \(\mathbb{R}^{D}\) the functional combination of \(N\) splines are a set of hyperplanes with slope parameters \(\alpha\in\mathbb{R}^{DxN}\) and offsets \(\beta\in\mathbb{R}^{N}\). As only one hyperplane is selected for any particular point in the domain this is equivalent to partitioning the domain into N regions. This problem requires the creation of complex polyhedra where each hyperplane meets another (Chu et al., 2018; Misener and Floudas, 2010; Bunel et al., 2020), termed knots in 1-D. If the function is convex then this step can be skipped and the maximum the hyperplanes can taken, avoiding the need for knots. This was exploited by Balestriero et al. (Balestriero et al., 2018; Balestriero and Baraniuk, 2021) to create a Max-Affine Spline (MAS) representation of a neural network layer. However, this relies on the use of PWL activation functions and some guarantees need to be made about convexity of the network function. Theoretical explanation is given to how non-convex and non-piecewise function could be approximated with a combination of MASs but no method for generating that was given.

### A convex-concave representation

It has been proven than any (PWL) function can be expressed as the difference between two convex PWL linear functions, alternatively the addition of a convex and concave function (Wang, 2004). As any continuous function can be approximated arbitrarily closely by a PWL function as the number of linear elements tends to infinity, by extension this means any continuous function can be approximated by the difference between two convex PWL functions. As was discussed in the MaxOut paper (Goodfellow et al., 2013) and by Balestriero et al. (Balestriero et al., 2018; Balestriero and Baraniuk, 2021) this means the difference between two MASs can approximate any continuous function. When applied to neural network approximation the difficulty then becomes finding the appropriate convex and concave function.

The Concave-Convex Procedure (Yuille and Rangarajan, 2003) was developed to find a suitable combination by minimising an energy function. In this method the Legendre transform is used to move to a suitable domain in which a cost function can be constructed to minimise over. The work of this paper aims to skip the need for optimisation by creating a creating a convex and concave representation by exploiting knowledge of the second derivative of the function that is being approximated.
## 3 Methodology

### Max-Affine Splines

Approximation theory aims to approximate a complex function with a simpler one. A spline function is an example of how a complex function can be approximated by the combination of linear segments. A max-affine spline (MAS) is a multivariate function (De Boor & De Boor, 1978) mapping \(\mathbb{R}^{D}\rightarrow\mathbb{R}\) which combines splines using the maximum function, taking the form below

\[\mathrm{mas}(x)=\max_{i=1,\ldots,N}\alpha_{i}\cdot x+\beta_{i}, \tag{1}\]

where \(\alpha_{i}\in\mathbb{R}^{D}\) is one of the \(N\) slope parameters, \(\beta_{i}\in\mathbb{R}\) is its corresponding offset and \(x\in\mathbb{R}^{D}\) are the input values. As \(N\rightarrow\infty\) this can approximate any convex function. Swapping the maximum function to a minimum allows to approximate any concave function.

### The Legendre transform

The Legendre transform is a powerful tool used to transform a function of one quantity into a conjugate quantity. This can be used to convert from a function of points \(x\) to a function of gradients \(m\). The Legendre transform of a function \(f(x)\) to a gradient formulation is shown in Eq. 2, where \(\sup\) is the supremum.

\[f^{*}(m)=\sup_{x\in\mathbb{R}}mx-f(x) \tag{2}\]

By sampling from \(f^{*}(m)\) a collection of gradients can be used to describe the original function. They take the form of a linear function \(y=mx+c\) where \(m\) is the gradient and \(c=f(x)\). The maximum operation can be used in place of the supremum and then this takes the form a MAS. The problem with taking a Legendre transform of a neural network, as is the aim of this work, is that it only works for convex functions.

### A convex-concave representation of any function

**Theorem 1:** Any function with a well-defined second derivative can be expressed as the sum of a convex function and a concave function (Klee, 1976).

**Theorem 2**: Any continuous PWL function can be expressed as the combination of a convex and concave PWL function (Wang, 2004).

For a function to be convex the second derivative must be greater than or equal to zero, \(f^{\prime\prime}(x)\geq 0\). This means if the second derivative of a function is know a convex representation can be constructed by adding a term to the second derivative to ensure it is always above zero. Using the sigmoid function as an example

\[f(x)=\frac{1}{1+e^{x}}=\sigma(x), \tag{3}\]

the second derivative is

\[f^{\prime\prime}(x)=\sigma(x)(1-\sigma(x))(1-2\sigma(x)), \tag{4}\]

which is minimum at \(x=\frac{1}{2}+\frac{1}{2\sqrt{3}}\) with \(f^{\prime\prime}(x)=\frac{-1}{6\sqrt{3}}\approx-0.96\). By adding \(0.96\) to the second derivative and twice integrating back a convex function is created. Any constant from integration can be set to zero as it does not effect the second derivative.

\[f_{\text{convex}}(x)=\frac{1}{1+e^{x}}+0.96x^{2}, \tag{5}\]

however, the original function is lost because of the additional \(x^{2}\) term. We can follow a similar method to create a concave representation of the function which can cancel out the \(x^{2}\) term,

\[f_{\text{concave}}(x)=\frac{1}{1+e^{x}}-0.96x^{2}, \tag{6}\]

which leads to

\[f(x)=0.5(f_{\text{convex}}+f_{\text{concave}}). \tag{7}\]

Due to the symmetry of the sigmoid function the minimum of the second derivative is the opposite sign of the maximum of the second derivative. In other more complicated functions the maximum magnitude of the maximum and minimum of the second derivative must be added to create complementary convex and concave parts. This leads to the general form

\[f(x)=0.5(f(x)+cx^{2}+f(x)-cx^{2}), \tag{8}\]

where

\[c=\max(|\max(f^{\prime\prime}(x))|,|\min(f^{\prime\prime}(x))|). \tag{9}\]

By splitting the function into a convex and concave part the Legendre transform can now be applied to create two complementary MAS approximations of the original function. In the implementation throughout this work \(f(x)\) and \(cx^{2}\) are kept separate to avoid any washing out of values of \(f(x)\) in floating point due to large values of \(cx^{2}\). It requires no more parameters than keeping a separate convex and concave function as \(c\) is common across both.

### A convex-concave neural network

A neural network with appropriate neuron activation is a bounded function with a well-defined second derivative. As \(\mathbf{x}\) is now a vector the first step requires taking the Hessian of the overall network. This creates a Hessian matrix of partial derivatives \(H=\nabla^{2}f(\mathbf{x})\). The original function is convex if and only if \(H\) is positive semi-definite (PSD) for all \(\mathbf{x}\). The Hessian is symmetric since

\[\frac{\delta}{\delta x_{i}\delta x_{j}}f(\mathbf{x})=\frac{\delta}{\delta x_{ j}\delta x_{i}}f(\mathbf{x}), \tag{10}\]
and a symmetric matrix is PSD if and only if the eigenvalues are positive.

**Proposition 1:** If \(\nabla^{2}f(\mathbf{x})\) entries are bounded, there exists \(c>0\) such that \(H+cI\) is PSD for all \(\mathbf{x}\).

If all eigenvalues are positive it is already convex and \(c=0\). Assuming the minimum eigenvalue, \(\varepsilon\), is less than zero \(c\) is non-zero. To calculate \(c\) let \(\lambda\) be an eigenvalue of \(H\) with eigenvector \(\mathbf{v}\) then

\[\lambda\mathbf{v} =H\mathbf{v}, \tag{11}\] \[\lambda\mathbf{v}+c\mathbf{v} =H\mathbf{v}+c\mathbf{v},\] (12) \[(\lambda+c)\mathbf{v} =(H+c)\mathbf{v}, \tag{13}\]

so \(\lambda+c\) is an eigenvalue of \(H+c\). Then set \(c=-\varepsilon\) and the Hessian is now PSD. But what is the convex function of \(\nabla^{2}f(\mathbf{x})+cI\)? Consider

\[g(\mathbf{x})=\mathbf{x}^{T}\mathbf{x}=\sum_{i}\mathbf{x}^{2}, \tag{14}\]

\[\frac{\delta}{\delta x_{i}\delta x_{j}}g(\mathbf{x})=\begin{cases}1&\text{if }i=j \\ 0&\text{otherwise,}\end{cases} \tag{15}\]

so the Hessian of \(g(\mathbf{x})=I\). Therefore \(\nabla^{2}f(\mathbf{x})+cI\) is the Hessian of \(f(\mathbf{x})+c\mathbf{x}^{T}\mathbf{x}\). Similar to the method described in Section 3.3 the minimum value of the eigenvalue is used for the convex and maximum for the concave function. The maximum absolute value is used to generate a common value of \(c\) which guarantees that both partitions of the neural network are complementary and respectively convex and concave. This procedure is carried out for each output of the network individually, creating a value of \(c\) for each.

Practically, when performing this step, as calculating the exact form of the Hessian at all points is computationally complex, the Hessian is evaluated at each input data point using the PyTorch Hessian function. The minimum and maximum value across all points is used to generate the value of \(c\) for each output.

#### 3.4.1 A convex-concave PWL neural network

Functions, such as ReLU, are twice differentiable but because of their piecewise nature their second derivative is not well-defined which is a requirement when generating the Hessian. Experimentally it was found that using the value of \(c\) calculated by a neural network using non PWL functions trained on the same task was sufficient. It will also be shown later that relatively arbitrary values of \(c\) can be used to create approximate convex and concave forms of PWL neural networks.

### Approximating MNIST

The MNIST dataset with the standard 60,000/10,000 train/test split is used for benchmarking neural network approximation (LeCun, 1998). 0.5 is subtracted from input values to centre them around zero. All networks in the following section were trained in PyTorch with stochastic gradient descent with momentum. Across all examples a learning rate of 0.05 used with a momentum value 0.9 and 50% dropout trains the network for 300 epochs with cross-entropy loss. Sigmoid activation is used unless otherwise stated. The CNN used uses two convolutional layers followed by a hidden layer of size 200. The first had 1 input channel, 16 output channels, a kernel size of 5x5, stride 1, and padding 2. Sigmoid activation and max pooling (kernel size 2x2) followed. The second convolution layer had 16 input channels, 32 output channels, and the same kernel size, stride, padding, activation, and max pooling.

The justification for using clustering to explore network complexity comes from each spline capturing elements of the original function. Therefore, the fewer planes required the simpler the function. When clustering the gradients and offsets of splines are concatenated into a single vector and clustered using the fast PyTorch K-means package. The clustering is run 10 times at each value of K to attain a mean and standard deviation. All experiments are run using a single NVIDIA a100 80GB GPU. Code for the experiments can be found at [https://github.com/adamgoodtime/Legendre_Net.git](https://github.com/adamgoodtime/Legendre_Net.git).

## 4 Experiments and Results

### Transforms in 1D

The first example displays the capability to create a Convex-Concave-Spline (CCS) representation of non-convex functions. Figure 1(a) shows the convex and concave pair which are constructed to approximate a Gaussian function. The faintly coloured lines show the uniformly sampled planes with the maximum of the green lines being used to create the convex component and the minimum of the blue lines used to create the concave component. These are then averaged to create the CCS approximation of the Gaussian function. Figure 1(b) shows the same procedure to approximate a sigmoid function.

Figure 1(c) applies the same technique to a combination of 400 Gaussian functions with mean, standard deviation and weight drawn uniformly from \([-3,3]\), \([0,0.2]\) and \([-1,1]\) respectively. 300 planes are uniformly sampled across the 1D domain, for both the convex and concave function, which corresponds to the same number of parameters as is used by the Gaussian combination. The error visible at various points is a result of the sampling frequency. The planes are sampled uniformly across the input space and, therefore, if the original function changes rapidly between sample locations the change in gradient is not precisely captured. This can be remedied by increasing the number of planes but this
\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Architecture & Test acc. \% & CCS acc. \% & Approx. loss \% \\ \hline \hline
200x5 & 98.47 & 97.69 & -0.78 \\ \hline
200x4 & 98.53 & 97.94 & -0.59 \\ \hline
200x3 & 97.9 & 97.35 & -0.55 \\ \hline
200x2 & 98.11 & 97.44 & -0.67 \\ \hline
1600x1 & 98.27 & 97.62 & -0.65 \\ \hline
800x1 & 98.18 & 97.64 & -0.54 \\ \hline
400x1 & 98.24 & 97.52 & -0.72 \\ \hline
200x1 & 98.1 & 97.74 & -0.36 \\ \hline CNN & 99.04 & 97.81 & -1.23 \\ \hline ReLU-200x2 & 98.5 & 97.64 & -0.86 \\ \hline ReLU-200x1 & 98.12 & 97.18 & -0.94 \\ \hline ReLU-200x1 c=5 & 98.12 & 97.11 & -1.01 \\ \hline \end{tabular}
\end{table}
Table 1: The MNIST test accuracy of the final models and the test accuracy of the CCS approximation using all planes. The final column shows the loss in accuracy following approximation. The architecture denotes the hidden layer width and depth using sigmoid activation unless otherwise stated. c=5 is a middling value of \(c\) for the ReLU network, although there is little variance around the chosen value of \(c\).

Figure 1: (a) A reconstruction of a Gaussian function with CCS. The faint lines show the planes of which the maximum (minimum) is taken to produce the convex (concave) function. (b) The same as (a) for a sigmoid activation function. (c) The CCS of 400 Gaussian functions combined. The inset shows the largest approximation error coming from limited sampling around a period of rapid change.
would require more computation resource. There is also the opportunity to non-uniformly sample planes and increase sampling around periods with large second derivatives, however, this is non-trivial.

### Approximating a network trained on MNIST

Table 4.1 shows the final test accuracy and approximation accuracy across a range of architectures. The approximation accuracy is relatively consistent across architecture settings. The most significant inaccuracy is from the single layer ReLU networks. Although, little difference is observed when using a uniform value of \(c\) or when using the calculated \(c\) from the 200x1 sigmoid network. The main difference in approximation loss is a result of the test accuracy of the network rather than the approximation accuracy as that is relatively consistent.

### Exploring network complexity with clustering

The CCS approximation generates a plane for every data point in the training set. This leads to a perfect correspondence in training accuracy between the original network and the approximate form but effects the testing accuracy. It can also lead to many planes, 60,000 for MNIST. To examine the redundancy of the CCS procedure planes are clustered using K-means. A range of K values is chosen to see how varying the number of planes changes the faithfulness of the approximation.

Across the plots of Figure 2 the CNN approximation is consistently lower, requiring around a magnitude more clusters to reach similar levels of approximation accuracy compared to other architectures. This can be understood as a measure of the complexity of the network. More planes are required match to function of the network meaning there is less redundancy. This also means the the network varies more across the input space and therefore is more computationally complex. This idea will be further explored in Section 4.4.

It can be seen in Figure 2(a) that layer size has little effect on the ability to approximate the network. This is likely a result of the expressivity of the network being fairly consistent across various layer sizes with added neurons not significantly effecting the test accuracy. This means the CCS is capturing a similarly shaped function and therefore achieves similar performance across different hidden layer sizes.

When comparing the approximation accuracy across different sizes of depth in Figure 2(b) a less clear picture emerges with the shallowest and deepest networks being the best approximated and layer depths of two and three containing the highest approximation errors. A possible explanation for this is that a shallow network has the lowest complexity and therefore can be approximated relatively well. As the number of layers increases the the complexity grows and with it the the algorithm struggles to approximate the complex function. However, once the depth goes beyond a certain point a degree of smoothing happens over the input space as a result of more information dispersal across the layers and regularisation via dropout. This will be explored in the Sec 4.4 where features will be extracted from various architectures and contrasted.

Examining the ReLU network approximation error shows that the choice of \(c\) does not need to be precise with a uniform and calculated value of \(c\) achieving similar approximation error. The value of \(c=5\) was chosen as the uniform value, although there was little sensitivity to the value between the range of 2 and 10. Less than 2 and the convex and concave components were not curved enough to capture the function of the network and beyond 10 there was diminishing returns from increased curvature due to floating point limitations. A surprising result from Figure 2(c) is that the best approximation came from ReLU network of depth 2 and the worst from a sigmoid network of depth 2. A possible explanation for this is the increased classification of the 2-layer ReLU network. Also, there is no significant increase in classification accuracy between the 2-layer sigmoid network and the 1-layer networks but there will be an increased complexity over the input space which will be harder for the approximation to capture.

### Examining features

Following the clustering of planes random clusters are selected to inspect the features of the network. A small value of \(K=10\) is used to increase the disparity between clusters. As was discussed in Section 3.3, the gradient and offset of \(f(x)\) and \(cx^{2}\) are kept separate, allowing the extraction of the gradient component of the plane that relates to the original function, \(f^{\prime}(x)\). This corresponds to the network's feature map for a particular output as the gradient shows what input would create the highest variability in response of the network.

Figure 3 compares the extracted features of four randomly selected clusters for the output zero. The features of shallow networks are consistent, with little variation across clusters. The deeper networks contain noticeable variation across clusters suggesting a higher information content captured by the network function. The 3-layer network has a significantly noisier feature map, which corresponds to the worse test accuracy. Although the 5-layer network has varied feature maps, the approximation is still good due to the quality of the features, which can be seen by their relative smoothness and lack of noise. The CNN has by far the smoothest and most varied feature maps. The features also look the least like the number zero. This suggests a high
Figure 2: K-means was run until convergence across all examples and then the classification accuracy on the test set was calculated using the selected planes. Clustering was performed 10 times with the line showing the mean and the shaded region the standard deviation. The effect of clustering the CNN planes is shown across all plots as a benchmark. (a) shows the effect of layer width in a neural network with one hidden layer. (b) keeps the layer width the same but alters the depth of the network. (c) displayed how using approximate forms of \(c\) for ReLU networks effects approximation and contrasts it with sigmoid networks of the same network size.
degree of variability in the CNN response across the range of possible inputs. This explains the need for more planes to approximate the CNN function as there are fewer redundant planes and a diverse set of responses. The smoothness of the response also displays a low sensitivity to noise.

## 5 Discussion and Limitations

It has been shown in this work that an alternate convex and concave representation of any bounded function with a well-defined second derivative can used to construct an approximate form using Max-Affine Splines (MAS), creating a Convex-Concave Spline (CCS) representation. This was displayed for complex non-convex functions and also applied to neural network approximation. This alternate form creates further bridges to approximation theory. It was also shown experimentally that approximate forms can be created for piecewise linear (PWL) functions which possess undefined second derivatives at their points of inflection.

The CCS representation allowed inspection of feature maps. This enabled the inspection and comparison of network function and an approximation of functional complexity. The construction of the MAS also lends itself to highly parallel computation as each spline can be computed independently. Networks with multiple layers and convolutional filers can be flattened into a combination of linear operators. In theory, the same method can be used for individual neurons, although this is left for future work.

Although CCS can model any continuous bounded function with a well-defined second derivative it is still a PWL approximation and the approximation error only tends to zero as the number of planes tends to infinity. In this method a sample is taken at each training data point, which lead to an exact correspondence between the network and the CCS in training data accuracy but lead to test accuracy dropping by around 0.5-1%. In the future, optimisation could be performed to create a more condensed CCS form. Also the Hessian is only evaluated around the training data values which means it may not capture the full potential dynamic range of the network. An exact form was created for the the 1D examples, however, this is left to future work for neural network approximation.

The addition to the second derivative to guarantee convexity is not an optimal choice as \(x^{2}\) can become very large in comparison to the original function. When integrating back the constant was set to zero, choosing a different value could alleviate this problem. In the work of Yuille and Rangarajan (Yuille & Rangarajan, 2003) an energy function is minimised to generate complementary convex and concave functions, although this has not been extended to neural network approximation.

Comparison across architectures enabled the interrogation of network complexity. As is often the problem with features map generation the interpretation is qualitative and left to subjective explanation. Future work would benefit from quantitative descriptions of the extracted feature maps to

Figure 3: The feature maps of different architectures for the MNIST class zero extracted from the CCS representation. A lighter shade corresponds to a positive weight and darker negative.
make precise and objective claims about the networks being investigated. A measure of variability across feature maps would be straightforward to implement. Designing a metric to measure noise present in a feature map would be more complicated when extending beyond image classification.
