Entropy-based Guidance of Deep Neural Networks for Accelerated
  Convergence and Improved Performance
Neural networks have dramatically increased our capacity to learn from large,
high-dimensional datasets across innumerable disciplines. However, their
decisions are not easily interpretable, their computational costs are high, and
building and training them are not straightforward processes. To add structure
to these efforts, we derive new mathematical results to efficiently measure the
changes in entropy as fully-connected and convolutional neural networks process
data. By measuring the change in entropy as networks process data effectively,
patterns critical to a well-performing network can be visualized and
identified. Entropy-based loss terms are developed to improve dense and
convolutional model accuracy and efficiency by promoting the ideal entropy
patterns. Experiments in image compression, image classification, and image
segmentation on benchmark datasets demonstrate these losses guide neural
networks to learn rich latent data representations in fewer dimensions,
converge in fewer training epochs, and achieve higher accuracy.
Mackenzie J. Meni, Ryan T. White, Michael Mayo, Kevin Pilkiewicz
# Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performance

###### Abstract

Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building and training them are uncertain processes. To add structure to these efforts, we derive new mathematical results to efficiently measure the changes in entropy as fully-connected and convolutional neural networks process data, and introduce entropy-based loss terms. Experiments in image compression and image classification on benchmark datasets demonstrate these losses guide neural networks to learn rich latent data representations in fewer dimensions, converge in fewer training epochs, and achieve better test metrics.

## 1 Introduction

In the past 15 years, neural networks have revolutionized our capabilities in computer vision using CNNs [1, 2] and vision Transformers [3], natural language processing [4, 5, 6] with Transformers [7], synthetic data generation with GANs [8] and diffusion models [9], and innumerable other domains.

While neural networks provide immense ability across innumerable applications, their decisions processes are difficult for humans to decipher. More interpretable models could provide actionable rationale for model architecture decisions, increase model efficiency, facilitate better collaboration with domain experts, and lead to stronger understanding of features and their importance. In addition, it would provide trust in neural networks critical to high-stakes use cases that impact human lives. From health care to national security, the ability to determine how these models are making their decisions is crucial.

Complex neural decision processes create further complications. Neural networks often learn from very high dimensional datasets, optimize millions to billions of parameters during training, and propagate data through huge architectures during inference. Hence, training and deploying these large models is demanding from a computational perspective, while explaining their decisions proves even more difficult.

Information theory demonstrates the potential to provide probabilistic explanations of how neural networks process data and make decisions. This can allow us to interpret model decisions, get more from smaller neural architectures to reduce data processing complexity, and optimize them more efficiently. In addition, analysis of patterns in information flow through neural networks provides a pathway to understand what trends during training and/or inference promote good
performance. This leads to information-theoretic metrics to consider and an array of hyperparameters to encourage ideal behavior.

This article seeks to explore these opportunities. We first derive novel probabilistic results for measuring entropy propagation through critical structures within neural networks: fully-connected and 2D convolutional layers. We then use these results to guide the training of neural networks, leading to better convergence times and model performance.

The main contributions of this work include:

* Development of mathematical formulas for the entropy propagation through dense and convolutional layers.
* Construction of novel entropy-based loss terms for both dense and convolutional layers that exploit the entropy change formula to enable entropy-based guidance of neural network training.
* An empirical analysis of entropy propagation patterns through well-trained supervised neural networks, providing an understanding of ideal information flows.
* Experimental demonstrations that entropy-based loss terms speed up convergence in image compression on MNIST and CIFAR10 and improve image classification performance on CIFAR10.

## 2 Related Work

**Information theoretic learning.** Principe et. al. [10] address the idea of information-theoretic learning as a way of decoding how machines learn from data. In this work, they discuss a possible framework utilizing Renyi's quadratic entropy to train linear or nonlinear mappers with the goal of entropy minimization or maximization. In later works [11], the authors continue the search for explainability by proposing the idea of a stochastic entropy estimator based on Parzen window estimates. While both ideas present a usefulness in potential entropy manipulation, the computational complexity of these methods hinder the practical applicability in real-time or resource-constrained environments.

Many additional methods have been suggested in the utilization of information theory to decode computer decisions and processes. Specific to deep learning models, information-theoretic loss functions have been a promising approach. Such loss functions include cross-entropy, F-divergence, MI losses, KL-divergence, and others derived from information theory [12].

**InfoMax.** Alternatively, InfoMax [13, 14, 15], short for Information Maximization, is a learning principle that aims to maximize the mutual information between the input data and some learned representation, typically a latent variable. The core idea is to design models that capture and retain as much information as possible from the input data in the learned representations, with the assumption that this information will be useful for downstream tasks like classification or generation. Deep InfoMax (DIM) [16] was created to estimate and maximize the MI between the input data and the latent representations. However, this has been shown to result in excessive and noisy information [12]. While DIM may lead to representations that capture more information from the input data, it might not guarantee that the learned representations are interpretable or meaningful to humans.

**Information Bottleneck.** One suggested method to mitigate such issues is the Information Bottleneck (IB) principle proposed by Tishby et. al. [17] The IB is an information-theoretic framework for learning representations in a way that balances the amount of information captured from the input data with its relevance to a target output. The main focus of IB is to extract a compressed and informative representation from the input data retaining only the essential information needed to predict the target output [18]. For example, Xu et. al. [19] recently used a "teacher" object detection Transformer model and distill knowledge into an efficient "student" quantized Transformer model. They use an IB-inspired idea to take alternating training steps to minimize the entropy of the student's latent representations conditioned on the teacher's while also maximizing the (unconditional) entropy of the student queries. While effective in such cases, IB assumes that the input-output relationship in the data can be accurately captured by a single target variable. This assumption might not hold true for all real-world scenarios where complex relationships exist between multiple variables.

**Generative AI.** Information theory has also been incorporated into generative AI. Vincent et. al. [20] show minimizing an autoencoder's reconstruction error coincides with maximizing a lower bound on information, similar to InfoMax. They pair this with a denoising criterion to learn latent representations from which the decoder then generates synthetic data. InfoGAN [21] enhances the original generative adversarial network (GAN) loss with a term that maximizes a lower bound for the mutual information between a small subset of the latent variables and the observation. Unlike standard GANs, this provides interpretable latent variables that can be manipulated to generate synthetic data with specific properties (e.g. synthetic MNIST digits from specified classes or with specific rotation or boldness). These efforts demonstrate information-theoretic losses can enable rich, interpretable latent representations that avoid mode collapse for generative neural networks, suggesting their application in supervised domains.
**RLHF and ChatGPT.** Reinforcement learning with human feedback (RLHF) allows reward learning where the rewards are defined by human judgment and has been used in recent years to fine-tune language models to generate better synthetic text. Recent works [22, 23] use human preference labels on text generated by GPT-2 to develop a reward model and train a policy that generates higher-quality text as judged by human preferences. In the reward model, KL divergence between the policy and language is penalized, which serves as an entropy bonus that encourages the policy to explore and avoid collapsing modes in addition to preventing the model from generating text straying far from what the reward model saw during training to maintain coherency and topicality. InstructGPT [6] incorporated this approach into GPT-3.5. Here, information-theoretic loss terms permit meaningfully constrained explorations of the latent space, though the scale of these models (13-175B parameters) results in an extremely high-dimensional latent space that is difficult to interpret.

## 3 Probabilistic Results

In this section, we state some definitions and results from the information-theoretic literature. We then establish several new results that allow the application of these ideas to fully-connected feedforward neural networks (i.e. multilayer perceptrons or MLPs) and convolutional neural networks (CNN). These results enable tracking the evolution of entropy of data as it passes through a neural network.

First, we provide a definition of entropy for use in analyzing neural networks.

**Definition 1**.: The (joint) differential entropy of a continuous random variable \(X\) valued in \(\mathbb{R}^{d}\) with joint probability density function (pdf) \(f\) is

\[H(X)=\mathbb{E}\left[-\log f(X)\right]. \tag{1}\]

Unless otherwise noted, we will use the word _entropy_ to refer to joint differentiable entropy, and we use natural logarithms throughout the article. While the pdf \(f\) of the random variable \(X\) is assumed to exist, it is not assumed to be known. We almost never know the pdf of high-dimensional datasets for computer vision, NLP, or other domains where deep neural networks are effective.

We will represent dense and 2D convolutional neural network layers as matrix-vector products with invertible, constant matrices multiplying random input data or latent representations of that data. Next is a known formula for the entropy propagation of a matrix-vector product \(WX\), where \(X\) is a random vector and \(W\) is a constant matrix. This permits efficient estimation of the change in entropy as data propagatesthrough the dense and convolutional layers of a neural network.

**Theorem 2**.: _(Cover and Thomas [24], Corollary to Theorem 8.6.4) Let \(X\) be a random variable valued in \(\mathbb{R}^{d}\) and constant \(W\in\mathbb{R}^{d\times d}\). If \(W\) is invertible, then the entropy of \(WX\) is_

\[H(WX)=H(X)+\log\left(\left|\det W\right|\right). \tag{2}\]

However, weight matrices in dense layers may not be invertible or even square, so it is unclear how to use this result to measure entropy propagation. Even worse, convolutions are typically not considered a matrix-vector product at all. The next two subsections offer remedies to these issues.

### Dense Layers

The following theorem is a novel result that computes the entropy of pre-activation values \(W^{\prime}X\) as the (unknown) input entropy of \(X\) plus an easily computable term.

**Theorem 3**.: _Suppose \(X:\Omega\rightarrow\mathbb{R}^{d\times N}\) (\(N>d\)) is a random matrix and \(W\in\mathbb{R}^{\min(d,m)\times\min(d,m)}\) is an invertible matrix. Then,_

\[H(W^{\prime}X)=H(X)+\log\left(\left|\det W\right|\right) \tag{3}\]

_where_

\[W^{\prime}=\begin{cases}\begin{pmatrix}W&W_{N\times(d-N)}\\ 0&I_{d-N}\end{pmatrix},&\text{if }N<d\\ W,&\text{if }N=d\\ \begin{pmatrix}W&0\\ W_{(N-d)\times d}&I_{N-d}\end{pmatrix},&\text{if }N>d\end{cases} \tag{4}\]
Proof.: We refer to \(W\) as the square part of \(W^{\prime}\). Note \(W^{\prime}\) is block upper diagonal, square, or block lower diagonal, depending on the input dimension \(d\) and output dimension \(N\). In any of the three cases,

\[\det\left(W^{\prime}\right)=\det\left(W\right).\]

Since \(W\) is invertible, \(\det\left(W^{\prime}\right)=\det\left(W\right)>0\), so \(W^{\prime}\) is invertible. Then, Theorem 2 implies

\[H(W^{\prime}X)=H(X)+\log\left(\left|\det W^{\prime}\right|\right)=H(X)+\log \left(\left|\det W\right|\right).\]

This formula clarifies how entropy propagates from input to pre-activations within dense layers.

**Example 4**.: If a weight matrix has more columns than rows (\(3=N<d=5\) in this case), Theorem 3 suggests modifying it as:

\[\text{Weight Matrix}=\begin{pmatrix}3&0&9&-3&4\\ 1&5&-1&4&2\\ 0&4&-2&1&5\end{pmatrix}\rightarrow\left(\begin{array}{cccc|c}3&0&9&-3&4\\ 1&5&-1&4&2\\ 0&4&-2&1&5\\ \hline 0&0&0&1\\ \end{array}\right)=\begin{pmatrix}W&W_{3\times 2}\\ 0&I_{2}\end{pmatrix}=W^{\prime},\]

while a similar manipulation occurs when there are more rows than columns, with a block of zeros placed on the upper right and identity block placed on the lower right.

### 2D Convolutions

Consider an \(p\)-by-\(q\) convolutional filter \(C\). If \(C\) convolves with strides \(1\times 1\) over an input image \(X\in\mathbb{R}^{l\times w}\), then an output pre-activation map \(Z=C*X\) is defined as

\[Z_{ij}=\sum_{k=0}^{p}\sum_{l=0}^{q}C_{kl}X_{i+k,j+l} \tag{5}\]

for \(i=0,1,...,l-p\) and \(j=0,1,...,w-q\). While this is typically envisioned as the 2D convolutional filter \(C\) scanning over the image \(X\) to extract features denoted, the operation can alternatively be represented as a matrix-vector product. We will establish \(Z_{F}=C_{M}X_{F}\), where \(X_{F}\) and \(Z_{F}\) are flattened versions of \(X\) and \(Z\) and \(C_{M}\) has a special structure constructed below.

First, we transpose the rows \(x_{i}^{T}\) of matrix \(X\in\mathbb{R}^{l\times w}\) into column vectors and then concatenate them into a single column vector \(X_{F}\in\mathbb{R}^{lw}\):

\[X_{F}=\texttt{flatten}(X)=\begin{pmatrix}x_{1}\\ \vdots\\ x_{l}\end{pmatrix}_{lw\times 1} \tag{6}\]

In addition, we will consider an arbitrary \(p\times q\) convolutional filter \(C\) made up of rows \(c_{1}^{T}\),..., \(c_{p}^{T}\). With these pieces, we show a numerical example to motivate the path to manipulating a 2D convolution operation into a matrix-vector product.

**Example 5**.: Suppose we have a small gray-scale image \(X\in\mathbb{R}^{4\times 4}\) of dimensions a convolution \(C\in\mathbb{R}^{3\times 2}\) with the resulting feature map \(Z=C*X\):

\[X=\begin{pmatrix}3&4&1&2\\ 0&0&5&6\\ 2&1&0&3\\ 1&4&2&5\end{pmatrix}\qquad\qquad C=\begin{pmatrix}2&1\\ 4&3\\ -2&1\end{pmatrix}\qquad\qquad Z=C*X=\begin{pmatrix}7&22&45\\ 13&3&26\end{pmatrix}\]
Alternatively, the operation can be defined as \(Z_{F}=C_{M}X_{F}\) where

\[C_{M}X_{F}=\left(\begin{array}{cccc|cccccccc|cccccccc}2&1&0&0&4&3&0&0&-2&1&0&0& 0&0&0&0\\ 0&2&1&0&0&4&3&0&0&-2&1&0&0&0&0&0\\ 0&0&2&1&0&0&4&3&0&0&-2&1&0&0&0&0\\ \hline 0&0&0&2&1&0&0&4&3&0&0&-2&1&0&0\\ 0&0&0&0&0&2&1&0&0&4&3&0&0&-2&1&0\\ 0&0&0&0&0&2&1&0&0&4&3&0&0&-2&1\end{array}\right)\left(\begin{array}{c}3\\ 4\\ 1\\ 2\\ 0\\ 5\\ 6\\ 2\\ 1\\ 3\\ 3\\ 1\\ 4\\ 2\\ 5\end{array}\right)=\left(\begin{array}{c}7\\ 2\\ 54\\ 1\\ 1\\ 2\\ 3\\ 26\end{array}\right)=Z_{F}\]

Annotating the blocks, this simplifies as

\[C_{M}X_{F}=\left(\begin{array}{cccc}B_{1}&B_{2}&B_{3}&0\\ 0&B_{1}&B_{2}&B_{3}\\ \end{array}\right)\left(\begin{array}{c}x_{1}\\ x_{2}\\ x_{3}\\ x_{4}\end{array}\right)=\left(\begin{array}{c}z_{1}\\ z_{2}\\ \end{array}\right)=Z_{F}\]

Reshaping \(Z_{F}\) to \(2\times 3\), we reconstruct \(Z\) as follows.

\[Z=\texttt{reshape}\left(Z_{F}\right)=\left(\begin{array}{c}z_{1}^{T}\\ z_{2}^{T}\end{array}\right)=\left(\begin{array}{rrrrrrrr}7&22&45\\ 13&3&26\end{array}\right).\]

Next, use of Theorem \(3\) to measure entropy propagation through a multiplication by a constant matrix \(C_{M}\) requires us to construct a square version of \(C_{M}\). We do this by adding identity matrices on the lower right of each block \(B_{j}\) as per (4) in the case where \(N<d\) (\(I_{1}\) in this case). Then, we do the same with \(C_{M}\) consisting of the square \(B_{j}^{\prime}\) blocks, appending zero blocks and identity matrices (\(I_{8}\) in this case). This results in the following.

\[C_{M}^{\prime}=\left(\begin{array}{cccc|cccc|cccc|cccc}2&1&0&0&4&3&0&0&-2&1& 0&0&0&0&0&0\\ 0&2&1&0&0&4&3&0&0&-2&1&0&0&0&0&0\\ 0&0&2&1&0&0&4&3&0&0&-2&1&0&0&0&0\\ 0&0&0&1&0&0&0&1&0&0&0&1&0&0&0&0\\ \hline 0&0&0&0&1&2&1&0&4&3&0&0&-2&1&0&0\\ 0&0&0&0&0&2&1&0&0&4&3&0&0&-2&1&0\\ 0&0&0&0&0&0&2&1&0&0&4&3&0&0&-2&1\\ 0&0&0&0&0&1&0&0&1&0&0&1&0&0&0&1\\ 0&0&0&0&0&0&0&1&1&0&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&1&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&1&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&1&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&1&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&1&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&0&1\end{array}\right)\left(\begin{array}{c}2&1&0&0& 0&0&0&0&0&0&0\\ 0&2&1&0&0&4&3&0&0&-2&1&0\\ 0&0&0&0&0&0&2&1&0&0&4&3&0&0&-2&1\\ 0&0&0&0&0&0&1&0&0&0&1&0&0&1\\ 0&0&0&0&0&0&0&0&1&0&0&0&0&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&1&0\\ 0&0&0&0&0&0&0&0&0&0&0&0&0&1\end{array}\right)\]

Since \(C_{M}^{\prime}\) is an upper diagonal square matrix, its determinant is the product of its diagonal elements. In general, it will be equal to \(c_{11}\) raised to the power of the number of rows of blocks (2) times the number of rows per block (3), where \(c_{11}\) is the upper left term in the convolutional filter \(C\). Here, \(\det C_{M}^{\prime}=c_{11}^{2,3}=2^{6}=64\). This determinant allows use of Theorem \(2\) to measure entropy propagation.
Generalizing the pattern observed in the example, we can construct the matrix \(C_{M}\) as

\[C_{M}=\begin{pmatrix}B_{1}&B_{2}&B_{3}&\cdots&B_{p}&0&0&\cdots&0\\ 0&B_{1}&B_{2}&\cdots&B_{p-1}&R_{p}&0&\cdots&0\\ 0&0&B_{1}&\cdots&B_{p-2}&B_{p-1}&B_{p}&\cdots&0\\ \vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\vdots&\ddots&\vdots\\ 0&0&0&\cdots&B_{1}&B_{2}&B_{3}&\cdots&B_{p}\end{pmatrix}=\left(C_{M}^{s} \quad C_{M}^{r}\right), \tag{7}\]

where

\[B_{j}=\begin{pmatrix}c_{j1}&c_{j2}&c_{j3}&\cdots&c_{jq}&0&0& \cdots&0\\ 0&c_{j1}&c_{j2}&\cdots&c_{j,q-1}&c_{jq}&0&\cdots&0\\ 0&0&c_{j1}&\cdots&c_{j,q-2}&c_{j,q-1}&c_{jq}&\cdots&0\\ \vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\vdots&\ddots&\vdots\\ 0&0&0&\cdots&c_{j1}&c_{j2}&c_{j3}&\cdots&c_{jq}\end{pmatrix}=\left(B_{j}^{s} \quad B_{j}^{r}\right). \tag{8}\]

The matrix \(C_{M}\) and the blocks \(B_{j}\) both have more columns (width) than rows (length). In both cases, a superscript of \(s\) indicates the largest square submatrix made by taking columns 1 through the length of the matrix. For example, the square portion \(B_{j}^{s}\) is the first \(w-q+1\) columns of \(B_{j}\) and \(B_{j}^{r}\) is the remaining \(q-1\) columns (the rectangular portion). In other words, \(C_{M}\) is a \(p\)-diagonal block Toeplitz matrix with blocks \(B_{1}\),..., \(B_{p}\), which are \(q\)-diagonal Toeplitz matrices.

Similar to Theorem 3, we convert \(C_{M}\) into a square version whose determinant is determined entirely by the elements of \(C_{M}\). First, the blocks are adjusted to \(w\times w\) as

\[B_{j}^{\prime}=\begin{pmatrix}B_{j}^{s}&B_{j}^{r}\\ 0&I_{q-1}\end{pmatrix}, \tag{9}\]

Then, we have the resulting \(lw\times lw\) matrix \(C_{M}^{\prime}\),

\[C_{M}^{\prime}=\begin{pmatrix}C_{sq}^{s}&C_{sq}^{r}\\ 0&I_{(p-1)w}\end{pmatrix} \tag{10}\]

The output pre-activation map \(Z\) is now obtained via the matrix-vector product \(C_{M}^{\prime}X_{C}\) if we then reshape the result into shape \((l-p+1)\times(w-q+1)\) by transposing each successive components into rows. Thus, applying a 2D convolution is equivalent to matrix-vector products, just like dense layers. This gives the following corollary to the previous theorem.

**Corollary 6**.: _Suppose \(X:\Omega\rightarrow\mathbb{R}^{l\times w}\) is a random matrix with rows \(X_{1}^{T},...,X_{l}^{T}\), then_

\[H(C*X) =H(X)+\log\left(|\det C_{M}^{\prime}|\right)\] \[=H(X)+(w-q+1)(l-p+1)\log\left(|c_{11}|\right) \tag{11}\]

Hence, the change in entropy in applying a single 2D convolutional filter is proportional to the difference between the filter and image width, difference between filter and image length, and the logarithm of the magnitude of the first weight in the filter.

## 4 Entropy-Based Guidance of Dense and Convolutional Neural Networks

The prior section established formulas for the entropy of hidden representations within dense and convolutional neural networks. In each case, the entropy is unknown, but the change in entropy can be computed using known parameters and hyperparameters.

The article proposes to guide the training of neural networks to produce ideal entropy propagation patterns. This provides a new lens through which models can be constructed, trained, and tuned. In addition, we provide two distinct tools to guide the training by controlling entropy propagation through:

1. Dense layers via a loss using determinants of modified weight matrices \(W^{\prime}\).
2. 2D convolutional layers via a loss using determinants of modified convolutional operations using \(C_{M}^{\prime}\).
These new loss terms will be used to construct a compound loss function:

\[L(\mathcal{C},\mathcal{W})=L_{\text{acc}}(\mathcal{C},\mathcal{W})+\lambda_{1}L_{ \text{dense}}(\mathcal{W})+\lambda_{2}L_{\text{conv}}(\mathcal{C}), \tag{12}\]

where \(\mathcal{C}\) consists of the 2D convolutional filters, and \(\mathcal{W}\) consists of the weight matrices of the dense layers.

\(L_{\text{acc}}\) is a standard loss measuring error of the model's primary task; for example, mean squared error (MSE) for regression or cross-entropy for classification. \(L_{\text{dense}}\), and \(L_{\text{conv}}\) are the entropy-based loss terms from dense and convolutional layers, respectively. The hyperparameters \(\lambda_{1}\) and \(\lambda_{2}\) control the strengths of the loss terms relative to the primary loss \(L_{\text{acc}}\). The specific formulas and variations for each entropy-based loss term is established in the following three subsections. Note the losses below generalize this to accommodate fine-grained tuning with layer-specific and channel-specific hyperparameters \(\lambda_{1}^{\ell}\) and \(\lambda_{2}^{\ell d}\).

### Dense Entropy Loss

From Theorem 3, we determined how entropy changes as dense layers process data. It is established that the term

\[\log\left(\left|\det W\right|\right) \tag{13}\]

describes the change in entropy. An entropy-based loss term for dense layers is therefore:

\[L_{\text{dense}}(\mathcal{W})=-\sum_{\ell}\lambda_{1}^{\ell}\log\left(\left| \det W_{\ell}\right|\right) \tag{14}\]

where \(\lambda_{1}^{\ell}\in\mathbb{R}\) are hyperparameters controlling the strength of the penalty in the \(\ell\)th dense layer, generalizing the single \(\lambda_{1}\) hyperparameter if losses are applied layer-wise. When \(\lambda_{1}^{\ell}>0\), the added loss term encourages weight matrices to reduce entropy. This provides the opportunity to minimize loss further or faster, and opens the door for smaller, more efficient architectures.

### 2D Convolutional Entropy Loss

According to Corollary 6, the change in entropy as an input is processed by a 2D convolution is

\[(l-p+1)(w-q+1)\log|c_{11}|, \tag{15}\]

so we introduce a loss term proportional to \(\log|c_{11}|\) for each convolutional filter.

Suppose the term \(c_{ij}^{\ell d}\) corresponds to the element in position \((i,j)\) of the \(d\)th convolutional channel of the \(\ell\)th convolutional layer. Then, the entropy-based loss term is

\[L_{\text{conv}}(\mathcal{C})=-\sum_{\ell,d}\lambda_{2}^{\ell d}\log\left( \left|c_{11}^{\ell d}\right|\right) \tag{16}\]

where \(\lambda_{2}^{\ell d}\in\mathbb{R}\) are layer- and channel-wise weighting hyperparameters, generalizing the simpler global \(\lambda_{2}\) from (4). If the \(\lambda_{2}^{\ell d}>0\) decreases in entropy will be penalized, resulting in entropy amplification/preservation. Similarly, \(\lambda_{2}^{\ell d}<0\) results in entropy suppression.

Note that we are considering all convolutions as convolving over single-channel 2D inputs. However, 2D convolutional filters running over multi-channel inputs (e.g. RGB images) are equivalent to several parallel 2D convolutions, or convolutions over different channels. Hence, the formula above works regardless of the number of channels of the input.

## 5 Experiments

We first carried out some qualitative analysis of entropy propagation patterns in well-trained, effective neural networks.
The left panel of Figure 1 displays the change in entropy per filter at each layer within a VGG16 CNN well-trained to classify the ImageNet dataset. [25] It preserves most of its entropy in its early convolutional layers, with mean entropy drops of only around -2.2 in the first layer. The entropy then drops precipitously as data propagates through the later convolutional layers. The entropy drops magnitudes drop exponentially and settle near -5.5 per filter. In contrast, the right panel shows that the randomly initialized (untrained) network fails to preserve early entropy as strongly and experiences entropy drops that increase only linearly in magnitude in later layers.

There is also an interesting pattern in the outlier filters. The ImageNet-trained network has few outliers, all of which correspond to larger drops in entropy than the norm. They are rare especially later in the network. In contrast, the randomly initialized network has far more outliers, again all corresponding to reductions in entropy. Further the outliers are especially common in the later layers. These trends suggest well-trained networks learn filters reduce entropy more uniformly across filters.

These pattern was observed across multiple well-trained neural networks, and hence we hypothesize penalizing entropy decay in early layers and encouraging entropy drops in later layers would promote better performance.

### Experimental Modification of Entropy-based Losses

The dense entropy-based loss term includes \(-\log(|\det W|)\), which approaches infinity exponentially if \(|\det W|\) approaches 0. To avoid exploding gradients, weights tend to be small, resulting in frequent tiny determinants. Even worse, the convolutional entropy-based loss term is a sum of \(-\log\left(|c_{11}^{\ell d}|\right)\) terms, which approach infinity if even a single \(c_{11}^{\ell d}\) approaches 0.

Figure 1: This plot shows the average change in entropy per filter in each layer of two VGG16 networks, one trained to classify ImageNet and one randomly initialized. The closed dots are means, box plots show first and third quartiles of entropy change per filter at each layer, and outliers are plotted as open dots.
To sidestep this issue, we frequently substitute the entropy-based loss terms above with similar functions that are more stable as follows.

\[\frac{1}{|\det W|+\varepsilon}\] (dense) \[\frac{1}{|c_{11}^{Id}|+\varepsilon}\] (convolutional)

Figure 2 shows the curves behave similarly near 0, but the loss terms modified with \(\varepsilon\) do not explode, even with the tiny determinants and \(|c_{11}|\) values seen during training. 1 Though the curves have different signs for inputs outside \([-1,1]\), these values are quite uncommon to ensure the stability of backpropagation.

Footnote 1: Note \(\varepsilon=0.5\) is too large in practice and we use \(\varepsilon<10^{-3}\), but it was chosen to make the plot more easily visible.

In addition, \(\frac{1}{|x|}\) has a larger derivative near 0 than \(\log(|x|)\). Note in the first panel of Figure 2, \(\frac{1}{|x|=\varepsilon}\) approaches its maximum at \(x=0\) more steeply. The second panel shows its absolute derivative more extreme in this location. This amplifies the gradients computed during training to help avoid the vanishing gradient problem, encouraging quicker convergence to better minima.

### Dense Autoencoders for Image Compression

To test the efficacy of the entropy-based loss function for dense layers, we trained simple autoencoders with different values of \(\lambda_{1}\) and hidden dimensions for image compression using two benchmark datasets, MNIST [26] and CIFAR-10 [27]. The \(L_{\text{acc}}\) and number of iterations required for convergence were then compared.

The autoencoder includes an input layer, one dense hidden layer, and an output layer. It maps the input data into a latent space, passes this latent representation through an activation function, and then reconstructs the input data with another dense layer. The overall goal is to reduce dimensionality of the input data by maximizing the total variation of that latent space for effective reconstruction. Models such as these can be beneficial in decreasing model sizes, detecting anomalies, denoising, and potentially making downstream tasks more interpretable.

MNIST is a classic benchmark dataset that includes 70,000 labeled images of handwritten digits from 0 to 9. Each image is grayscale, contains a centered handwritten digit, and is 28x28 pixels (784 dimensions). For higher dimensional experiments, we use CIFAR-10, a benchmark dataset of 60,000 tiny color images belonging to 10 classes of objects, including cats and airplanes. Images are all RGB and 32x32 pixel format with a total dimension of 3072. Training on these datasets allows us to assess the implications of the input dimensionality.
Autoencoder models were trained on each dataset separately with the same activation and optimizer to ensure fair comparison. In this experiment, we compare results on latent dimension widths of 20, 60, 100,..., 260 and \(\lambda_{1}\) values in \(\{0,0.0001,0.001,0.01,0.1,1,10\}\). Each autoencoder uses Adam optimizer, sigmoid activation, and MSE for base \(L_{\text{acc}}\). We use early stopping to end training when the loss saturates for 5 epochs.

Use of the the dense entropy-based loss term results in convergence up to 4x faster to minima within \(10^{-3}\) of the standard MSE reconstruction error, often less than models trained without entropy loss.

Figure 3 depicts the number of iterations the models required to converge with different latent dimensions. The dashed lines show autoencoders trained without the addition of the dense entropy-based loss. Note that the number of iterations they take is much higher than the models trained with nonzero \(\lambda_{1}\), while maintaining similar and often smaller test \(L_{\text{acc}}\) (MSE reconstruction error).

This speed-up occurs when the autoencoder is sufficiently wide. The networks have a harder time converging when the latent dimension is 20. However, when we widen the latent dimension to 60, we have the ability to find the minima at a faster rate. This can provide guidance when choosing the width of these networks as convergence.

### CNN Image Classifiers

Next, we perform image classification experiments on the benchmark dataset CIFAR10 with the entropy-based loss function for convolutional layers, \(L_{\text{conv}}\). Here, we explored the impact of the convolutional entropy-based loss term with different weighting hyperparameters \(\lambda_{2}\) for CNN classifiers of varying widths and depths.

CNNs in these experiments have 1-3 successive blocks containing 1 convolutional layer (filter size \(3\times 3\), stride \(1\times 1\)) and 1 max pooling layer (size \(2\times 2\), stride \(2\times 2\)) followed by a softmax classifier. Leaky ReLU activations are used with each convolutional layer. Widths of 32, 64, and 128 convolutional filters in each layer are tested with each depth. The CNNs use cross-entropy for the base classification loss \(L_{\text{acc}}\) and are trained with the Adam optimizer. Each architecture

Figure 3: This plot shows how the addition of \(L_{\text{dense}}(\mathcal{W})\) affects the training speed and accuracy of an autoencoder on MNIST and CIFAR-10 data.
is trained with varying \(\lambda_{2}\in\{0,0.0001,0.001,0.01,0.1,1,10\}\), applied in only the first convolutional layer in one set of experiments and applied to all layers in another set.

The pattern in entropy changes in the high-quality VGG classifier observed in Figure 1 prompted a hypothesis that encouraging entropy preservation in the early layers will have a positive impact on classification accuracy. The results of experiments with the entropy-based loss applied only to the first convolutional layer are shown in Table 1.

As expected, the base performance tends to improve with more filters and depth. Training accuracy almost always improves with the entropy-based loss, with gains as high as 6.6%, where more depth and width enhance the benefits. These gains are not preserved in test accuracy in shallower nets, but we see up to 2.8% gains in test accuracy in deeper nets.

Drilling down into the deepest 3-convolutional block nets, Figure 4 (left) displays a strongly positive relationship between the \(\lambda_{2}\) values and training performance, particularly the wider nets with 64 and 128 filters per layer. On the right, note test accuracy always peaks with nonzero \(\lambda_{2}\), indicating improved classification performance when the entropy-based loss is enabled.

Interestingly, applying the entropy-based loss to all layers with the same weighting hyperparameters yields no significant gains, neither for train nor test accuracy. This supports the hypothesis that encouraging entropy preservation in the early layers trains the CNN to extract higher-quality latent representations of the input data, enabling better downstream classification performance.

\begin{table}
\begin{tabular}{l|c|c|c|c|} Architecture & Base Train Acc. & Base Test Acc. & \(\Delta\) Train Acc. & \(\Delta\) Test Acc. \\ \hline \hline \([32]\) & 0.7616 & 0.6479 & \(-0.0002\) & \(-0.0004\) \\ \([64]\) & 0.7779 & 0.6441 & +0.0229 & +0.0121 \\ \([128]\) & 0.7543 & 0.6519 & +0.0307 & \(-0.0013\) \\ \hline \([32,32]\) & 0.7211 & 0.6648 & +0.0384 & +0.0168 \\ \([64,64]\) & 0.7972 & 0.7050 & +0.0123 & +0.0013 \\ \([128,128]\) & 0.8451 & 0.7064 & +0.0167 & +0.0041 \\ \hline \([32,32,32]\) & 0.7265 & 0.6737 & +0.0414 & +0.0156 \\ \([64,64,64]\) & 0.7952 & 0.7012 & +0.0483 & +0.0091 \\ \([128,128,128]\) & 0.8389 & 0.7098 & +0.0660 & +0.0280 \\ \end{tabular}
\end{table}
Table 1: Experimental results for CIFAR10 classification with CNNs. The base train and test accuracy for \(\lambda_{2}=0\) is displayed alongside the best gain in accuracies with \(\lambda_{2}>0\) applied to the first convolutional layer.

Figure 4: Train and test accuracies on CIFAR10 classification for CNNs with three convolutional blocks and different entropy-based loss hyperparameters.
## 6 Conclusion

This article addresses the complexity of the decision making process of neural networks by utilizing information theory. We derived novel information theoretic results of entropy propagation through dense and 2D convolutional layers. These results provided a foundation for entropy-based loss terms that allow us to guide the neural decision process.

We analyzed the performance of networks with these loss terms by performing experiments on image compression and image classification tasks on benchmark datasets MNIST and CIFAR10. Specifically, the incorporation of the \(L_{\text{dense}}(\mathcal{W})\) term in an autoencoder for image compression showed increase in convergence speed and often provided smaller minima. Additionally, performance gains are demonstrated on image classification with CNNs utilizing the \(L_{\text{conv}}(\mathcal{C})\) term, and we confirm the hypothesis that encouraging entropy preservation in early layers promotes better downstream performance.

This work provides strong foundational findings that allow practical information-theoretic guidance of neural networks. It can allow theory-backed, principled construction of neural architectures, specifically in terms of depth, width, and layer structure.

Related works have often searched for similar methods, but have been focused on the estimation of entropy. This often leads to highly complex calculations that can be difficult for many practical use cases. Our work avoids these difficult estimations of entropy, and focuses instead on the more easily computable change in entropy, localized to specific layers and channels of the network. This calculation is cheap in comparison, and can be used in larger practical models.

This work took strides towards practical use of information-theoretic guidance of neural network training. This suggests that exploring larger models, incorporating diverse architectural components, and using higher-dimensional datasets could yield valuable insights for building and training efficient neural networks. Additionally, such explorations could provide a deeper understanding of the decision-making processes within them.

## Acknowledgement

The authors would like to thank Olivia Raney for her invaluable editorial improvements to this article. M. Meni would also like to thank Dr. Kaleb Smith for his willingness to share his knowledge and provide thoughtful recommendations to overcome challenges. R. T. White wishes to thank the NVIDIA Applied Research Accelerator Program for providing hardware support for this effort.
