Correlative Information Maximization: A Biologically Plausible Approach
  to Supervised Deep Neural Networks without Weight Symmetry
The backpropagation algorithm has experienced remarkable success in training
large-scale artificial neural networks; however, its biological plausibility
has been strongly criticized, and it remains an open question whether the brain
employs supervised learning mechanisms akin to it. Here, we propose correlative
information maximization between layer activations as an alternative normative
approach to describe the signal propagation in biological neural networks in
both forward and backward directions. This new framework addresses many
concerns about the biological-plausibility of conventional artificial neural
networks and the backpropagation algorithm. The coordinate descent-based
optimization of the corresponding objective, combined with the mean square
error loss function for fitting labeled supervision data, gives rise to a
neural network structure that emulates a more biologically realistic network of
multi-compartment pyramidal neurons with dendritic processing and lateral
inhibitory neurons. Furthermore, our approach provides a natural resolution to
the weight symmetry problem between forward and backward signal propagation
paths, a significant critique against the plausibility of the conventional
backpropagation algorithm. This is achieved by leveraging two alternative, yet
equivalent forms of the correlative mutual information objective. These
alternatives intrinsically lead to forward and backward prediction networks
without weight symmetry issues, providing a compelling solution to this
long-standing challenge.
Bariscan Bozkurt, Cengiz Pehlevan, Alper T Erdogan
Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

###### Abstract

The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks, however, its biological-plausibility is disputed, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartmental pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.

## 1 Introduction

How biological neural networks learn in a supervised manner has long been an open problem. The backpropagation algorithm Rumelhart et al. (1986), with its remarkable success in training large-scale artificial neural networks and intuitive structure, has inspired proposals for how biologically plausible neural networks can perform the necessary efficient credit-assignment for supervised learning in deep neural architectures (Whittington and Bogacz, 2019). Nonetheless, certain aspects of the backpropagation algorithm, combined with the oversimplified nature of artificial neurons, have been viewed as impediments to proposals rooted in this inspiration Crick (1989).

One of the primary critiques regarding the biological plausibility of the backpropagation algorithm is the existence of a parallel backward path for backpropagating error from the output towards the input, which uses the same synaptic weights as the forward path (Rumelhart et al., 1986; Whittington and Bogacz, 2019; Grossberg, 1987). Although such weight transport, or weight symmetry, is deemed highly unlikely based on experimental evidence Crick (1989), Grossberg (1987), some biologically plausible frameworks still exhibit this feature, which is justified by the symmetric structure of the
Hebbian updates employed in these frameworks Whittington and Bogacz (2019); Xie and Seung (2003); Scellier and Bengio (2017).

The concerns about the simplicity of artificial neurons have been addressed by models which incorporate multi-compartment neuron models into networked architectures and ascribe important functions to dendritic processing in credit assignment (Larkum, 2013; Urbanczik and Senn, 2014; Sacramento et al., 2018; Golkar et al., 2022). This new perspective has enabled the development of neural networks with improved biological plausibility.

In this article, we propose the use of correlative information maximization (CorInfoMax) among consecutive layers of a neural network as a new supervised objective for biologically plausible models, which offers

* a principled solution to the weight symmetry problem: our proposed information theoretic criterion aims to maximize the linear dependence between the signals in two neighboring layers, naturally leading to the use of linear or affine transformations in between them. A key property of this approach is that employing two alternative expressions for the correlative mutual information (CMI) results in potentially _asymmetric forward and backward prediction networks_, offering a natural solution to the weight transport problem. Consequently, predictive coding in both directions emerges as the inherent solution to the correlative information maximization principle, fostering signal transmission in both forward and top-down directions through asymmetrical connections. While the CorInfoMax principle enhances information flow in both directions, the introduction of set membership constraints on the layer activations, such as non-negativity, through activation nonlinearities and lateral inhibitions, encourages compression of information and sparse representations Bozkurt et al. (2023).
* a normative approach for deriving networks with multi-compartment neurons: the gradient search-based optimization of the CorInfoMax objective naturally leads to network models that employ multi-compartment pyramidal neuron models accompanied by interneurons as illustrated in Figure 1.

As derived and explained in detail in Section 2, the resulting networks incorporate lateral connections and auto-synapses to increase the entropy of a layer, promoting utilization of all dimensions within the representation space of that layer. Meanwhile, asymmetric feedforward and feedback connections act as forward and backward predictors, respectively, to reduce the conditional entropies between layers, targeting the elimination of redundancy.

### Related work

#### 1.1.1 Multi-compartmental neuron model based biologically plausible approaches

Experimentally grounded studies, such as (Larkum, 2013; Petreanu et al., 2009), have been influential for considering a role for dendritic-processing in multi-compartmental neurons for learning and credit assignment Richards and Lillicrap (2019). Subsequent research has explored biologically plausible models with supervised learning functionality, such as the two-compartment neuron model by Urbanczik and Senn (2014) and the three-compartment pyramidal neuron model by Sacramento et al. (2018). Both models integrate non-Hebbian learning and spike-time dependent plasticity, while the latter includes SST interneurons (Urban-Ciecko and Barth, 2016). Similar frameworks have been proposed by (Guerguiev et al., 2017) and (Golkar et al., 2022), with the latter introducing a normative framework based on multi-compartmental neuron structure, top-down feedback, lateral and feedforward connections, and Hebbian and non-Hebbian learning rules, emerging from the optimization of a prediction error objective with a whitening constraint on co-layer neurons.

In a similar vein to (Golkar et al., 2022), we propose an alternative normative framework based on information maximization principle. In this framework, the three-compartment structure and associated forward, top-down and lateral synaptic connections stem from the maximization of CMI between adjacent layers, without the imposition of any whitening constraint.

#### 1.1.2 Weight symmetry problem

A central concern regarding the biological plausibility of the backpropagation algorithm pertains to the weight symmetry issue: synaptic weights in the feedback path for error backpropagation are transposes of those used in the forward inference path (Whittington and Bogacz, 2019; Crick, 1989; Grossberg, 1987). The requirement of tied weights in backpropagation is questionable for
physically distinct feedforward and feedback paths in biological systems, leading many researchers to focus on addressing the weight symmetry issue.

Various strategies have been devised to address the weight symmetry issue, encompassing the employment of random and fixed feedback weights (Lillicrap et al., 2016), and the introduction of antisymmetry through separate random initializations (Amit, 2019). Liao et al. (2015) showed that the sign of the feedback weights (rather than their magnitude) affects the learning performance, and proposed the sign-symmetry algorithm.

Intriguingly, this symmetric weight structure is also observed in biologically plausible frameworks such as predictive coding (PC) (Rao and Ballard, 1999, Whittington and Bogacz, 2017, Song et al., 2020), equilibrium propagation (EP) (Scellier and Bengio, 2017b, Laborieux et al., 2021, Laborieux and Zenke, 2022), and similarity matching (Qin et al., 2021). This phenomenon can be rationalized by the transpose symmetry of the Hebbian update with respect to inputs and outputs. The EP framework in (Laborieux et al., 2021) unties forward and backward connections inspired by (Scellier et al., 2018, Kolen and Pollack, 1994), and only yields small performance degradation. A more recent approach by Golkar et al. (2022) addresses this challenge by integrating two alternative forward prediction error loss function terms associated with the same network layer and leveraging presumed whitening constraints to eliminate shared feedback coefficients.

In existing predictive coding-based schemes such as (Rao and Ballard, 1999, Whittington and Bogacz, 2017, Song et al., 2020), the loss function contains only forward prediction error terms. The feedback connection with symmetric weights, which backpropagates forward prediction error, emerges due to the gradient-based optimization of the PC loss. In contrast, our framework's crucial contribution is the adoption of two alternative expressions for the correlative mutual information between consecutive network layers as the central normative approach. Utilizing these two alternatives naturally leads to both forward and backward prediction paths with asymmetric weights, promoting information flow in both feedforward and top-down directions. Unlike the work of (Golkar et al., 2022), our method circumvents the need for layer whitening constraints and additional forward prediction terms to achieve asymmetric weights.
#### 1.1.3 Correlative information maximization

Information maximization has been proposed as a governing or guiding principle in several machine learning and neuroscience frameworks for different tasks: (i) The propagation of information within a self-organized network as pioneered by Linsker (1988). (ii) Extracting hidden features or factors associated with observations by maximizing information between the input and its internal representation such as independent component analysis (ICA-InfoMax) approach by Bell and Sejnowski (1995). In the neuroscience domain, the motivation has been to provide normative explanations to the behaviour of cortical activities evidenced by experimental work, such as orientation and visual stimuli length selectivity of primary visual cortex neurons (Hubel and Wiesel, 1959; Bell and Sejnowski, 1997). The same idea has been recently extended in the machine learning field by the Deep Infomax approach where the goal is to transfer maximum information from the input of a deep network to its final layer, while satisfying prior distribution constraints on the output representations (Hjelm et al., 2019). (iii) Matching representations corresponding to two alternative augmentations or modalities of the same input in the context of self-supervised learning (Becker and Hinton, 1992).

Correlative mutual information maximization has been recently proposed as an alternative for Shannon Mutual Information (SMI), due to its desirable properties (Erdogan, 2022): (i) maximization of CMI is equivalent to maximizing linear dependence, which may be more relevant than establishing arbitrary nonlinear dependence in certain applications (Ozsoy et al., 2022), (ii) it is based only on the second order statistics, making it relatively easier to optimize. Erdogan (2022) proposed the use of CorInfoMax for solving blind source separation (BSS) problem to retrieve potentially correlated components from their mixtures. Ozsoy et al. (2022) proposed maximizing the CMI between the representations of two different augmentations of the same input as a self-supervised learning approach. More recently, Bozkurt et al. (2023) introduced an unsupervised framework to generate biologically plausible neural networks for the BSS problem with infinitely many domain selections using the CMI objective.

In this article, we suggest employing the CorInfoMax principle for biologically plausible supervised learning. The key difference compared to the unsupervised framework presented in (Bozkurt et al., 2023) is the utilization of two alternative forms of mutual information. This leads to a bidirectional information flow that enables error backpropagation without encountering the weight symmetry issue.

## 2 Deep correlative information maximization

### Network data model

We assume a dataset with \(L\) input data points \(\mathbf{x}[t]\in\mathbb{R}^{m},t=1,\ldots,L\), and let \(\mathbf{y}_{T}[t]\in\mathbb{R}^{n}\) be the corresponding labels. We consider a neural network with \(P-1\) hidden layers whose activities are denoted by \(\mathbf{r}^{(k)}\in\mathbb{R}^{N_{k}},k=1,\ldots,P-1\). For notational simplicity, we also denote input and output of the network by \(\mathbf{r}^{(0)}\) and \(\mathbf{r}^{(P)}\), i.e., \(\mathbf{r}^{(0)}[t]=\mathbf{x}[t]\) and \(\mathbf{r}^{(P)}[t]=\hat{\mathbf{y}}[t]\). We consider polytopic constraints for the hidden and output layer activities, i.e., \(\mathbf{r}^{(k)}\in\mathcal{P}^{(k)}\), where \(\mathcal{P}^{(k)}\) is the presumed polytopic domain for the \(k\)-th layer (Bozkurt et al., 2023; Tatih and Erdogan, 2021). We note that the polytopic assumptions are plausible as the activations of neurons in practice are bounded. In particular, we will make the specific assumption that \(\mathcal{P}^{(k)}=\mathcal{B}_{\infty,+}=\{\mathbf{r}:\mathbf{0}\leq\mathbf{r }\leq\mathbf{1}\}\), i.e., (normalized) activations lie in a nonnegative unit-hypercube. Such nonnegativity constraints have been connected to disentangling behavior (Plumbley, 2003; Pehlevan et al., 2017; Whittington et al., 2023), however, we consider extensions in the form of alternative polytopic sets corresponding to different feature priors Bozkurt et al. (2023). More broadly, the corresponding label \(\mathbf{y}_{T}\) can be, one-hot encoded label vectors for a classification problem, or discrete or continuous valued vectors for a regression problem.

### Correlative information maximization based signal propagation

#### 2.2.1 Stochastic CorInfoMax based supervised criterion

We propose the total correlative mutual information among consecutive layers, augmented with the mean-square-error (MSE) training loss, as the stochastic objective:

\[J(\mathbf{r}^{(1)},\ldots,\mathbf{r}^{(P)})=\sum_{k=0}^{P-1}I^{(\varepsilon_ {k})}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})-\frac{\beta}{2}E(\|\mathbf{y}_{T} -\mathbf{r}^{(P)}\|_{2}^{2}), \tag{1}\]
where, as defined in [Erdogan, 2022, Ozsoy et al., 2022] and in Appendix A,

\[I^{\overset{\rightarrow}{(\epsilon_{k})}}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})= \frac{1}{2}\log\det\left(\mathbf{R}_{\mathbf{r}^{(k+1)}}+\epsilon_{k}\mathbf{I} \right)-\frac{1}{2}\log\det\left(\mathbf{R}_{\overset{\leftarrow}{\mathbf{e}}_{*} ^{(k+1)}}+\epsilon_{k}\mathbf{I}\right), \tag{2}\]

and \(\mathbf{R}_{\mathbf{r}^{(k+1)}}=E(\mathbf{r}^{(k+1)}\mathbf{r}^{(k+1)}{}^{T})\), \(\mathbf{R}_{\mathbf{r}^{(k)}\mathbf{r}^{(l)}}=E(\mathbf{r}^{(k)}\mathbf{r}^{(l )}{}^{T})\) are the autocorrelation and the cross-correlation matrices corresponding to the layer activations, respectively. Furthermore, \(\mathbf{R}_{\overset{\rightarrow}{\mathbf{e}}_{*}^{(k+1)}}=\mathbf{R}_{\mathbf{ r}^{(k+1)}}-\mathbf{R}_{\mathbf{r}^{(k+1)}\mathbf{r}^{(k)}}{}^{T}( \mathbf{R}_{\mathbf{r}^{(k)}}+\epsilon_{k}\mathbf{I})^{-1}\mathbf{R}_{\mathbf{r}^ {(k)}\mathbf{r}^{(k+1)}}\) corresponds to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of \(\mathbf{r}^{(k+1)}\) from \(\mathbf{r}^{(k)}\). We refer to this problem as the _regularized forward prediction problem_ represented by the optimization

\[\underset{\mathbf{W}_{ff}^{(k)}}{\text{minimize}}\ E(\|\overset{\rightarrow}{ \mathbf{e}}^{(k+1)}\|_{2}^{2})+\epsilon_{k}\|\mathbf{W}_{ff}^{(k)}\|_{F}^{2}\ \ \ \ \text{s.t.}\ \ \ \overset{\rightarrow}{\mathbf{e}}^{(k+1)}=\mathbf{r}^{(k+1)}-\mathbf{W}_{ff}^{(k )}\mathbf{r}^{(k)}, \tag{3}\]

and \(\mathbf{e}_{*}^{(k+1)}\) is the forward prediction error corresponding to the optimal forward predictor \(\mathbf{W}_{ff,*}^{(k)}\).

An equal and alternative expression for the CMI can be written as (Appendix A)

\[I^{\overset{\leftarrow}{(\epsilon_{k})}}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)} )=\frac{1}{2}\log\det(\mathbf{R}_{\mathbf{r}^{(k)}}+\epsilon_{k}\mathbf{I})-\frac {1}{2}\log\det\left(\mathbf{R}_{\overset{\leftarrow}{\mathbf{e}}_{*}^{(k)}}+ \epsilon_{k}\mathbf{I}\right), \tag{4}\]

where \(\mathbf{R}_{\overset{\leftarrow}{\mathbf{e}}_{*}^{(k)}}=\mathbf{R}_{\mathbf{r}^ {(k)}}-\mathbf{R}_{\mathbf{r}^{(k+1)}\mathbf{r}^{(k)}}{}^{T}(\mathbf{R}_{ \mathbf{r}^{(k+1)}}+\epsilon_{k}\mathbf{I})^{-1}\mathbf{R}_{\mathbf{r}^{(k+1)} \mathbf{r}^{(k)}}\) corresponds to the error auto-correlation matrix for the best linear regularized minimum MSE predictor of \(\mathbf{r}^{(k)}\) from \(\mathbf{r}^{(k+1)}\). The corresponding _regularized backward prediction problem_ is defined by the optimization

\[\underset{\mathbf{W}_{fb}^{(k)}}{\text{minimize}}\ E(\|\overset{\leftarrow}{ \mathbf{e}}^{(k)}\|_{2}^{2})+\epsilon_{k}\|\mathbf{W}_{fb}^{(k)}\|_{F}^{2}\ \ \ \ \text{s.t.}\ \ \ \overset{\leftarrow}{\mathbf{e}}^{(k)}=\mathbf{r}^{(k)}-\mathbf{W}_{fb}^{(k)} \mathbf{r}^{(k+1)}. \tag{5}\]

We observe that the two alternative yet equivalent representations of the correlative mutual information between layers \(\mathbf{r}^{(k)}\) and \(\mathbf{r}^{(k+1)}\) in (2) and (4) are intrinsically linked to the forward and backward prediction problems between these layers, which are represented by the optimizations in (3) and (5), respectively. As we will demonstrate later, the existence of these two alternative forms for the CMI plays a crucial role in deriving a neural network architecture that overcomes the weight symmetry issue.

#### 2.2.2 Sample-based supervised CorInfoMax criterion

Our aim is to construct a biologically plausible neural network that optimizes the total CMI, equation 1, in an adaptive manner. Here, we obtain a sample-based version of (1) as a step towards that goal.

We first define the weighted sample auto and cross-correlation matrices as follows:

\[\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t]=\frac{1-\lambda_{\mathbf{r}}}{1- \lambda_{\mathbf{r}}^{t}}\sum_{i=1}^{t}\lambda_{\mathbf{r}}^{t-i}\mathbf{r}^{ (k)}[i]\mathbf{r}^{(k)}[i]^{T},\hat{\mathbf{R}}_{\mathbf{r}^{(k)}\mathbf{r}^ {(k+1)}}[t]=\frac{1-\lambda_{\mathbf{r}}}{1-\lambda_{\mathbf{r}}^{t}}\sum_{i=1 }^{t}\lambda_{\mathbf{r}}^{t-i}\mathbf{r}^{(k)}[i]\mathbf{r}^{(k+1)}[i]^{T}, \tag{6}\]

for \(k=0,\ldots,P\), respectively, where \(0\ll\lambda_{\mathbf{r}}<1\) is the forgetting factor. Next, we define two equivalent forms of the sample-based CMI, \(\hat{I}^{(\epsilon)}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})[t]\):

\[\hat{I}^{\overset{\rightarrow}{(\epsilon_{k})}}(\mathbf{r}^{(k)},\mathbf{r} ^{(k+1)})[t] =\frac{1}{2}\log\det(\hat{\mathbf{R}}_{\mathbf{r}^{(k+1)}}[t]+ \epsilon_{k}\mathbf{I})-\frac{1}{2}\log\det(\hat{\mathbf{R}}_{\overset{\leftarrow}{ \mathbf{e}}^{(k+1)}}[t]+\epsilon_{k}\mathbf{I}), \tag{7}\] \[\hat{I}^{\overset{\leftarrow}{(\epsilon_{k})}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)})[t] =\frac{1}{2}\log\det(\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t]+ \epsilon_{k}\mathbf{I})-\frac{1}{2}\log\det(\hat{\mathbf{R}}_{\overset{\leftarrow}{ \mathbf{e}}^{(k)}}[t]+\epsilon_{k}\mathbf{I}), \tag{8}\]

where \(\hat{\mathbf{R}}_{\overset{\leftarrow}{\mathbf{e}}^{(k+1)}}[t]=\hat{\mathbf{R}}_{ \mathbf{r}^{(k+1)}}[t]-\hat{\mathbf{R}}_{\mathbf{r}^{(k)}\mathbf{r}^{(k+1)}}[ t]^{T}(\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t]+\epsilon_{k}\mathbf{I})^{-1}\hat{ \mathbf{R}}_{\mathbf{r}^{(k)}\mathbf{r}^{(k+1)}}[t]\) is the autocorrelation matrix for the forward prediction error at level-\((k+1)\), \(\overset{\rightarrow}{\mathbf{e}}^{(k+1)}[t]\), corresponding to the best linear weighted regularized least squares predictor of \(\mathbf{r}^{(k+1)}[t]\) from the lower level activations \(\mathbf{r}^{(k)}[t]\). Similarly, \(\hat{\mathbf{R}}_{\overset{\leftarrow}{\mathbf{e}}^{(k)}}[t]=\hat{\mathbf{R}}_{ \mathbf{r}^{(k)}}[t]-\hat{\mathbf{R}}_{\mathbf{r}^{(k)}\mathbf{r}^{(k+1)}}[t]( \hat{\mathbf{R}}_{\mathbf{r}^{(k+1)}}[t]+\epsilon_{k}\mathbf{I})^{-1}\hat{\mathbf{R }}_{\mathbf{r}^{(k)}\mathbf{r}^{(k+1)}}[t]^{T}\) is the autocorrelation matrix for the backward prediction error at level-\((k)\), \(\overset{\leftarrow}{\mathbf{e}}^{(k)}[t]\), corresponding to the best linear weighted regularized least squares predictor of \(\mathbf{r}^{(k)}[t]\) from the higher level activations \(\mathbf{r}^{(k+1)}[t]\).
The sample-based CorInfoMax optimization can be written as:

\[\underset{\mathbf{r}^{(k)}[t],k=0,\dots,P}{\operatorname{maximize}} \sum_{k=0}^{P-1}\hat{I}^{(\epsilon_{k})}(\mathbf{r}^{(k)},\mathbf{r}^ {(k+1)})[t]-\frac{\beta}{2}\|\mathbf{y}_{T}[t]-\mathbf{r}^{(P)}[t]\|_{2}^{2} \tag{9a}\] \[\operatorname{subject\ to} \mathbf{r}^{(k)}[t]\in\mathcal{P}^{(k)},k=1,\dots,P,\] (9b) \[\mathbf{r}^{(0)}[t]=\mathbf{x}[t], \tag{9c}\]

The first-order Taylor series approximation of the \(\log\det\) terms on the right of (7) and (8) are:

\[\log\det\left(\hat{\mathbf{R}}_{\overline{\mathbf{e}}^{(k+1)}}[t]+ \epsilon_{k}\mathbf{I}\right)\approx\frac{1}{\epsilon_{k}}\operatorname{Tr} \left(\hat{\mathbf{R}}_{\overline{\mathbf{e}}^{(k+1)}}[t]\right)+N_{k+1}\log( \epsilon_{k})\] \[=\frac{1}{\epsilon_{k}}\sum_{i=1}^{t}\lambda_{\mathbf{r}}^{t-i} \|\mathbf{r}^{(k+1)}[i]-\mathbf{W}_{ff,*}^{(k)}[t]\mathbf{r}^{(k)}[i]\|_{2}^{ 2}+\epsilon_{k}\|\mathbf{W}_{ff,*}^{(k)}[t]\|_{F}^{2}+N_{k+1}\log(\epsilon_{k}), \tag{10}\] \[\quad\log\det\left(\hat{\mathbf{R}}_{\overline{\mathbf{e}}^{(k)}}[t] +\epsilon_{k}\mathbf{I}\right)\approx\frac{1}{\epsilon_{k}}\operatorname{Tr} \left(\hat{\mathbf{R}}_{\overline{\mathbf{e}}^{(k)}}[t]\right)+N_{k}\log(\epsilon_ {k})\] \[=\frac{1}{\epsilon_{k}}\sum_{i=1}^{t}\lambda_{\mathbf{r}}^{t-i} \|\mathbf{r}^{(k)}[i]-\mathbf{W}_{fb,*}^{(k)}[t]\mathbf{r}^{(k+1)}[i]\|_{2}^{ 2}+\epsilon_{k}\|\mathbf{W}_{fb,*}^{(k)}[t]\|_{F}^{2}+N_{k}\log(\epsilon_{k}). \tag{11}\]

Note that in (10), \(\mathbf{W}_{ff,*}^{(k)}[t]\) denotes the optimal linear regularized weighted least squares forward predictor coefficients in predicting \(\mathbf{r}^{(k+1)}[i]\) from \(\mathbf{r}^{(k)}[i]\) for \(i=1,\dots,t\). Likewise, \(\mathbf{W}_{fb,*}^{(k)}[t]\) in (11) represents the optimal linear regularized weighted least squares backward predictor coefficients in predicting \(\mathbf{r}^{(k)}[i]\) from \(\mathbf{r}^{(k+1)}[i]\) for \(i=1,\dots,t\). Consequently, the optimal choices of forward and backward predictor coefficients are coupled with the optimal choices of layer activations.

In the online optimization process, we initially relax this requirement and start with random predictor coefficient selections. During the learning process, we apply a coordinate ascent-based procedure on activation signals and predictor coefficients. Specifically, at time step-\(t\), we first optimize with respect to the activations \(\{\mathbf{r}^{(k)}[t],k=1,\dots,P\}\), where we assume predictor coefficients to be fixed. Next, we update the forward and backward predictor coefficients \(\mathbf{W}_{ff}^{(k)}\) and \(\mathbf{W}_{fb}^{(k)}\), for \(k=1,\dots,P\), to reduce the corresponding forward and backward prediction errors, respectively. As the algorithm iterations progress, the predictor coefficients converge to the vicinity of their optimal values.

For the first phase of the online optimization, we employ a projected gradient ascent-based approach for activations: for \(k=1,\dots,P-1\), the layer activation vector \(\mathbf{r}^{(k)}[t]\) is included in the objective function terms \(\hat{I}^{(\epsilon)}(\mathbf{r}^{(k-1)},\mathbf{r}^{(k)})[t]\) and \(\hat{I}^{(\epsilon)}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})[t]\). Therefore, to calculate the gradient with respect to \(\mathbf{r}^{(k)}[t]\), we can use the modified form of expressions in (7) and (8), where we use the approximations in (10)-(11), and the optimal predictors are replaced with their current estimates:

\[\nabla_{\mathbf{r}^{(k)}}\hat{J}_{k}(\mathbf{r}^{(k)})[t]=\nabla_ {\mathbf{r}^{(k)}}\hat{I}^{\overset{\rightarrow}{\leftarrow}}_{(\epsilon_{k-1 })}(\mathbf{r}^{(k-1)},\mathbf{r}^{(k)})[t]+\nabla_{\mathbf{r}^{(k)}}\hat{I}^{ \overset{\leftarrow}{\leftarrow}}_{(\epsilon_{k})}(\mathbf{r}^{(k)},\mathbf{ r}^{(k+1)})[t]\] \[=\tfrac{1}{2}\nabla_{\mathbf{r}^{(k)}}(\log\det(\hat{\mathbf{R}} _{\mathbf{r}^{(k)}}[t]+\epsilon_{k-1}\mathbf{I})+\log\det(\hat{\mathbf{R}}_{ \mathbf{r}^{(k)}}[t]+\epsilon_{k}\mathbf{I}))-\tfrac{1}{\epsilon_{k-1}}\overline{ \mathbf{e}}^{(k)}[t]-\tfrac{1}{\epsilon_{k}}\overline{\mathbf{e}}^{(k)}[t], \tag{12}\]

where

\[\overline{\mathbf{e}}^{(k)}[t]=\mathbf{r}^{(k)}[t]-\mathbf{W}_{ff}^{(k-1)}[t] \mathbf{r}^{(k-1)}[t],\quad\overset{\leftarrow}{\mathbf{e}}^{(k)}[t]= \mathbf{r}^{(k)}[t]-\mathbf{W}_{fb}^{(k)}[t]\mathbf{r}^{(k+1)}[t] \tag{13}\]

are forward and backward prediction errors at level-\(k\), respectively. Following the procedure in Bozkurt et al. (2023), for the gradient term in (12), we can write:

\[\frac{1}{2}\nabla_{\mathbf{r}^{(k)}}(\log\det(\hat{\mathbf{R}}_{ \mathbf{r}^{(k)}}[t]+\epsilon_{k-1}\mathbf{I})+\log\det(\hat{\mathbf{R}}_{\mathbf{ r}^{(k)}}[t]+\epsilon_{k}\mathbf{I}))=2\gamma\mathbf{B}_{\mathbf{r}^{(k)}}[t]\mathbf{r}^{(k)}[t], \tag{14}\]

where \(\mathbf{B}_{\mathbf{r}^{(k)}}[t]=(\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t]+\epsilon_ {k-1}\mathbf{I})^{-1}\approx(\hat{\mathbf{R}}_{\mathbf{r}^{(k)}}[t]+\epsilon_{k} \mathbf{I})^{-1}\) and \(\gamma=\frac{1-\lambda_{\mathbf{r}}}{\lambda_{\mathbf{r}}}\). The gradient of the objective for the final layer can be expressed as:

\[\nabla_{\mathbf{r}^{(P)}}(\hat{I}^{\overset{\rightarrow}{\leftarrow}}( \mathbf{r}^{(P-1)},\mathbf{r}^{(P)})[t]-\frac{\beta}{2}\|\mathbf{r}^{(P)}[t]- \mathbf{y}_{T}[t]\|_{2}^{2})\] \[=\gamma\mathbf{B}_{\mathbf{r}^{(P)}}[t]\mathbf{r}^{(P)}[t]-\frac{1}{ \epsilon_{P-1}}\overline{\mathbf{e}}^{(P)}[t]-\beta(\mathbf{r}^{(P)}[t]-\mathbf{y}_ {T}[t]).\]
### Neural network formulation based on information maximization

In this section, we develop a biologically plausible neural network grounded on the correlative information maximization-based network propagation model outlined in Section 2.2. To achieve this, we employ projected gradient ascent optimization for determining layer activations \(\mathbf{r}^{(1)}[t],\mathbf{r}^{(2)}[t],\ldots,\mathbf{r}^{(P)}[t]\), which shape the network structure and dynamics, as well as updating the corresponding synapses that govern the learning dynamics.

#### 2.3.1 Network structure and neural dynamics

In this section, we show that the projected gradient ascent solution to the optimization in (9) defines a multilayer recurrent neural network. To this end, we introduce the intermediate variable \(\mathbf{u}^{(k)}\) as the updated layer-\(k\) activations prior to the projection onto the domain set \(\mathcal{P}^{(k)}\). Utilizing the gradient expressions in (12)-(14), we can express the network dynamics for layers \(k=1,\ldots,P-1\) as follows:

\[\tau_{\mathbf{u}}\frac{d\mathbf{u}^{(k)}[t;s]}{ds} =-g_{lk}\mathbf{u}^{(k)}[t;s]+\frac{1}{\epsilon_{k}}\boldsymbol{ M}^{(k)}[t]\mathbf{r}^{(k)}[t;s]-\frac{1}{\epsilon_{k-1}}\overset{\rightarrow}{ \mathbf{e}}_{u}^{(k)}[t;s]-\frac{1}{\epsilon_{k}}\overset{\leftarrow}{ \mathbf{e}}_{u}^{(k)}[t;s], \tag{15}\] \[\overset{\rightarrow}{\mathbf{e}}_{u}^{(k)}[t;s] =\mathbf{u}^{(k)}[t;s]-\boldsymbol{W}_{ff}^{(k-1)}[t]\mathbf{r}^ {(k-1)}[t;s],\quad\overset{\leftarrow}{\mathbf{e}}_{u}^{(k)}[t;s]=\mathbf{u} ^{(k)}[t;s]-\boldsymbol{W}_{fb}^{(k)}[t]\mathbf{r}^{(k+1)}[t;s],\] (16) \[\mathbf{r}^{(k)}[t;s] =\sigma_{+}(\mathbf{u}^{(k)}[t;s]), \tag{17}\]

where \(\tau_{\mathbf{u}}\) is the update time constant, \(\boldsymbol{M}^{(k)}[t]=2\epsilon_{k}(\gamma\boldsymbol{B}^{(k)}[t]+g_{lk} \boldsymbol{I})\), and \(\sigma_{+}\) represents the elementwise clipped-ReLU function corresponding to the projection onto the nonnegative unit-hypercube \(\mathcal{B}_{\infty,+}\), defined as \(\sigma_{+}(u)=\min(1,\max(u,0))\).

To reinterpret the dynamics in (15) to (17) as a multi-compartmental neural network, for \(k=1,\ldots,P-1\), we define the signals:

\[\mathbf{v}_{A}^{(k)}[t;s]=\boldsymbol{M}^{(k)}[t]\boldsymbol{r}^{(k)}[t;s]+ \boldsymbol{W}_{fb}^{(k)}[t]\mathbf{r}^{(k+1)}[t;s],\quad\mathbf{v}_{B}^{(k)} [t;s]=\boldsymbol{W}_{ff}^{(k-1)}[t]\mathbf{r}^{(k-1)}[t;s], \tag{18}\]

which allow us to rewrite the network activation dynamics (15) to (17) as:

\[\tau_{\mathbf{u}}\frac{d\mathbf{u}^{(k)}[t;s]}{ds}=-g_{lk} \mathbf{u}^{(k)}[t;s]+g_{A,k}(\mathbf{v}_{A}^{(k)}[t;s]-\mathbf{u}^{(k)}[t;s] )+g_{B,k}(\mathbf{v}_{B}^{(k)}[t;s]-\mathbf{u}^{(k)}[t;s]), \tag{19}\] \[\mathbf{r}^{(k)}[t;s]=\sigma_{+}(\mathbf{u}^{(k)}[t;s]), \tag{20}\]

where \(g_{A,k}=\frac{1}{\epsilon_{k-1}}\) and \(g_{B,k}=\frac{1}{\epsilon_{k}}\). Similarly, for the output layer, we employ the same expressions as (19) and (20) with \(k=P\), except that in this case we have:

\[\mathbf{v}_{A}^{(P)}[t;s]=\boldsymbol{M}^{(P)}[t]\mathbf{r}^{(k)}[t;s]-( \mathbf{r}^{(P)}[t;s]-\boldsymbol{y}_{T}[t]),\quad\mathbf{v}_{B}^{(P)}[t;s]= \boldsymbol{W}_{ff}^{(P-1)}[t]\mathbf{r}^{(P-1)}[t;s], \tag{21}\]

where \(g_{B,P}=\frac{1}{\epsilon_{P-1}}\), \(g_{A,P}=\beta\) and \(\boldsymbol{M}^{(P)}[t]=\beta^{-1}(\gamma\boldsymbol{B}^{(P)}[t]+g_{lk} \boldsymbol{I})\).

Remarkably, the equations (18) to (21) reveal a biologically plausible neural network that incorporates three-compartmental pyramid neuron models, as presented in (Sacramento et al., 2018; Golkar et al., 2022). This intricate architecture, of which two-layer segment is demonstrated in Figure 1, naturally emerges from the proposed correlative information maximization framework. In this network structure:

* \(\mathbf{u}^{(k)}\) embodies the membrane potentials for neuronal somatic compartments of the neurons at layer-\(k\), where \(\tau_{\mathbf{u}}\) is the membrane leak time constant of soma.
* \(\mathbf{v}_{B}^{(k)}\) corresponds to membrane potentials for basal dendrite compartments, receiving feedforward input originating from the previous layer.
* \(\mathbf{v}_{A}^{(k)}\) denotes the membrane potentials for distal apical dendrite compartments, which gather top-down input from the subsequent layer and lateral inputs represented by \(\boldsymbol{M}^{(k)}[t]\mathbf{r}^{(k)}\) in (18) and (21). Decomposing \(\boldsymbol{M}^{(k)}\) into \(\boldsymbol{D}^{(k)}-\boldsymbol{O}^{(k)}\), we find that \(\boldsymbol{D}^{(k)}\) mirrors autapses (Lubke et al., 1996), and the off-diagonal component \(\boldsymbol{O}^{(k)}\) corresponds to lateral inhibition synapses. We use \(\mathbf{i}^{(k)}=-\boldsymbol{O}^{(k)}\mathbf{r}^{(k)}\) to represent the activations of SST interneurons (Urban-Ciecko and Barth, 2016) that generate lateral inhibitions to the apical dendrites.
* Forward (backward) prediction errors manifest in the membrane voltage differences between soma and basal (distal) compartments of the pyramidal neurons.
* Forward (backward) prediction coefficients \(\mathbf{W}_{ff}^{(k)}\) (\(\mathbf{W}_{fb}^{(k)}\)) are associated with feedforward (top-down) synapses connecting layers \((k)\) and \((k+1)\).
* The inverse of the regularization coefficient \(\epsilon_{k}\) is related to the conductance between soma and dendritic compartments. In contrast, at the output layer, the augmentation constant \(\beta\) corresponds to the conductance between soma and distal compartments. This relationship can be motivated by modifying the objective in (9a) as \[\sum_{k=0}^{P-1}\hat{I}^{(\epsilon_{k})}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})[ t]+\frac{1}{2}\hat{I}^{\overset{\leftarrow}{(\beta^{-1})}}(\mathbf{r}^{(P)}, \mathbf{y}_{T})[t],\] (22) where, through the first-order approximation, the \(\mathbf{r}^{(P)}[t]\) dependent portion of \(\hat{I}^{(\beta^{-1})}(\mathbf{r}^{(P)},\mathbf{y}_{T})[t]\) can be expressed as \(-\beta\|\mathbf{r}^{(P)}[t]-\mathbf{W}_{fb}^{(P)}\mathbf{y}_{T}[t]\|_{2}^{2}\). For accuracy, we enforce \(\mathbf{W}_{fb}^{(P)}=\mathbf{I}\).

### Learning dynamics

Network parameters consists of feedforward \(\mathbf{W}_{ff}^{(k)}\), feedback \(\mathbf{W}_{fb}^{(k)}\) and lateral \(\mathbf{B}^{(k)}\) coefficients.The learning dynamics of these coefficients are elaborated below:

* _Feedforward Coefficients_ are connected to the forward prediction problem defined by the optimization in (3). We can define the corresponding online optimization objective function as \[C_{ff}(\mathbf{W}_{ff}^{(k)})=\epsilon_{k}\|\mathbf{W}_{ff}^{(k)}\|_{F}^{2}+\| \overset{\leftarrow}{\mathbf{e}}^{(k+1)}[t]\|_{2}^{2}\text{ for which the the partial derivative is given by }\] \[\frac{\partial C_{ff}(\mathbf{W}_{ff}^{(k)}[t])}{\partial\mathbf{W}_{ff}^{(k)}}=2 \epsilon_{k}\mathbf{W}_{ff}^{(k)}[t]-2\overset{\rightarrow}{\mathbf{e}}^{(k+1)}[ t]\mathbf{r}^{(k)}[t]^{T}.\] (23) In Appendix C, we provide a discussion on rewriting (23) in terms of the membrane voltage difference between the distal apical and soma compartments of the neuron, based on the equilibrium condition for the neuronal dynamics: \[-\overset{\rightarrow}{\mathbf{e}}^{(k+1)}[t]\mathbf{r}^{(k)}[t]^{T}=g_{B,k} ^{-1}(g_{A,k}\mathbf{v}_{A}^{(k)}[t]-(g_{lk}+g_{A_{k}})\mathbf{u}_{*}^{(k)}[t] +\mathbf{h}_{*}[t])\mathbf{r}^{(k)}[t]^{T},\] (24) where \(\mathbf{h}_{*}[t]\) is nonzero only for neurons that are silent or firing at the maximum rate.
* Similarly, _Feedback Coefficients_ are connected to the backward prediction problem defined by the optimization in (5), and the corresponding online optimization objective function as \(C_{fb}(\mathbf{W}_{fb}^{(k)})=\epsilon_{k}\|\mathbf{W}_{ff}^{(k)}\|_{F}^{2}+\|\overset{ \leftarrow}{\mathbf{e}}^{(k)}[t]\|_{2}^{2}\) for which the partial derivative is given by \[\frac{\partial C_{fb}(\mathbf{W}_{fb}^{(k)}[t])}{\partial\mathbf{W}_{fb}^{(k)}}=2 \epsilon_{k}\mathbf{W}_{fb}^{(k)}[t]-2\overset{\leftarrow}{\mathbf{e}}^{(k)}[t] \mathbf{r}^{(k+1)}[t]^{T}.\] (25) To compute the updates of both feedforward and feedback coefficients, we use the EP approach [Scellier and Bengio, 2017b], where the update terms are obtained based on the contrastive expressions of partial derivatives in (23) and (25) for the nudge phase, i.e., \(\beta=\beta^{\prime}>0\), and the free phase, i.e., \(\beta=0\), : \[\delta\mathbf{W}_{ff}^{(k)}[t]\propto\frac{1}{\beta^{\prime}}\left(( \overset{\leftarrow}{\mathbf{e}}^{(k+1)}[t]\mathbf{r}^{(k)}[t]^{T})\bigg{|}_ {\beta=\beta^{\prime}}-(\overset{\leftarrow}{\mathbf{e}}^{(k+1)}[t]\mathbf{ r}^{(k)}[t]^{T})\bigg{|}_{\beta=0}\right),\] (26) \[\delta\mathbf{W}_{fb}^{(k)}[t]\propto\frac{1}{\beta^{\prime}}\left(( \overset{\leftarrow}{\mathbf{e}}^{(k)}[t]\mathbf{r}^{(k+1)}[t]^{T})\bigg{|}_ {\beta=\beta^{\prime}}-(\overset{\leftarrow}{\mathbf{e}}^{(k+1)}[t]\mathbf{ r}^{(k+1)}[t]^{T})\bigg{|}_{\beta=0}\right).\] (27)
* _Lateral Coefficients_, \(\mathbf{B}^{(k)}\) are the inverses of the \(\epsilon\mathbf{I}\) perturbed correlation matrices. We can use the update rule in [Bozkurt et al., 2023] for their learning dynamics after the nudge phase: \[\mathbf{B}^{(k)}[t+1]=\lambda_{\mathbf{r}}^{-1}(\mathbf{B}^{(k)}[t]-\gamma\mathbf{z} ^{(k)}[t]\mathbf{z}^{(k)}[t]^{T}),\text{ where }\mathbf{z}^{(k)}=\mathbf{B}^{(k)}[t]\mathbf{r}^{(k)}[t].\]
## 3 Discussion of results

* In (12), we devise an update for layer activation \(\mathbf{r}^{(k)}\) by employing two distinct forms of the CMI associated with \(\mathbf{r}^{(k)}\): \(\widehat{I}^{(\epsilon_{k-1})}(\mathbf{r}^{(k-1)},\mathbf{r}^{(k)})[t]\), the CMI with the preceding layer, encompassing the forward prediction error for estimating \(\mathbf{r}^{(k)}\), and \(\widehat{I}^{(\epsilon_{k})}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})[t]\), the CMI with the subsequent layer, incorporating the backward prediction error for estimating \(\mathbf{r}^{(k)}\). Employing these alternative expressions is crucial in circumventing the weight transport problem and offering a more biologically plausible framework. For further discussion, please refer to Appendix B.
* In the context of the proposed correlative information maximization framework, predictive coding naturally emerges as a crucial mechanism. By incorporating both alternative expressions of CMI, the framework focuses on minimizing both forward and backward prediction errors between adjacent layers via feedforward and feedback connections. These connections foster bidirectional information flow, thereby enhancing the overall learning process.
* Figure 1 depicts the interplay between the CorInfoMax objective and the corresponding network architecture. The emergence of lateral connections and autapses can be attributed to the maximization of the unconditional layer entropy component of the CMI, which allows for efficient utilization of the available representation dimensions. Simultaneously, the minimization of conditional entropies between adjacent layers gives rise to feedforward and feedback connections, effectively reducing redundancy within representations.
* We employ time-contrastive learning, as in GenRec (O'Reilly, 1996), EP (Scellier and Bengio, 2017b) and CSM (Qin et al., 2021), by implementing separate phases with Hebbian and anti-Hebbian updates, governed by an assumed teaching signal. It has been conjectured that the teaching signal in biological networks can be modeled by the oscillations in the brain (Whittington and Bogacz, 2019; Baldi and Pineda, 1991; Ketz et al., 2013). Although the oscillatory rhythms and their synchronization in the brain are elusive, they are believed to play a crucial role in adaptive processes such as learning and predicting upcoming events (Fell and Axmacher, 2011; Engel et al., 2001).

## 4 Numerical experiments

In this section, we evaluate the performance of our CorInfoMax framework on image classification tasks using three popular datasets: MNIST (LeCun and Cortes, 2010), Fashion-MNIST (Xiao et al., 2017), and CIFAR10 (Krizhevsky et al., 2009). We compare the effectiveness of our approach against other contrastive methods, such as EP (Scellier and Bengio, 2017b) and CSM (Qin et al., 2021), as well as explicit methods, including PC (Whittington and Bogacz, 2017) and PC-Nudge (Millipore et al., 2023), when training multilayer perceptron (MLP) architectures.

We examine two distinct constraints on the activations of CorInfoMax Networks: (i) \(\mathcal{B}_{\infty,+}\), representing the nonnegative part of the unit hypercube, and (ii) \(\mathcal{B}_{1,+}=\{\mathbf{r}:\mathbf{r}\geq 0,\|\mathbf{r}\|_{1}\leq 1\}\), denoting the nonnegative part of the unit \(\ell_{1}\)-norm ball (Tatli and Erdogan, 2021). Table 1 presents the test accuracy results for each algorithm, averaged over 10 realizations along with the corresponding standard deviations. These findings demonstrate that CorInfoMax networks can achieve comparable or superior

\begin{table}
\begin{tabular}{l l l l} \hline \hline  & MNIST & FashionMNIST & CIFAR10 \\ \hline
**CorInfoMax-\(\mathcal{B}_{\infty,+}\)** (Appendix E.3) & \(97.62\pm 0.1\) & \(88.14\pm 0.3\) & \(51.86\pm 0.3\) \\
**CorInfoMax-\(\mathcal{B}_{1,+}\)** (Appendix E.5) & \(97.71\pm 0.1\) & \(88.09\pm 0.1\) & \(51.19\pm 0.4\) \\ EP & \(97.61\pm 0.1\) & \(88.06\pm 0.7\) & \(49.28\pm 0.5\) \\ CSM & \(98.08\pm 0.1\) & \(88.73\pm 0.2\) & \(40.79^{*}\) \\ PC & \(98.17\pm 0.2\) & \(89.31\pm 0.4\) & - \\ PC-Nudge & \(97.71\pm 0.1\) & \(88.49\pm 0.3\) & \(48.58\pm 0.7\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Test accuracy results (mean \(\pm\) standard deviation from \(n=10\) runs) for CorInfoMax networks are compared with other biologically-plausible algorithms. The performance of CSM on the CIFAR10 dataset is taken from (Qin et al., 2021), while the remaining results stem from our own simulations.
performance in relation to the state-of-the-art methods for the selected tasks. Additional information regarding these experiments, as well as further experiments, can be found in the Appendix. We also provide the code used for these experiments in the supplementary document.

## 5 Conclusion

In this article, we have presented the correlative information maximization (CorInfoMax) framework as a biologically plausible approach to constructing supervised neural network models. Our proposed method addresses the long-standing weight symmetry issue by providing a principled solution, which results in asymmetric forward and backward prediction networks. Furthermore, the CorInfoMax framework offers a normative approach for developing network models that incorporate multi-compartment pyramidal neuron models, aligning more closely with the experimental findings about the biological neural networks.

One potential limitation of our framework, shared by other supervised approaches, is the necessity for model parameter search to improve accuracy. We discuss this issue in detail in Appendix F.
