The Cascaded Forward Algorithm for Neural Network Training
Backpropagation algorithm has been widely used as a mainstream learning
procedure for neural networks in the past decade, and has played a significant
role in the development of deep learning. However, there exist some limitations
associated with this algorithm, such as getting stuck in local minima and
experiencing vanishing/exploding gradients, which have led to questions about
its biological plausibility. To address these limitations, alternative
algorithms to backpropagation have been preliminarily explored, with the
Forward-Forward (FF) algorithm being one of the most well-known. In this paper
we propose a new learning framework for neural networks, namely Cascaded
Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF.
Unlike FF, our framework directly outputs label distributions at each cascaded
block, which does not require generation of additional negative samples and
thus leads to a more efficient process at both training and testing. Moreover,
in our framework each block can be trained independently, so it can be easily
deployed into parallel acceleration systems. The proposed method is evaluated
on four public image classification benchmarks, and the experimental results
illustrate significant improvement in prediction accuracy in comparison with
the baseline.
Gongpei Zhao, Tao Wang, Yidong Li, Yi Jin, Congyan Lang, Haibin Ling
# The Cascaded Forward Algorithm for Neural Network Training

###### Abstract

Backpropagation algorithm has been widely used as a mainstream learning procedure for neural networks in the past decade, and has played a significant role in the development of deep learning. However, there exist some limitations associated with this algorithm, such as getting stuck in local minima and experiencing vanishing/exploding gradients, which have led to questions about its biological plausibility. To address these limitations, alternative algorithms to backpropagation have been preliminarily explored, with the Forward-Forward (FF) algorithm being one of the most well-known. In this paper we propose a new learning framework for neural networks, namely **C**ascaded **F**orward (**CaFo**) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples and thus leads to a more efficient process at both training and testing. Moreover, in our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems. The proposed method is evaluated on four public image classification benchmarks, and the experimental results illustrate significant improvement in prediction accuracy in comparison with the baseline. Our code is available at: [https://github.com/Graph-ZKY/CaFo](https://github.com/Graph-ZKY/CaFo).

## 1 Introduction

Backpropagation (BP) algorithm [1] is a powerful technique that has proven to be effective for training deep neural networks on a wide range of tasks, including image classification [2], semantic segmentation [3] and object detection [4]. The algorithm's ability of adjusting the weights based on the error between the prediction and the ground truth allows the network to learn and improve over time, making it a cornerstone of deep learning and artificial intelligence. Nevertheless, despite its effectiveness, BP also suffers from several limitations in practical applications. These limitations include the problems of local minima [5], vanishing/exploding gradients [6], overfitting [7], slower convergence and non-convex optimization [8], which may negatively impact the training process. Additionally, BP relies on a complete understanding of the computations performed during the forward pass in order to correctly calculate the derivatives, making it difficult to be generalized to the black-box systems where the internal workings are not transparent. Therefore it is important to be aware of these limitations and to consider alternative algorithms for training deep neural networks.

Due to the aforementioned limitations and the apparent difference in mechanisms between deep neural network and real cortical neurons, some researchers [9] have raised concern about the biological plausibility of backpropagation, questioning whether the brain implements BP and whether it has some other way of getting the gradients needed to adjust the weights on connections. This has prompted researchers to search for alternate algorithms to train deep neural networks. One of such alternatives is the Forward-Forward (FF) algorithm, which replaces the forward and backward passes of BP with two forward passes that work on positive and negative data respectively, with opposite optimization objectives. FF has shown to be a potential alternative to BP, for its simplicity and flexibility in network architecture design. It makes some preliminary investigations on this area, and there is still a lot of room for further research and in-depth exploration.

In this paper we present a flexible and effective learning framework for neural networks, namely **C**ascaded **F**orward (**CaFo**) algorithm, which offers a promising alternative to the traditional BP algorithm. Our CaFo framework consists
of multiple cascaded neural blocks, with a layer-wise predictor attached to each block. Each neural block is composed of one or more convolutional layers, pooling layers, normalization layers, activation layers, etc, of which the aim is to project the input feature map into a new feature space. Note that the parameters of all neural blocks are randomly initialized and are fixed across both training and testing, while only the attached layer-wise predictors are trainable. Each layer-wise predictor takes the feature map computed by the corresponding neural block as input and outputs the prediction of the task, and it is trained independently without backpropagation according to the errors between the output and the ground truth. During the test period, the final prediction is a combination of the predictions from all predictors.

Roughly speaking, instead of performing backpropagation using the chain rule, the CaFo network just performs forward pass to directly estimate the prediction errors and updates the parameters in-situ. Overall, our algorithm is a step forward on the basis of FF algorithm and takes several advantages. Firstly, it eliminates the necessary for negative sampling, thereby reducing the impact of sampling quality on model performance and increasing stability. Secondly, our method can directly output the prediction of multi-class distribution rather than a simple goodness estimation, and it is thus more suitable for multi-class prediction tasks. Finally, although the neural blocks are cascaded, the predictor attached to each block is trained independently without the need for the training results from prior blocks, so the CaFo algorithm can be easily deployed into parallel acceleration systems.

For evaluation we test our algorithm on four public image classification benchmarks, in comparison with FF, the state-of-the-art non-backpropagation algorithm. The experimental results show that our CaFo algorithm exhibits effectiveness and efficiency on image classification task, and outperforms FF by a significant margin.

## 2 Related Work

### Backpropagation Algorithm

Backpropagation algorithm is a widely used training algorithm for deep neural networks, the goal of which is to optimize the parameters of the networks by minimizing the prediction error between the network's output and the ground truth. BP uses the gradient descent algorithm for optimization. The gradient of the loss function is calculated using the chain rule [10] of differentiation, where the error is propagated backwards through the network and the parameters are updated in the direction of the negative gradient.

BP is favored for its ease of implementation, computational efficiency, and effectiveness on a variety of problems. However, it also suffers from certain limitations that have been deeply studied [11; 12; 13; 14; 15; 16; 17; 18; 19]. Despite these efforts, problems with BP still occur stochastically, highlighting the need for a deep understanding of deep learning and experience in model tuning. Due to the limitations mentioned above, some recent researches [9; 20; 21] have even raised doubts about the biological plausibility of BP.

### Forward-Forward Algorithm

The recent doubt raised by Hinton [9] about the biological plausibility of the BP has led to the introduction of the Forward-Forward algorithm as an alternative. FF replaces the forward and backward passes of BP with two forward passes that work on positive and negative data with opposite optimization objectives. FF has been validated to be an effective alternative to BP in some tasks, due to its simplicity and flexibility for network architecture and its ability to handle non-differentiable components. Nevertheless, some potential limitations of FF make it hard to be integrated into existing learning systems, including the high computational demanding of two forward passes, the high dependence on positive and negative sampling strategies, and the rough approximation of optimization objective. More recently, an ingenious learning process, the predictive forward-forward [22] (PFF) process proposed as a generalization of the FF, generalizes and combines the forward-forward algorithm and predictive coding into a robust neural system. It simultaneously learns a representation and generative model in a biologically-plausible manner, providing a promising brain-inspired form of forward-only learning.

Our CaFo network, different from the FF algorithm, requires only one forward pass for each training step and eliminates the need for negative sampling, and updates the parameters directly according to the errors between the predicted output and the ground truth. It simplifies both the training and the testing process, and gains significant improvement in prediction accuracy.
## 3 The Proposed Method

In this section, we describe the details of the proposed CaFo framework. Following the work of FF, we apply this method to the task of image classification, a well-established challenge in computer vision. To ensure a clear and consistent presentation throughout the paper, we firstly provide an overview of the notations and mathematical symbols used in our analysis. This is followed by an elaboration of the overall architecture of the CaFo framework. Finally we describe the training and inference processes of the proposed approach.

### Notations

To ensure clarity and consistency in our presentation, we provide a brief overview of the notations we used in this paper. The scalars used in our analysis are represented by italic lowercase letters (e.g., \(m\), \(n\)), while the vectors used in our method are represented by bold lowercase letters (e.g., \(\mathbf{x}\), \(\mathbf{y}\)). All vectors in this paper are assumed to be column vectors. Bold uppercase letters (e.g., \(\mathbf{W}\), \(\mathbf{Z}\)) are used to represent matrices. Furthermore, the subscripts and superscripts in our notations represent the indices and exponents, respectively. In the rest of the paper, we will provide more detailed definitions and explanations of the notations as needed.

### Pipeline Overview

The pipeline of the proposed CaFo framework is depicted in Figure 1, which consists of multiple repeated stacks of two major key components:

**Neural Block:** The neural block, abbreviated as _block_ for convenience in the rest of the paper, is comprised of multiple fundamental units, such as convolutional layer, pooling layer, normalization layer, activation layer, etc. Its architecture is dependent on the specific task, and for image classification in our experiments it consists of a convolutional layer followed by a ReLU activation function, a max-pooling layer and a batch normalization layer. Multiple blocks with the same or different architecture are cascaded to extract intermediate representations of images at different scales. It is worth noting that the block can be either parametric or non-parametric, which provides flexibility in its construction.

**Predictor:** As illustrated in Figure 1, each block is equipped with a layer-wise predictor that consists of a fully connected layer followed by an activation function (e.g., softmax) if necessary. The predictor takes the intermediate

Figure 1: Overall architecture of the CaFo network. Two main components are cascaded in the network: (1) a neural block that extracts representation from source data (e.g., images) at a specific scale, and (2) a predictor that takes the intermediate representation extracted by the block as input and outputs the prediction results. The parameters of predictors can be optimized either in parallel or in sequence by minimizing the loss between the predicted output and the ground truth.
representation extracted by the block as input and outputs a prediction result for the task. Since there is only one fully connected layer, the predictor can be trained _in situ_ without backpropagation.

### Training Process

In the CaFo network, we randomly initialize the parameters of each block with Kaiming uniform [23] and keep these parameters fixed across both the training and testing process. The blocks act as filters with fixed template coefficients. The experimental results in Tables 1 and 2 demonstrate that, despite their simplicity, these untrainable blocks can still produce discriminative representations for prediction and achieve better performance in comparison with FF.

Only the parameters of each predictor are updated during the training phase. It is worth noting that our method enables both parallel and sequential updates of predictors, as the intermediate features extracted by the blocks with fixed parameters, can be obtained by only one forward pass. During the training phase, each predictor independently makes its own greedy decision about the class to which the input belongs, eliminating the need for backpropagation to learn a discriminative model.

For each predictor, we optimize it by minimizing the errors between the prediction and the ground truth. Different measure functions can be adopted in our framework, which lead to different optimization strategies and algorithms. In experiments we compare three different loss function: mean square error loss (MSE), cross-entropy loss (CE) and sparsemax loss (SL) [24]. Here we describe the corresponding formulations and optimization algorithms respectively.

MSE loss:The optimization objective of the predictor can be expressed as follows:

\[\arg\min_{\textbf{W}}\frac{1}{2m}\|\textbf{H}\textbf{W}-\textbf{Y}\|_{F}^{2}, \tag{1}\]

where \(m\) denotes the number of training samples, \(\textbf{H}\in\mathbb{R}^{m\times d}\) the \(d-\)dimensional intermediate representation output by the block, \(\textbf{Y}\in\mathbb{R}^{m\times c}\) the one-hot labels of training samples, \(\textbf{W}\in\mathbb{R}^{d\times c}\) the parameters of predictor to be optimized, \(\|\cdot\|_{F}\) the Frobenius norm, and \(c\) indicates the number of classes. The close-form solution for Eq. 1 can be obtained by solving the following equation for **W**:

\[\frac{\partial}{\partial\textbf{W}}(\frac{1}{2m}\|\textbf{H}\textbf{W}-\textbf {Y}\|_{F}^{2})=\frac{1}{m}\textbf{H}^{T}(\textbf{H}\textbf{W}-\textbf{Y})=0, \tag{2}\]

and the solution of Eq. 1 is:

\[\textbf{W}=(\textbf{H}^{T}\textbf{H})^{-1}\textbf{H}^{T}\textbf{Y}. \tag{3}\]

Cross-entropy loss:The optimization objective of the predictor is written as:

\[\arg\min_{\textbf{W}}-\frac{1}{m}\sum_{i=1}^{m}\sum_{j=1}^{c}\textbf{Y}_{i,j} \cdot\ln\textbf{P}_{i,j}, \tag{4}\]

\[\textbf{P}_{i,j}=\frac{\exp(\textbf{H}_{i,:}\textbf{W}_{:,j})}{\sum_{k=1}^{c} \exp(\textbf{H}_{i,:}\textbf{W}_{:,k})}, \tag{5}\]

where \(\textbf{Y}_{i,j}\) is the value at \(i\)-th row and \(j\)-th column of **Y**, and \(\textbf{P}_{i,j}\) the output of softmax layer that denotes the confidence level the \(i\)-th training sample belongs to \(j\)-th class. \(\textbf{H}_{i,:}\) and \(\textbf{W}_{:,j}\) respectively denote the \(i\)-th row of **H** and the \(j\)-th row of **W**.

As the close-form solution for Eq. 4 is unavailable, we use gradient descent algorithm to optimize it. To do this, we need to calculate the gradient for **W** according to the Jacobian matrix, and set a fixed step size to optimize Eq. 4 iteratively. The Jacobian matrix is calculated by:

\[\textbf{J}=\textbf{P}-\textbf{Y}, \tag{6}\]

and the gradient for **W** is computed as:

\[\frac{1}{m}\sum_{i=1}^{m}[\textbf{H}_{i,:}]^{T}\otimes\textbf{J}_{i,:}, \tag{7}\]

where **J** denotes the Jacobian matrix, and \(\otimes\) the Kronecker product.
Sparsemax loss:Different to softmax, the sparsemax function [24] produces sparse output probabilities by enforcing a constraint that the output confidence vector has at most a certain number of non-zero elements. It encourages the model to only assign high probabilities to the most relevant classes, while setting all other probabilities to zero. The sparsemax function achieves this by projecting the input vector onto a simplex, which is a convex polytope whose vertices lie on the coordinate axes. The formulaic expression for sparsemax is as follows:

\[\mathrm{sparsemax}(\mathbf{z}):=\arg\min_{\mathbf{p}\in\triangle^{c-1}}\| \mathbf{p}-\mathbf{z}\|^{2}, \tag{8}\]

where \(\triangle^{c-1}:=\{\mathbf{p}\in\mathbb{R}^{c}|\mathbf{1}^{T}\mathbf{p}=1, \mathbf{p}\geq\mathbf{0}\}\) is the \((c-1)\)-dimensional simplex. The resulting projection is unique and can be computed efficiently using a sorting algorithm [24].

We also use the gradient descent algorithm to approximate the sparsemax loss optimization objective:

\[\arg\min_{\mathbf{W}}\frac{1}{m}\sum_{\mathbf{z}\in\mathbf{Z}}\sum_{k=1}^{c}L_ {\mathrm{sparsemax}}(\mathbf{z};k), \tag{9}\]

\[L_{\mathrm{sparsemax}}(\mathbf{z};k)=-\mathbf{z}_{k}+\frac{1}{2}\sum_{j\in S (\mathbf{z})}(\mathbf{z}_{j}^{2}-\tau^{2}(\mathbf{z}))+\frac{1}{2}, \tag{10}\]

where \(\mathbf{z}\in\mathbb{R}^{c}\) is the output logits, \(\mathbf{Z}=[\mathbf{z}_{1};\mathbf{z}_{2};...;\mathbf{z}_{m}]^{T}\in\mathbb{ R}^{m\times c}\) the set of output logits of \(m\) training samples, \(S(\mathbf{z})\) the support of \(\mathrm{sparsemax}(\mathbf{z})\), and \(\tau(\mathbf{z})=\frac{(\sum_{j\in S(\mathbf{z})}\mathbf{z}_{i})-1}{|S( \mathbf{z})|}\). The Jacobian matrix is calculated by:

\[\mathbf{J}=\mathrm{sparsemax}(\mathbf{Z})-\mathbf{Y}, \tag{11}\]

and the gradient for \(\mathbf{W}\) is computed using the same formula as Eq. 7.

### Inference Process

In the inference phase each predictor outputs a prediction and the final prediction is determined by combining all the individual predictors. In image classification tasks this is typically done by summing the predictions of all the predictors and choosing the index of the maximum value as the predicted class.

The inference process for FF involves generation of samples for every label and computation of their goodness, whereas our method is able to directly provide multi-class label distributions. Our method gains higher inference efficiency and is more suitable for multi-class prediction tasks.

The detailed optimization algorithm is shown in Algorithm 1. During the training stage, the model generates the prediction block by block at step 3 and step 4, where the \(\mathbf{h}_{j}^{i}\in\mathbb{R}^{d}\) and \(\mathbf{\tilde{y}}_{j}^{i}\in\mathbb{R}^{c}\) are the intermediate representation and prediction of \(i\)-th block for \(j\)-th training sample. At step 5, the predictors are updated based on a specific loss function (e.g., Eqs. 1, 4, and 9), using the corresponding optimization strategy as described in Section 3.3. The predictors consist of a fully connected layer for MSE, and an additional activation layer is followed for CE and SL. During the inference stage, each predictor outputs a prediction for the test set at step 3 and step 4, where \(\mathbf{\hat{y}}_{j}^{i}\in\mathbb{R}^{c}\) is the prediction of \(i\)-th block for the \(j\)-th test sample. Then the predictions of each block (e.g., \(\mathbf{\hat{Y}}^{i}\in\mathbb{R}^{n\times c}\)) are summed to derive the final prediction \(\mathbf{\hat{Y}}\in\mathbb{R}^{n\times c}\). The predicted class for \(j\)-th test samples is then calculated at step 7 as: \(\hat{y}_{j}=\arg\max_{k}\mathbf{\hat{Y}}_{j,k}\).

## 4 Discussion

### Relationship to the Forward-Forward Algorithm

Both FF and our method are alternative approaches to backpropagation, as highlighted in [9]. They both have advantages over the traditional backpropagation method, such as the ability to handle non-differentiable components and increased flexibility in network architecture. However, the differences between the two methods are still significant.

The graphical depiction and comparison of BP, FF, and proposed CaFo are shown in Figure 2. The diagram style used in the figure is adapted from [22], and we follow the same approach to depict CaFo and highlight the differences between
our model and the other two methods. For BP, the model conducts a forward pass to derive the global loss, and then the gradients of the loss function with respect to the weights of the network are computed using the chain rule of calculus. The weights are updated using the obtained gradients and a learning rate hyper-parameter, which controls the size of the step taken in the opposite direction of the gradient. For FF, positive and negative samples are selected and samples are used to calculate a local loss for each layer. Once the local loss of the \(i\)-th layer is calculated (i.e., \(\mathbf{L}_{i}\)), the weights of the \(i\)-th layer (i.e., \(\mathbf{W}_{i}\)) will be updated according to \(\mathbf{L}_{i}\), and after that the \(\mathbf{W}_{i}\) will be fixed during the next training stage for the \((i+1)\)-th layer. The loss for each layer is calculated in sequence, and the layers of FF are sequentially updated by greedy decision.

Compared with FF, our CaFo network only has one input stream, avoiding the negative sampling process. Although the sequential update of each block in our framework is similar to FF, the local loss in CaFo directly measures the error between the predicted multi-class distribution and ground truth, rather than a simple goodness estimation. In addition, as the weights of each block (i.e. \(\mathbf{G}_{N}\)) are fixed, the predictor (i.e., \(\mathbf{W}_{N}\)) attached to each block is trained independently without the need for the training results from prior blocks, which makes the framework allow for the parallel training of each predictor.

### Relationship to Bagging

Bagging (Bootstrap Aggregating) [25] is an ensemble learning technique that combines multiple models to enhance the overall performance of a machine learning algorithm. The basic idea of bagging is to train some individual models on different subsets of the training data, and then combine their outputs by taking the average of majority vote. Bagging eliminates the variance of the individual models and prevents overfitting by reducing the impact of noisy data points or
outliers. This is achieved by generating multiple random samples of the training data, each of which is adopted to train a different model. The results are then combined to make a final prediction.

Our method can be viewed as a special case of Bagging, where the predictors are trained on intermediate features extracted at various scales, rather than on the original images. In other words, our approach does not train individual models on different subsets of the training data, but assigns each individual model with a specific intermediate representations of the complete training data, which has the advantage of utilizing the information contained in intermediate features extracted at different scales, thus helping to capture more complex patterns and increase the accuracy of the model.

### Relationship to Co-training

Co-training [26] is a popular semi-supervised learning method that trains multiple classifiers on the same dataset with different data views to improve classification accuracy by leveraging unlabeled data. The process involves initializing several classifiers with different feature sets and training them on labeled data subsets. The classifiers then exchange predictions on unlabeled data and retrain on new labeled data subsets using these predictions. This iterative process continues until convergence, with classifiers refining their feature sets and predictions.

As a semi-supervised learning technique, co-training is usually applied in scenarios where the labeled data is scarce but there is a large amount of unlabeled data available. In contrast, our method is a fully-supervised approach that trains multiple predictors for each block in parallel, allowing for more efficient and stable optimization of the model in supervised learning scenarios. As our method and co-training share the same idea of training multiple modules to improve the overall performance, our approach can be seen as the first stage of a co-training process, and can be easily transformed into a co-training method if an augmentation strategy for training set is introduced.

## 5 Experiments

The aim of this paper is to introduce the CaFo algorithm and to show that it works in relatively small neural networks containing a few million connections. We evaluate the proposed algorithm on four benchmarks in comparison with backpropagation and FF. Future research will focus on investigating the scalability of the proposed approach to handle large neural networks with a much greater number of connections.

### Datasets and Experimental Settings

The experiments are conducted on the MNIST [27], CIFAR-10 [28], CIFAR-100 [28] and Mini-ImageNet [29] datasets, all of which are widely-used in evaluating image classification algorithms. The standard splits of MNIST, CIFAR-10

Figure 2: Comparison of BackPropagation (BP) algorithm, Forward-Forward (FF) algorithm and the proposed Cascaded Forward (CaFo) algorithm.
and CIFAR-100 are directly used. In the case of Mini-ImageNet, which is often used for evaluating few-shot learning algorithms, all the classes are mixed, and for each class, 500 images are randomly sampled for training and 100 images for testing.

We compare the proposed CaFo algorithm with two baseline algorithms: backpropagation and the FF algorithm. Specifically, for MNIST and CIFAR-10, we report the results of both the original version of FF [9] and our reproduced version. For CIFAR-100 and Mini-ImageNet, we only report the results of our reproduced version, as the source code of the original version is not publicly available.

For the proposed CaFo algorithm, we set the number of blocks \(r\)=3. Each block consists of a convolutional layer followed by a ReLU activation function, a max-pooling layer and a batch normalization layer. The convolutional layer of the three blocks have the same kernel size, padding, and stride, which are set to be \(3\times 3\), 1, 1, and the output channels are set to be 32, 128, and 512 respectively.

As described in Section 3.3, different error measure functions can be adopted in our framework to guide the training. For analysis of the affects of different measure functions, in this section we report the results of three versions of our CaFo algorithms that adopt MSE (CaFo+MSE), cross-entropy (CaFo+CE) and sparsemax loss (CaFo+SL) as the measure function respectively. The predictor is a fully connected layer without bias, followed by a softmax layer for CE and SL. For CaFo+CE and CaFo+SL, each predictor is trained for 5000 epochs on MNIST and CIFAR-10, and for 1000 epochs on CIFAR-100 and Mini-ImageNet. All experiments are run on AMD EPYC 7542 32-Core Processor with a Nvidia GeForce RTX 3090 GPU.

### Performance Comparison

The comparison of these algorithms on four datasets is summarized in Tables 1 and 2. It is obvious our method achieves overall improvements in test error rate compared with FF. Even the simplest version (CaFo+MSE) outperforms the reproduced FF on all datasets. For FF, we observe a significant discrepancy in its results between the two tables. FF has low training error when the dataset is easy to discriminate (e.g., CIFAR-10, MNIST), but has high training error when tackling much more complex datasets (e.g., CIFAR-100, Mini-ImageNet). The reason for the underfitting of FF probably lies in that the contrastive loss function based on positive and negative samples may not provide effective guidance. If the training samples for each class are scarce (e.g., CIFAR-10 vs. CIFAR-100) or the samples contain more irrelevant information (e.g., CIFAR-10 vs. Mini-ImageNet), the goodness estimation of positive and negative samples may not effectively help the FF model learn the potential distribution for each class because of the uncertain

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**CIFAR-10**} \\ \hline
**Error rate (\%)** & Training & Test & Training & Test \\ \hline
**BP (from [9])** & - & 1.40 & 2.00 & 39.00 \\
**FF (from [9])** & - & 1.37 & 24.00 & 41.00 \\
**FF (reprodu.)** & 0.57 & 2.02 & 10.36 & 46.03 \\ \hline
**CaFo+MSE** & 0.71 & 1.55 & 12.56 & 34.79 \\
**CaFo+CE** & 0.04 & 1.30 & 9.22 & 32.57 \\
**CaFo+SL** & 0.06 & 1.20 & 9.58 & 33.74 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Error rate on MNIST and CIFAR-10

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**CIFAR-100**} & \multicolumn{2}{c}{**Mini-ImageNet**} \\ \hline
**Error rate (\%)** & Training & Test & Training & Test \\ \hline
**FF (reprodu.)** & 29.32 & 78.67 & 27.84 & 91.71 \\ \hline
**CaFo+MSE** & 6.06 & 63.94 & 6.66 & 85.50 \\
**CaFo+CE** & 0.06 & 59.24 & 0.01 & 78.20 \\
**CaFo+SL** & 0.02 & 61.96 & 0.02 & 81.47 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Error rate on CIFAR-100 and Mini-ImageNet
quality of negative samples and the lack of supervision information directly related to the category information. This may lead to the poor fitting for the training set.

In contrast, our method exhibits a smaller discrepancy between the two kinds of datasets. The training error rate of CaFo is less than 10% in most trials, demonstrating better fitting ability than FF. For test error rate, the performance gap between the two is still significant, especially on more complex datasets. For fair comparison, the results of CaFo are reported without deployment of any regularization trick into the model, and thus the performance of CaFo still has the potential to be improved by means of regularization strategies.

Comparing the three variants of our method, we find that they obtain overall similar performance. CaFo+MSE has the highest test error and train error on all the datasets, which indicates MSE may not be good enough for error estimation in comparison with other two variants that adopt classification-oriented loss. In contrast, both CaFo+CE and CaFo+SL exhibit low and very similar training and test error on the four datasets, indicating the effectiveness of gradient descent optimization strategy and the superiority of CE and SL compared with MSE.

### Time Comparison

Furthermore, we report the time required for training and testing our method in comparison with FF as shown in Table 3. For each trial, we report the time required for the entire training process and the test time that includes the time for the forward pass of the test samples and the time for calculating the predicted category. The experimental settings are the same as those in Section 5.1.

Comparing our three variants with FF, we observe that our methods exhibit significant improvements in training efficiency, especially for CaFo+MSE, which has a much shorter training time than the other three approaches due to its direct calculation of the close-form solution without iterative training. CaFo+CE and CaFo+SL also show significant improvements in training efficiency on MNIST, CIFAR-10 and CIFAR-100, while taking more time on Mini-ImageNet. This is because we divide the training set of CIFAR-100 and Mini-ImageNet into 20 and 200 training batches, respectively, for the two variants due to memory limitation, which leads to the extra time consumption of data movement. However in the trials of MNIST and CIFAR-10 where the batch number equals one, we find the two CaFo variants are trained significantly faster than FF.

In terms of test time, our methods demonstrate comparable efficiency on MNIST and CIFAR-10, but significantly shorter test time on CIFAR-100 and Mini-ImageNet. The reason for this lies in that our methods directly output the prediction of multi-class distribution, rather than compute the goodness estimation for each candidate label as that in FF. It tremendously improves the test efficiency when the number of categories is relatively large, such as on CIFAR-100 and Mini-ImageNet.

### Investigation of the Number of Blocks

In Tables 1 and 2 we fix the number of blocks \(r\)=3 for fair comparison with [9], in which the results of BP and FF with three layers are reported. However, the number of blocks may have diverse effects on different datasets. Here we investigate how the number of blocks influences the performance of our method.

In Figure 3 the error rates of CaFo+MSE and CaFo+CE with the number of blocks ranging from 1 to 10 are reported. As CIFAR-10 allows for a maximum of five \(2\times 2\) max-pooling layers with \(stride\)=2, and four are available for MNIST, we remove the max-pooling layer from some blocks to keep the number of max-pooling layers within the upper bound if necessary. We observe that the training error rates (dashed lines) continuously decrease with the increasing number of blocks, indicating an enhancement of fitting ability as more learnable predictors are introduced. However, the test error curves (solid lines) show that the model is more or less affected by overfitting when the number of blocks is large.

\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**CIFAR-10**} & \multicolumn{2}{c}{**CIFAR-100**} & \multicolumn{2}{c}{**Mini-ImageNet**} \\ \hline
**Time (s)** & Training & Test & Training & Test & Training & Test & Training & Test \\ \hline
**FF (reprodu.)** & 1655.22 & 0.02 & 2374.30 & 0.04 & 2376.77 & 0.42 & 3668.35 & 2.03 \\ \hline
**CaFo+MSE** & 0.62 & 0.06 & 1.21 & 0.07 & 1.40 & 0.08 & 22.55 & 0.32 \\ \hline
**CaFo+CE** & 612.22 & 0.05 & 873.56 & 0.07 & 1178.42 & 0.08 & 8101.42 & 0.52 \\
**CaFo+SL** & 837.56 & 0.06 & 1082.45 & 0.09 & 2192.56 & 0.10 & 14128.06 & 0.49 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Time for training and testing
### Performance of each Predictor

To investigate the performance of each predictor, we report the test error rate of each predictor for CaFo+MSE and CaFo+CE, on three datasets. As shown in Figure 4, the test error of the final prediction is lower than that of each individual predictor. In addition, we surprisingly find that a deep-layer predictor tends to obtain better performance than a shallow-layer one, demonstrating that the deep features play a more important role in enabling the predictor to make correct classifications. Overall, the results in Figure 4 well validate that the combination of all the predictors effectively contributes to the final prediction. Although both FF and our method demonstrate that the simple summing of each block's prediction (goodness) is good enough for comparable performance, however, designing more ingenious heuristic combination strategies for these non-backpropagation approaches remains to be studied. We will concentrate on this problem in future work.

## 6 Conclusion

In this paper, we propose a new learning framework for deep neural networks that provides a viable alternative to the backpropagation algorithm. The proposed CaFo demonstrates the feasibility of using only a single forward pass without

Figure 4: Test error rate of each predictor.

Figure 3: Error rates with different numbers of blocks.
the need for additional backward pass in the neural network training process, which offers significant improvements in prediction accuracy, training efficiency and simplicity compared with FF. Extensive experiments well validate the superior performance over the competitive methods on several public benchmarks.
