Split-Boost Neural Networks
The calibration and training of a neural network is a complex and
time-consuming procedure that requires significant computational resources to
achieve satisfactory results. Key obstacles are a large number of
hyperparameters to select and the onset of overfitting in the face of a small
amount of data. In this framework, we propose an innovative training strategy
for feed-forward architectures - called split-boost - that improves performance
and automatically includes a regularizing behaviour without modeling it
explicitly. Such a novel approach ultimately allows us to avoid explicitly
modeling the regularization term, decreasing the total number of
hyperparameters and speeding up the tuning phase. The proposed strategy is
tested on a real-world (anonymized) dataset within a benchmark medical
insurance design problem.
Raffaele Giuseppe Cestari, Gabriele Maroni, Loris Cannelli, Dario Piga, Simone Formentin
# Split-Boost Neural Networks

###### Abstract

The calibration and training of a neural network is a complex and time-consuming procedure that requires significant computational resources to achieve satisfactory results. Key obstacles are a large number of hyperparameters to select and the onset of overfitting in the face of a small amount of data. In this framework, we propose an innovative training strategy for feed-forward architectures - called _split-boost_ - that improves performance and automatically includes a regularizing behaviour without modeling it explicitly. Such a novel approach ultimately allows us to avoid explicitly modeling the regularization term, decreasing the total number of hyperparameters and speeding up the tuning phase. The proposed strategy is tested on a real-world (anonymized) dataset within a benchmark medical insurance design problem.

Artificial intelligence, deep learning, machine learning, neural networks, hyperparameter tuning, regularization +
Footnote †: footnoteinfo
first are explicitly obtained. However, our work differs from traditional ELMs, where the parameters of the first layer are fixed a priori (and not updated) after initial randomization, see Huang et al. (2021). Additionally, backpropagation in ELMs is removed. Instead, we still perform backpropagation to train the first layer's parameters. Likewise, a one-shot optimization is performed (ridge regression) with the substantial difference in how the data are divided to parallelize the computation on 2 different batches to retrieve the parameters of the second layer explicitly. This strategy is expected to reduce the occurrence of overfitting without resorting to a regularization term.

The paper is organized as follows. In Section 2 a review of the state of the art, the main challenges, limitations, and the benchmark network structure are presented. In Section 3 the mathematical formulation of the training strategy of the _split-boost_ neural network is presented: in the first part of the section the mathematical notation is introduced, then in Section 3.1 the _split-boost_ optimization problem for optimal weights computation is defined, and finally in Section 3.2 the details on training procedure, the policy for best epoch retrieval and the learning rate switching strategy are described. In Section 4 a numerical comparison between the traditional feed-forward neural network training and the split-boost training is presented. The paper is ended by some concluding remarks.

## 2 Problem Statement

The goal of this work is to propose an alternative training strategy for a classic feed-forward neural network. Without modification of the network structure, we want to show that using the same amount of data in a different way allows us to improve training performances and achieve implicit regularization, namely to obtain a regularization effect without modeling it explicitly. This allows us to simplify the tuning procedure of the network, reducing the number of hyperparameters to be calibrated.

In this section, the basic structure of the architecture of a feed-forward neural network is introduced without going into details as this morphology is widely studied in the literature, see Rosenblatt (1958), Bishop (1995), Nair and Hinton (2010), Goodfellow et al. (2016). A feed-forward neural network is a deep learning model which, based on the interaction of several processing units, called neurons, introducing non-linearities on the inputs, can perform various tasks ranging from classification, regression and prediction of time series. The parameters that build this model are represented by the weight matrices that correspond to the different layers of neurons building the architecture. These weights are updated through epochs (e.g., several passes through the dataset) via gradient descent (or one of its variants) and the help of the well-known backpropagation algorithm Rumelhart et al. (1986), for the calculation of the gradient of the errors with respect to the weights of the network.

Neural networks are models with a high descriptive capacity but are characterized by the phenomenon of overfitting, which can cause negative repercussions on the generalization capacity. Overfitting is one of the curses of general statistical learning. It often discourages users of artificial intelligence as the lack of a sufficient amount of data makes the architectures prone to its onset. In the literature, there are several documents relating to the description of the problem (see Tieleman and Hinton (2009), Wan and et al. (2013), Keskar and et al. (2016), Zhang and et al. (2021)). In the case of feed-forward neural networks, there are several strategies that can counter overfitting such as dropout, early stopping, regularization, data augmentation, batch normalization (see, e.g., Srivastava and et al. (2014), Ioffe and Szegedy (2015), Hinton et al. (2015), Loshchilov and Hutter (2017)). The goal of these methodologies is to prevent the neural network from overly relying (e.g. fitting the noise) on the data used in training. To do this, the idea is to reduce the number of network hyperparameters (or limit their norm) or artificially increase the available data. In this study, we assume _regularization_ as a benchmark of anti-overfitting methodology. This technique consists in adding into the cost function that must be minimized (or equivalently the reward that must be maximized) a term that depends on the norm of the weights of the different layers of the neural network. During the training step, the neural network is forced to keep the norm of these weights limited to prevent an excessive increase in the cost term.

In Figure 1 a sketch of the neural network architecture in matrix notation used in this work is shown. The considered structure consists of 2 layers (hidden and output) and is designed to solve a regression problem. Each input is processed by each layer (and by each neuron per layer) through the non-linear activation function \(f_{1}\) (rectified linear unit, RELU). \(X\in\mathbb{R}^{N\times D}\) is the input data matrix, where \(N\) is the number of samples and \(D\) is the number of features. \(W_{1}\in\mathbb{R}^{D\times H}\) is the weight matrix of the first (hidden) layer, where \(H\) is the number of neurons. \(W_{1}^{b}\in\mathbb{R}^{H\times 1}\) is the bias vector of the first (hidden) layer. \(Z_{1}\in\mathbb{R}^{N\times H}\) are the pre-activations of the first layer. \(X_{1}\in\mathbb{R}^{N\times H}\) are the activations of the first layer. \(W_{2}\in\mathbb{R}^{H\times 1}\) is the weight matrix of the second (output) layer. \(W_{2}^{b}\in\mathbb{R}\) is the bias of the output layer. \(Y,\hat{Y}\in\mathbb{R}^{N\times 1}\) are respectively the matrix of the targets and the matrix of the prediction, and finally, \(J\) is the loss function. Layer biases \(W_{1}^{b}\) and \(W_{2}^{b}\) follow the same training procedure of the corresponding layer weights. For sake of compactness of notation, they are omitted.

The optimization problem that must be solved to find the optimal weights of a feed-forward neural network with 2 fully connected layers, following the traditional training procedure known in literature has the structure of the following unconstrained minimization problem:

\[\min_{W_{1},W_{2}}\frac{1}{2N_{t}}||Y_{t}-f_{1}(X_{t}\cdot W_{1})\cdot W_{2} ||_{2}^{2}+\frac{\lambda}{2}\sum_{i=1}^{2}||W_{i}||_{2}^{2} \tag{1}\]

where the subscript \(t\) refers to the training set and \(\lambda\) is a hyperparameter that controls the intensity of the regularization. In the next section, we present an alternative, novel, way to write down the same optimization problem for training a 2-layer fully connected neural network in a way that allows us to exploit at best the training information introducing an implicit regularizing effect and

Figure 1: Neural network architecture
simultaneously reducing the number of parameters to be tuned. Indeed, within this framework, the regularization factor is not needed anymore.

## 3 Mathematical Formulation of Split-Boost Networks

This section illustrates this alternative training strategy of the proposed Split-Boost Neural Network. The updating of the weight parameters takes place separately for the 2 layers (hidden and output) that characterize the network architecture. The idea is to formulate the training procedure as a bilevel optimization problem, whose outer optimization variables are the weights of the hidden layer \(W_{1}\), and the inner optimization variables are the weights of the output layer \(W_{2}\). For a regression problem, the optimal values of the parameters of the output layer \(W_{2}\) can be obtained in closed form by solving two least square problems. These least square problem solutions represent the constraints of the first optimization problem.

The algorithm involves first a _splitting step_, in which the training set is divided into two sub-sets (a reasonable choice is to divide it equally, we do not claim that this choice is the only possible one, however, it guarantees that both optimization problems see the same amount of data, avoiding the unbalancing towards one of the two partitions.). Both subsets are then used to solve, separately, two least squares problems. Once the least squares problems are solved (e.g. the optimal values for \(W_{2}\) with respect to the two sub-sets are found), the whole training set is used to update the values of \(W_{1}\).

From here, _the boosting idea_. The optimal values obtained with the first sub-set are used to generate the prediction for the data belonging to the second sub-set and vice-versa. Our goal is to show that this methodology can effectively replace the regularization term in a traditional feed-forward neural network, overcoming its performance. This step represents one of the main differences compared to traditional network training: dividing the training set into 2 batches, used to calibrate the parameters of the second layer independently, allows us to improve the information content extraction. In Table 1 the symbols and their description is summarized.

### Optimization problem, forward and backward propagations.

The heart of the training algorithm is represented by the optimization problem shown in Equation (2). Since this cannot be solved in closed form, it is solved iteratively through a descending gradient problem. The value of the \(W_{1}\) parameters is therefore updated as the epochs pass.

\[\min_{W_{1}} J =\frac{1}{2N_{b}}\|Y_{b}-f_{1}(X_{b}\cdot W_{1})\cdot W_{2a}^{*}(W_ {1})\|_{2}^{2}\] \[+\frac{1}{2N_{a}}\|Y_{a}-f_{1}(X_{a}\cdot W_{1})\cdot W_{2b}^{*} (W_{1})\|_{2}^{2} \tag{2a}\] \[W_{2a}^{*}(W_{1}) =\operatorname*{argmin}_{W_{2}}\frac{1}{2N_{a}}\|Y_{a}-f_{1}(X_{a }\cdot W_{1})\cdot W_{2}\|_{2}^{2}\] (2b) \[W_{2b}^{*}(W_{1}) =\operatorname*{argmin}_{W_{2}}\frac{1}{2N_{b}}\|Y_{b}-f_{1}(X_{b }\cdot W_{1})\cdot W_{2}\|_{2}^{2} \tag{2c}\]

In what follows, the subscripts \(k,j\) are used to refer both to training subsets A and B. Since the description of the equations referring to the two training sets mirror each other, the notation will follow as \(\forall k,j\in[a,b],k\neq j\). All the procedure is repeated for both admissible values of \(k\) and \(j\), training is _splitted_. The subscript notation is summarized in Table 2.

In the following equations, the forward propagation is described. We do not go into details as this step is well-known in literature (see Rumelhart et al. (1986), Werbos (1988), Fahlman and Lebiere (1989), Bartlett et al. (2006), LeCun and et al. (1998)).

\[Z_{1k} =X_{k}\cdot W_{1} \tag{3a}\] \[X_{1k} =f_{1}(Z_{1k})\] (3b) \[\hat{Y}_{k} =X_{1k}\cdot W_{2j}\] (3c) \[J_{W_{2k}} =\frac{1}{2N_{k}}\|Y_{k}-\hat{Y}_{k}\|_{2}^{2} \tag{3d}\]

We can compute explicitly the optimal values for the output layer parameters \(W_{2}\) solving a least square problem:

\[W_{2k}^{*}=(X_{1k}^{T}\cdot X_{1k})^{-1}\cdot X_{1k}^{T}Y_{k}. \tag{4}\]

Solving explicitly Equation (4) we derive the optimal values for the output layer parameters computed with the forward pass in the two separate training sub-sets. Notice that \(W_{2k}^{*}\) is function of \(W_{1}\) through the dependency on \(X_{1k}\). For sake of brevity in the following derivation, we write \(W_{2k}^{*}\) in place of \(W_{2k}^{*}(W_{1})\). For sake of compactness, the Jacobian expression is derived in scalar form. Nonetheless, it must be interpreted as the matrix of first-order partial derivatives of a vector-valued function with respect to its input variables. Since \(W_{2k}^{*}\) is an optimum computed explicitly, it is true that:

\[\frac{\partial J_{k}(W_{1},W_{2k})}{\partial W_{2k}}\bigg{|}_{W_{2k}=W_{2k}^{ *}}=\frac{\partial J_{k}(W_{1},W_{2k}^{*})}{\partial W_{2k}^{*}}=0 \tag{5}\]

Differentiating both sides with respect to \(W_{1}\) and using the chain rule according to the influence diagram in Figure 2:

\begin{table}
\begin{tabular}{|l|l|} \hline
**Symbol** & **Description** \\ \hline \(N\) & Number of samples \\ \(t,v,ts\) & Training, validation and test sets \\ \(a\) & Training set partition “A” \\ \(b\) & Training set partition “B” \\ \(D\) & Number of input features \\ \(X\in\mathbb{R}^{N\times D}\) & Input data \\ \(Y,\hat{Y}\in\mathbb{R}^{N\times 1}\) & Output data, prediction \\ \(Z\in\mathbb{R}^{N\times H}\) & Pre-activations \\ \(X_{1}\in\mathbb{R}^{N\times H}\) & Activations \\ \(H\) & Number of neurons per layer \\ \(W_{1}\in\mathbb{R}^{D\times H}\) & Weights of hidden layer \\ \(W_{2}\in\mathbb{R}^{H\times 1}\) & Weights of output layer \\ \(f_{1}:\mathbb{R}^{N\times H}\rightarrow\mathbb{R}^{N\times H}\) & Activation function \\ \(J\) & Cost function \\ \(\lambda\) & Regularization parameter \\ \(\gamma\) & Learning rate \\ \hline \end{tabular}
\end{table}
Table 1: Notation.

\begin{table}
\begin{tabular}{|l|l|l|} \hline
**Subscript** & **Set A** & **Set B** \\ \hline \(k\) & \(a\) & \(b\) \\ \(j\) & \(b\) & \(a\) \\ \hline \end{tabular}
\end{table}
Table 2: Training sets notation.
\[\frac{\partial}{\partial W_{1}}\left(\frac{\partial J_{k}(W_{1},W_{2k} ^{*})}{\partial W_{2k}^{*}}\right)=0\] \[\frac{\partial}{\partial W_{1}}\left(\frac{\partial J_{k}(W_{1},W_ {2k}^{*})}{\partial W_{2k}^{*}}\right)+\] \[+\frac{\partial}{\partial W_{2k}^{*}}\left(\frac{\partial J_{k}(W_ {1},W_{2k}^{*})}{\partial W_{2k}^{*}}\right)\cdot\frac{\partial W_{2k}^{*}}{ \partial W_{1}}=0\] \[\frac{\partial^{2}J_{k}(W_{1},W_{2k}^{*})}{\partial W_{1}\partial W _{2k}^{*}}+\frac{\partial^{2}J_{k}(W_{1},W_{2k}^{*})}{\partial W_{2k}^{*} \partial W_{2k}^{*}\Upsilon}\cdot\frac{\partial W_{2k}^{*}}{\partial W_{1}}=0\]

Solving for the Jacobian \(\frac{\partial W_{2k}^{*}}{\partial W_{1}}\):

\[\frac{\partial W_{2k}^{*}}{\partial W_{1}}=-\left(\frac{\partial^{2}J_{k}(W_ {1},W_{2k}^{*})}{\partial W_{2k}^{*}\partial W_{2k}^{*}\Upsilon}\right)^{-1} \cdot\frac{\partial^{2}J_{k}(W_{1},W_{2k}^{*})}{\partial W_{1}\partial W_{2k}^ {*}} \tag{7}\]

And then with notation abuse (we use the approximation symbol to treat the Jacobian as in the scalar case even though, as commented earlier, it must be interpreted as the matrix of first-order partial derivatives):

\[\mathbf{J}_{W_{1}}W_{2k}^{*}\approx\frac{\partial W_{2k}^{*}}{\partial W_{1}} \tag{8}\]

The Jacobian \(\mathbf{J}_{W_{1}}W_{2k}^{*}\) described in Equations (7) and (8) is a fundamental ingredient in the evaluation of the gradient of the cost function \(J\), to update \(W_{1}\) values. To complete the training procedure, the backward propagation step must be performed. Similarly to forward propagation, backward propagation also takes place by working simultaneously on both sub-sets of the training set. This allows us to derive the expressions of the gradients necessary for the final calculation of the gradient of \(J\) with respect to \(W1\):

\[\nabla_{\hat{Y}_{k}}J_{k} =\frac{1}{N_{k}}(\hat{Y}_{k}-Y_{k}) \tag{9a}\] \[\nabla_{X_{1k}}J_{k} =\nabla_{\hat{Y}_{k}}J_{k}\cdot W_{2j}^{*\tau}\] (9b) \[\nabla_{W_{2j}^{\prime}}J_{k} =X_{1k}^{T}\cdot\nabla_{\hat{Y}_{k}}J_{k}\] (9c) \[\nabla_{Z_{1k}}J_{k} =\nabla_{X_{1k}}J_{k}\circ f_{1}^{\prime}(Z_{1k}) \tag{9d}\]

For a deeper understanding of how backpropagation works, refer to Hecht-Nielsen (1987), Bottou (1991), Glorot et al. (2011).

Merging together the results obtained by solving Equations (3), (4) and (9) we derive the expression of the gradient of the cost function (2a) with respect to the weights of the first layer, \(W_{1}\), described in the following equation:

\[\nabla_{W_{1}}J= X_{b}^{T}\nabla_{Z_{1k}}J_{b}+\mathbf{J}_{W_{1}}W_{2a}^{*\tau} \nabla_{W_{2a}^{*}}J_{b}+\] \[+X_{a}^{T}\nabla_{Z_{1a}}J_{a}+\mathbf{J}_{W_{1b}}W_{2b}^{*\tau} \nabla_{W_{2a}^{*}}J_{a} \tag{10a}\] \[W_{1} =W_{1}-\gamma\nabla_{W_{1}}J\] (10b) \[W_{2} =\frac{W_{2a}^{*}+W_{2b}^{*}}{2} \tag{10c}\]

Thanks to Equations (10) and (4) we can update network parameters between two consecutive epochs. Notice the difference: \(W_{1}\) is updated through gradient descent in Equation (10b) while \(W_{2}\) is obtained as the average of the optimal \(W_{2}^{*}\) computed explicitly by looking at the two training sub-sets separately in Equation (10c). Notice that, in general, this closed-form solution for \(W_{2}\) might not be practicable (e.g. in the case of classification problems, the concatenation of non-linearities through the different layers of the network would make it unattainable to formulate a least squares problem, as in this case, efficiently solved analytically). In that case, a similar expression to (10a) must be derived and update \(W_{2}\) with gradient descent. However, for this setting, it improves the computational time. Equations (10) represent the heart of the algorithm, defining the _boosting_ step. The mixing of the two sub-sets of the training set embedded in the gradient expression and in optimal values for \(W_{2}\) can improve training performance achieving the same training cost in a lower number of epochs and at the same time avoiding overfitting without the aid of regularization terms. This is critical for reducing the number of hyperparameters. With the updated weights obtained in Equations (10), the prediction is computed as follows:

\[\hat{Y}=f_{1}(X\cdot W_{1})W_{2} \tag{11}\]

### Details on training procedure

Early stopping procedure is used. The stop condition for training is obtained by monitoring the status of the validation cost, as follows:

\[|J_{v}(k)-J_{v}(k-1)|<\epsilon \tag{12}\]

where \(k\) is the epoch index and \(\epsilon\)) is the stop threshold. If the variation of the validation cost between two consecutive epochs is less than \(\epsilon\), training stops and the optimal number of epochs is retrieved. After the computation of the optimal number of epochs, the network is re-trained using as new training set the sum of the original training set, and the validation set. Performances are evaluated considering the test set.

The split-boost strategy has a learning rate varying with the number of epochs. This choice avoids oscillations in training cost (an excessively large learning rate leads to fluctuations and consequent degradation of performance). In the example reported in the next section, the following switch condition is adopted:

\[\begin{cases}\gamma=\gamma^{*}\ if\ J_{t}(k)-J_{t}(k-1)>0\\ \gamma=\frac{\gamma^{*}}{10}\ otherwise\end{cases} \tag{13}\]

where \(\gamma^{*}\) is the best learning rate, chosen after a sensitivity analysis carried on the validation cost.

## 4 An Experimental Case Study

In this section, we show the comparison between a traditional feed-forward neural network and the split-boost neural network applied to a real-world regression problem. The code is written in Python 3.7.13. The Python library used to develop the neural networks is PyTorch Paszke and et al. (2019). Simulations run on an Intel Core i7-8750H with 6 cores, at 2.20 GHz (maximum single core frequency: 4.10 GHz), with 16 GB RAM.

Figure 2: Influence diagram describing the interaction between network layer parameters.
### Dataset description

The case-study is the medical insurance forecast of patients living in the U.S., given a set of clinical features. Data are open-source and offered in Lantz (2013). In Table 4.1 data features and targets are summarized. The goal is to predict the medical insurance charge (\(\$\)) given a set of \(D=6\) features: age, sex, BMI, number of children, smoking condition, and region of residence.

The number of people in the study is \(N=1338\). Dataset is split into training, validation and test sets according to the proportions shown in Figure 3. Test set is of \(20\%\) of the total. The validation set is \(16\%\). The training set is \(64\%\). In the case of the split-boost neural network, the training set is further divided into two halves, \(32\%\) each.

### Networks Hyperparameters

In this section, we show the hyperparameter tuning procedure. Sensitivity analysis of the learning rate \(\gamma\) and the regularization factor \(\lambda\) is performed. In the case of the split-boost neural network, there is the advantage of not having a regularization term which, instead, is replaced by the boosting procedure. Sensitivity analysis with respect to \(\gamma\) is described in the upper plot of Figure 4. The validation cost \(J_{v}=\frac{1}{2N_{v}}(Y_{v}-\hat{Y}_{v})^{2}\) associated to different choices of \(\gamma\) is shown. The best choice for both networks is \(\gamma=0.1\). In the lower plot of Figure 4 the sensitivity analysis with respect to the regularization hyper-parameter \(\lambda\) is shown (only for the feed-forward neural network), evaluating the validation cost. The goal is to compare the best possible feed-forward regularized neural network with the split-boost network. The best choice is \(\lambda=0.01\). In Table 4 the network parameters are summarized. Network architecture is the same. Same number of layers (2), neurons per layer (\(H=10\)) and activation function (RELU). Split-boost strategy does not need any regularization parameter.

### Numerical Simulations

In this section we show numerical insights about the training procedure of the split-boost network. Figure 5 shows the trend of the training cost along the training epochs for the two networks. The split-boost network is represented by the solid red line, the feed-forward is represented by the blue lines for varying values of \(\lambda\). The solid blue line corresponds to the best \(\lambda\) identified (see Table 4). As \(\lambda\) increases, the training cost of the feed-forward network increases. For smaller values of \(\lambda\), it decreases. Split-boost training cost converges to values close to those of the feed-forward regime for a substantially lower number of epochs, \(E_{TB}^{*}=50\) epochs against \(E_{FF}^{*}=200\) epochs. This allows us to conclude that the split-boost procedure, in this case, can converge to the maximum information content extractable from the training set, in a smaller number of epochs, and with an implicit regularization effect.

Notice that the selection of the best number of epochs (implementing early stopping strategy) is performed looking at validation cost as discussed in Section 3.2 for both the networks. After the best epoch retrieval, both the networks are re-trained considering as training set the union of the previous training and validation sets.

\begin{table}
\begin{tabular}{|l|l|l|} \hline
**Hyperparameter** & **FF** & **Split-Boost** \\ \hline \(L\) (Layers) & 2 & 2 \\ \(H\) & 10 & 10 \\ \(f_{1}\) & RELU & RELU \\ \(\gamma\) & 0.1 & 0.1 \\ \(\lambda\) & 0.01 & \(-\) \\ \hline \end{tabular}
\end{table}
Table 4: Networks Hyperparameters.

Figure 4: Hyperparameter tuning: sensitivity analysis with respect to the learning rate \(\gamma\) (upper panel); sensitivity analysis with respect to the regularization parameter \(\lambda\) (lower plot).

Figure 5: Training cost: comparison between _split-boost_ and _feed-forward_ network for different values of the regularization hyper-parameter \(\lambda\). Solid line is the best configuration of \(\lambda\).

\begin{table}
\begin{tabular}{|l|l|} \hline
**Features** & **Description** \\ \hline Age & Age of primary beneficiary \\ Sex & Insurance contractor gender \\ BMI & Body mass index \\ Children & Number of children \\ Smoker & Smoker condition \\ Region & Residential area in the U.S. \\ \hline
**Target** & **Description** \\ \hline Charge & Medical insurance bill [8] \\ \hline \end{tabular}
\end{table}
Table 3: Medical Insurance Dataset: features and target.
In Figure 6 the computational training time required by the 2 strategies is shown over 200 epochs. Split-boost strategy shows an average training time of \(T_{SB}=1.679\,s\) while the feed-forward of \(T_{FF}=1.179\,s\). On average, each epoch of the split-boost takes 42% more time than the feed-forward. However, considering the training convergence shown in Figure 5, which highlights that split-boost training cost converges at the regime in \(E^{*}_{SB}=50\) epochs against the \(E^{*}_{FF}=200\) of the feed-forward, leads to the average computational requirements of:

\[E^{*}_{SB}\cdot T_{SB}=83.95\,s\leq E^{*}_{FF}\cdot T_{FF}=235.8\,s. \tag{14}\]

In Figure 7 the re-training procedure of the split-boost neural network is shown. On the left the training and test costs are shown. If compared with Figure 5, the number of epochs to which the training cost regime is reached is higher. This depends on the fact that the training set is larger (it includes also the previous validation set). In the middle, the plot of the regression prediction versus target values for each person in the training set is shown. On the right plot, regression prediction versus target values for test data is shown. In the middle and right panels it is shown that the neural network can map successfully the non-linear relationship between the features collected in Table 1 and the regression target. There are some patients whose characteristics escape mapping: with similar features compared to the remaining patients, a higher medical insurance cost is attributed to them by the experts.

To evaluate the performance of the split-boost strategy with respect to the best regularized feed-forward neural network with \(\lambda=0.01\), derived from the sensitivity analysis carried on the regularizing term for the traditional FFNN in Fig. 4, 50 Monte Carlo randomizations of the dataset were performed, extracting 50 different combinations of training, validation and test sets. The results obtained on the test set were collected within the boxplots in Figure 8. The test cost \(J_{ts}=\frac{1}{2N_{ts}}(Y_{v}-\dot{Y}_{ts})^{2}\) obtained after the Monte Carlo randomizations in the case of the split-boost strategy is statistically lower. Split-boost test cost is lower in 72% of the cases. This proves that, with statistical evidence, the split-boost neural network overcomes the feed-forward neural network also in terms of prediction accuracy.

## 5 Conclusions

In this article, we have shown an alternative training approach for feed-forward neural networks. We have called this strategy "split-boost" to recall the idea that dividing (_split_) the dataset and combining the subsets _might_ lead to an improvement (_boost_) in performance.

In the considered real-world case study, the "split-boost" approach turns out to: lead to higher predictive performance than traditional training; and be computationally advantageous since in the training phase it converges within a smaller number of epochs, although the computational time per epoch is greater. The proposed strategy also implicitly counteracts overfitting.

Future activities will focus on an extensive validation of the proposed training strategy, as well as on its generalization and extension to multi-layer networks.
