Training Feedforward Neural Networks with Bayesian Hyper-Heuristics
The process of training feedforward neural networks (FFNNs) can benefit from
an automated process where the best heuristic to train the network is sought
out automatically by means of a high-level probabilistic-based heuristic. This
research introduces a novel population-based Bayesian hyper-heuristic (BHH)
that is used to train feedforward neural networks (FFNNs). The performance of
the BHH is compared to that of ten popular low-level heuristics, each with
different search behaviours. The chosen heuristic pool consists of classic
gradient-based heuristics as well as meta-heuristics (MHs). The empirical
process is executed on fourteen datasets consisting of classification and
regression problems with varying characteristics. The BHH is shown to be able
to train FFNNs well and provide an automated method for finding the best
heuristic to train the FFNNs at various stages of the training process.
Arné Schreuder, Anna Bosman, Andries Engelbrecht, Christopher Cleghorn
# \({}^{\star}\)Training Feedforward Neural Networks with Bayesian Hyper-Heuristics

###### Abstract

The process of training _feedforward neural networks_ (FFNNs) can benefit from an automated process where the best heuristic to train the network is sought out automatically by means of a high-level probabilistic-based heuristic. This research introduces a novel population-based _Bayesian hyper-heuristic_ (BHH) that is used to train _feedforward neural networks_ (FFNNs). The performance of the BHH is compared to that of ten popular low-level heuristics, each with different search behaviours. The chosen heuristic pool consists of classic gradient-based heuristics as well as _meta-heuristics_ (MHs). The empirical process is executed on fourteen datasets consisting of classification and regression problems with varying characteristics. The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

keywords: hyper-heuristics, meta-learning, feedforward neural networks, supervised learning, Bayesian statistics +
Footnote †: journal: Information Sciences

## 1 Introduction

A popular field of focus for studying _artificial neural networks_ (ANNs) is the process by which these models are trained. ANNs are trained by optimisation algorithms known as heuristics. Many different heuristics have
been developed and used to train ANNs [1]. Each of these heuristics has different search behaviours, characteristics, strengths and weaknesses. It is necessary to find the best heuristic to train ANNs in order to yield optimal results. This process is often non-trivial and time-consuming. Selection of the best heuristic to train ANNs is often problem specific [2].

A recent suggestion related to the field of _meta-learning_ is to dynamically select and/or adjust the heuristic used throughout the training process. This approach focuses on the hybridisation of learning paradigms. One such form of hybridisation of learning paradigms is that of hybridisation of different _heuristics_ as they are applied to some optimisation problem [3]. These methods are referred to as _hyper-heuristics_ (HHs) and focus on finding the best heuristic in _heuristic space_ to solve a specific problem.

In the general context of optimisation, many different types of HHs have been implemented and applied to many different problems [3]. However, research on the application of HHs in the context of ANN training is scarce. Nel [4] provides some of the first research in this field, applying a HH to _feedforward neural network_ (FFNN) training.

This research takes a particular interest in developing a population-based, selection HH that makes use of probability theory and Bayesian statistical concepts to guide the heuristic selection process. This paper presents a novel _Bayesian hyper-heuristic_ (BHH), a new high-level heuristic that utilises a statistical approach, referred to as _Bayesian analysis_, which combines prior information with new evidence to the parameters of a selection probability distribution. This selection probability distribution is the mechanism by which the HH selects appropriate heuristics to train FFNNs during the training process.

The selection mechanism implemented by the BHH is different from the _multialgorithm, genetically adaptive multiobjective_ (AMALGAM) and _biobjective hyperheuristic training algorithm_ (BOHTA) methods used by Nel [4], as well as the _hyper-heuristic Bayesian optimisation algorithm_ (HHBOA) proposed by Oliva and Martins [5]. The key differences include that the BHH does not follow an evolutionary approach to the selected low-level heuristics. As such the population does not generate offspring, but rather reuses entities in the population. Furthermore, the BHH implements a discrete credit assignment mechanism, not making use of pareto fronts as in the AMALGAM and BOHTA methods.

The remainder of this article is structured as follows: Section 2 provides background information on ANNs. Section 3 provides details on various
types of heuristics that have been used to train FFNNs. Section 4 presents background information on HHs and meta-learning. Section 5 presents background information on probability theory. Section 6 presents the developed BHH. Section 7 presents a detailed description of the empirical process and the setup of each experiment. Section 8 provides and discusses the results of the empirical study. Section 9 summarises the research that is done along with a brief overview of the findings.

## 2 Artifical Neural Networks

This research focuses on a particular type of ANN, referred to as _feedforward neural networks_ (FFNNs). FFNNs were the first and simplest type of ANNs developed [6] and implement an architecture consisting of input, hidden and output layers by arranging them in sequential order. Furthermore, FFNNs implement fully connected topologies, where each _artificial neuron_ (AN) in one layer is connected to all the ANs in the next, without any cycles [7]. In FFNNs, information moves forward, in one direction, from the input nodes, through the hidden nodes and finally to the output nodes.

_Training_ is the process whereby the weights of the FFNN are systematically changed with the aim of improving the _performance_ of the FFNN. Finding the optimal weights that produce the best performance on a given task is an optimisation problem. The optimisation algorithm used to find the optimal weights is referred to as a _heuristic_. Heuristics search for possible solutions in the solution-space and make use of information from the search space to guide to process.

During the training process, the FFNN is exposed to data while trying to produce some target outcome. The degree to which the produced outcome differs from the target outcome is referred to as _loss_. Since training of FFNNs is an optimisation problem, the goal of the training process is to minimise the loss. The loss is calculated using an error function.

## 3 Heuristics

A heuristic refers to an algorithmic search technique that serves as a guide to a search process where good solutions to an optimisation problem are being sought out. Many different techniques have been used to train FFNNs [8]. At the time of writing, the majority of work that is published on the training of FFNNs involves the use of gradient-based techniques [4].
Gradient-based heuristics are optimisation techniques that make use of derivatives obtained from evaluating the ANN error function. In the context of supervised learning, loss functions produce a scalar value that represents the error between the output of the ANN and the desired output. When using _gradient descent_ (GD) to train ANNs, the gradients of the loss function is used to adjust the weights of the ANN in order to minimise the error [9].

There are many variants of gradient-based heuristics. However, they all fundamentally apply the same generic GD framework that propagates the error signal backwards through the ANN. This algorithm is known as _back-propagation_ (BP), was popularised by Werbos [10].

The simplest type of GD algorithm is referred to as _stochastic gradient descent_ (SGD), which implements a gradient-based weight update step for each training pattern. In the context of this research, the implementation of SGD refers to the mini-batch training implementation of GD, where a small batch of training patterns are fed to the FFNN at once and the error function is aggregated across all training patterns.

Alternative variants have been proposed that lead to better control over the convergence characteristics of SGD. This research focuses on a number of these variants that include _momentum_ (Momentum) [11], _Nesterov accelerated gradients_ (NAG) [12], _adaptive gradients_ (Adagrad) [13], Adadelta [14], _root mean squared error propagation_ (RMSProp) [15] and _adaptive moment estimation_ (Adam) [8].

Gradient-based heuristics are sensitive to the problem that they are applied to, with hyper-parameter selection often dominating the research focus [16]. Blum and Roli [17] mention that since the 1980s, a new kind of approximate algorithm has emerged which tries to combine basic heuristic methods in higher level frameworks aimed at efficiently and effectively exploring a search space. These methods are referred to as MHs.

The biggest difference between MHs and gradient-based heuristics is that MHs make use of meta-information obtained as a result of evaluating the FFNN during training and is not limited to information about the search space [17]. This also means that MHs do not necessarily require the error function to be differentiable. Blum and Roli [17] provides advantages of MHs that include the following: they are easy to implement, they are problem independent and do not require problem-specific knowledge, and they are generally designed to find global optima, while gradient-based approaches can get stuck in local optima more often. Similar to gradient-based heuristics, a number of different meta-heuristics have been used to successfully train
FFNNs [1; 18; 19]. This research takes a particular interest in population-based MHs that have been used to train FFNNs. These include _particle swarm optimisation_ (PSO) [20], _differential evolution_ (DE) [21] and _genetic algorithms_ (GAs) [22].

## 4 Hyper-Heuristics

Burke et al. [23] define HHs as search methods or learning mechanism for selecting or generating heuristics to solve computational search problems. Burke et al. [24] mention that a HH is a high-level heuristic approach that, given a particular problem instance and a number of low-level heuristics, can select and apply an appropriate low-level heuristic at each decision point. HHs implement a form of _meta-learning_ that is concerned with the selection of the best heuristic from a pool of heuristics to solve a given problem. It can be said that HHs are concerned with finding the best heuristic in _heuristic space_, while the underlying low-level heuristics find solutions in the feasible _search/solution space_.

Burke et al. [23] proposed a classification scheme used to classify HHs. According to the proposed classification scheme, HHs are classified in two dimensions. These include the _source of feedback_ used during learning and the nature of the _heuristic search space_. For the dimension that involves the source of feedback, HHs can be classified as either _no learning_, _online learning_ or _offline learning_. For the dimension that involves the nature of the _heuristic search space_, HHs can be classified as either _heuristic selection_ or _heuristic generation_. Further distinction is made between _construction_ of heuristics and _perturbation_ of heuristics.

In the general context of optimisation, many different types of HHs have been implemented and applied to many different problems. Some notable examples include [23; 25; 26; 27]. Research on the application of HHs in the context of FFNN training is still scarce. Nel [4] provides some of the first research in this field, applying BOHTA, a novel adaptation of an evolutionary-based HH, known as the AMALGAM HH [28], to FFNN training. Furthermore, Oliva and Martins [5] provide HHBOA, the first use of Bayesian optimisation in a HH context. The method proposed by Oliva and Martins [5] uses a Bayesian selection operator to evolve combinations of low-level heuristics while looking for good problem solutions to a benchmark of optimisation functions, but does not apply a HH to the training of FFNNs.
This research takes a particular interest in a population-based, selection approach for HHs, with the particular intent of training FFNNs. In the context of population-based HHs, an entity pool exists that represents a pool of candidate solutions to the given problem. Each entity in the entity pool is assigned its own low-level heuristic from the heuristic pool. The selection of the best heuristic to apply to a candidate solution is based on the performance of the heuristic relative to that particular candidate solution at a particular point in the search process. Selection methods often make use of probabilistic approaches.

## 5 Probability

In general, there are two main views to probability and statistics. These include the _frequentist_ and the _Bayesian_ view of statistics. Naturally, Bayesian statistics is based on Bayes' theorem [29].

Bayesian statistics describe the probability of an event in terms of some belief, based on previous knowledge of the event and the conditions under which the event happened [30]. Bayes' theorem expresses how a degree of belief, expressed as a probability, should rationally change to account for the availability of related evidence.

One of the many applications of Bayes' theorem is to do statistical inference. Like FFNNs, Bayesian models need to be _trained_, a process known as _Bayesian analysis_. Bayesian analysis is the process by which prior beliefs are updated as a result of observing new data/evidence.

Bayesian analysis utilises the concept of conjugate priors. Wackerly et al. [31] state that conjugate priors are prior probability distributions that result in posterior distributions that are of the same functional form as the prior, but with different parameter values. The conjugate prior to a Bernoulli probability distribution is the Beta probability distribution and the conjugate prior to a categorical and multinomial probability distribution is the Dirichlet probability distribution [31].

## 6 Bayesian Hyper-Heuristics

This section presents the novel BHH. The general concept of the BHH is summarised as follows: the BHH implements a high-level heuristic selection mechanism that learns to select the best heuristic from a pool of low-level heuristics. These low-level heuristics are applied to a population of entities,
each implementing a candidate solution to a FFNN. The intent of the BHH is to optimise both the underlying FFNN and the FFNN training process. The BHH does so by learning the probability that a given heuristic will perform well at a given stage in the FFNN training process. These probabilities are then used as heuristic selection probabilities in the next step of the training process.

According to the classification scheme for HHs by Burke et al. (2017), the BHH is a population-based, meta-hyper-heuristic that utilises selection and perturbation of low-level heuristics in an online learning fashion. Figure 1 provides an illustration of the high-level architecture of the BHH. Algorithm 1 provides the high level pseudo-code implementation of the BHH. Discussions follow on the most important components of the BHH.

### Heuristic Pool

Generally speaking, the heuristic pool is a collection of low-level heuristics under consideration by the BHH. The heuristic pool contains the set of low-level heuristics that, together with their performance information, make up the heuristic space. Importantly, the heuristic pool must consist of a diverse set of low-level heuristics with varying capabilities. This research takes an interest in including both gradient-based heuristics as well as MHs in the heuristic pool. This approach is referred to as a _multi-method_ approach.

### Proxies

Heuristics often maintain a set of parameters that are used to control the behaviour of the heuristic. These parameters are refered to as heuristic _state_. The concept of proxies arises from the sparsity of state as maintained by different heuristics. Since heuristics maintain (possibly) different states, there is an uncertainty of state transition when switching between heuristics. A solution to state indifference is to _proxy_ heuristic state update operations. State is then maintained in two parts: primary and proxied state. Primary state refers to the state that is originally maintained by a heuristic. proxied state refers to the state that is not directly maintained by the heuristic, but can be updated by outsourcing the required state update operation to another heuristic. The BHH thus incorporates a mapping of proxied state update operations as given in the example in Table 1.

From the example given in Table 1, when heuristic A is selected, it will outsource state update operations from heuristic B for state parameter 2. Heuristic B will outsource from heuristic A for state parameter 3. Finally,
Figure 1: An illustration of the architecture and high level components of the _Bayesian hyper-heuristic_ (BHH).
**Algorithm 1** The pseudo-code for the implementation of the _Bayesian hyper-heuristic_ (BHH)

\begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{2}{l}{step \(\leftarrow\) 0} \\ \multicolumn{2}{l}{select initial heuristics} \\ \multicolumn{2}{l}{initialise population and entities} \\ \multicolumn{2}{l}{evaluate entities’ initial position} \\ \multicolumn{2}{l}{update population state} \\ \multicolumn{2}{l}{**while** stopping condition not met **do**} \\ \multicolumn{2}{l}{**for** all entities in entity pool **do**} \\ \multicolumn{2}{l}{**if** selected heuristic is gradient-based **then**} \\ \multicolumn{2}{l}{get gradients} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{apply low-level heuristic and proxy operations} \\ \multicolumn{2}{l}{update population state} \\ \multicolumn{2}{l}{log performance metrics to performance log} \\ \multicolumn{2}{l}{**if** step \(\mathbf{i}\) burn-in window size **then**} \\ \multicolumn{2}{l}{select heuristic} \\ \multicolumn{2}{l}{**else**} \\ \multicolumn{2}{l}{**if** step \% reanalysis interval = 0 **then**} \\ \multicolumn{2}{l}{apply Bayesian analysis} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{**if** step \% reselection interval = 0 **then**} \\ \multicolumn{2}{l}{select heuristic} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{**if** step \(<\) replay window size **then**} \\ \multicolumn{2}{l}{prune performance log} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{**end if**} \\ \multicolumn{2}{l}{**end for**} \\ \multicolumn{2}{l}{step \(\leftarrow\) step + 1} \\ \multicolumn{2}{l}{**end while**} \\ \hline \hline \end{tabular}
\end{table}
Table 1: An example of a mapping of proxied state update operation maintained by the BHH.
heuristic C will outsource from heuristic A and B for state parameters 2 and 3 respectively. In this way, all heuristics maintain all the state parameters.

### Entity Pool

The entity pool refers to a collection or _population_ of _entities_ that each represent a candidate solution to the underlying FFNN being trained. The BHH selects from the heuristic pool a low-level heuristic to be applied to an individual entity. The outcome of this selection process is a mapping table that tracks which heuristic has been selected for which entity. These heuristic-entity combinations are applied to the underlying FFNN. The BHH tracks the performance of each of these combinations throughout the training process in a performance log.

Entities represent candidate solutions to the model's trainable parameters (weights) and other heuristic-specific state parameters. These state parameters are referred to as _local_ state. Entities are treated as physical _particles_ in a hyper-dimensional search environment. Entities model concepts from physics. For example, the candidate solution is represented as the entity's position. The velocity and acceleration is then analogous to the gradient and momentum of the entity respectfully [32]. Examples of entity state parameters, as derived from various low-level heuristics, include entity position, velocity, gradient, position delta, first and second moments of the gradient, the loss, personal best positions, losses, and so on. The entity state parameters are updated by the associated heuristic.

The population state refers to a collection of parameters that are shared between the entities in the population. Population state is also referred to as _global_ state and represents the population's memory. The population state generally contains state parameters that are of importance to multiple heuristics, and usually tracks the state of the population and not individual heuristic. Some examples of population state that can arise from different heuristics include the population of entities themselves, the global best entity found so far, the overall best loss achieved thus far, and so on.

### Performance Log

Heuristic selection probability is calculated based on heuristic-entity performance over time. Evidence of heuristic-entity performance is thus required for the BHH to learn. Historical heuristic-entity performance outcomes are stored in a performance log. The performance log tracks information such as the current step, selected heuristic, associated entity, the loss achieved and
so on. Since the performance log can become very big, only a sliding window of the performance history is maintained at each step in the learning process. The sliding window is also referred to as a _replay_ window/buffer.

### Credit Assignment Strategy

The credit assignment strategy is a mechanism that assigns a discrete credit indicator to heuristics that perform well, based on their performance metrics such as loss. The credit assignment strategy implements a component of the "move acceptance" process as proposed by Ozcan et al. [33] and addresses the credit assignment problem as discussed by Burke et al. [23]. A good credit assignment strategy will correctly allocate credit to the appropriate heuristic-entity combination. This research implements the following credit assignment strategies to choose from: _ibest_ (iteration best), _pbest_ (personal best), _gbest_ (global best), _rbest_ (replay window best), and _symmetric_, where credit is assigned to all entity-heuristic combinations, regardless of their performance.

### Selection Mechanism

The BHH implements a probabilistic predictive model based on the fundamentals of the Naive Bayes algorithm. The BHH thus distinguishes between the following events: \(\mathbf{H}\), the event of observing _heuristics_, \(\mathbf{E}\), the event of observing _entities_, and \(\mathbf{C}\), the event of observing _credit assignments_ that indicate that the credit assignment _performance criteria_ are met. By Bayes' theorem, the selection mechanism implemented by the BHH is given as

\[P(\mathbf{H}|\mathbf{E},\mathbf{C};\mathbf{\theta},\mathbf{\phi},\mathbf{\psi})\propto P(\mathbf{E}|\mathbf{H} ;\mathbf{\phi})P(\mathbf{C}|\mathbf{H};\mathbf{\psi})P(\mathbf{H}|\mathbf{\theta}) \tag{1}\]

The predictive model thus models the _proportional_ probability of the event (selection of) heuristic \(\mathbf{H}\), given allocation to entity \(\mathbf{E}\) and credit requirement \(\mathbf{C}\), parameterised by sampled \(\mathbf{\theta}\sim Dir(\mathbf{\alpha};K)\), \(\mathbf{\phi}\sim Dir(\mathbf{\beta};K)^{J}\) and \(\mathbf{\psi}\sim Beta(\gamma_{1},\gamma_{0})\). In the aforementioned, \(K\) is the heuristic pool size and \(J\) is the entity pool size. The parameters \(\mathbf{\alpha}\), \(\mathbf{\beta}\), \(\gamma_{1}\) and \(\gamma_{0}\) are referred to as concentration parameters. The concentration parameters are used to parameterise the prior probability distributions. A provides mathematical derivations of the predictive model.
### Optimisation Step

The intent of the BHH is to gather evidence that can be used to update prior beliefs about which heuristics perform well during training. These beliefs are represented by the concentration parameters \(\mathbf{\alpha}\), \(\mathbf{\beta}\), \(\gamma_{1}\) and \(\gamma_{0}\). A change in prior beliefs is represented by a change in these concentration parameters. Specifically, it can be said that the optimisation process implemented by the BHH updates _pseudo counts_ of events that are observed in the performance logs. These pseudo counts track the occurrence of a heuristic, an entity, and resulting performance of these two elements. Through the credit assignment strategy, these pseudo counts are biased towards entity-heuristic combinations that meet performance requirements and yield credit allocations.

Generally, there are two different techniques that are used to train Naive Bayes classifiers. The frequentist approach implements _maximum likelihood estimation_ (MLE) and the Bayesian approach implements _maximum a posteriori estimation_ (MAP).

#### 6.7.1 Maximum A Posteriori Estimation

MAP is an approach to optimise the values for \(\hat{\theta}_{k}\), \(\hat{\phi}_{j,k}\) and \(\hat{\psi}_{k}\) by optimising the parameters of their probability distributions. This process is referred to as _Bayesian analysis_. Bayesian analysis makes use of the _posterior_ probability distribution. The concentration update operations yielded by MAP, are given as follows:

\[\alpha_{k}(t+1)=N_{k}+\alpha_{k}(t) \tag{2}\]

\[\beta_{j,k}(t+1)=N_{j,k}+\beta_{j,k}(t) \tag{3}\]

\[\gamma_{1,k}(t+1)=N_{1,k}+\gamma_{1,k}(t) \tag{4}\]

\[\gamma_{2,k}(t+1)=N_{0,k}+\gamma_{2,k}(t) \tag{5}\]
where \(N_{k}\) is a summary variable denoting the count of occurrences of heuristic \(k\), \(N_{j}\) is a summary variable denoting the count of occurrences of entity \(j\), \(N_{j,k}\) is a summary variable denoting the count of occurrences of heuristic \(k\) for entity \(j\), \(N_{1,k}\) and \(N_{0,k}\) are summary variables denoting the count of occurrences where heuristic \(k\) meets performance requirements and where heuristic \(k\) does not not meet performance requirements.

It can be said that the BHH implements a Gaussian process [34]. Since the reselection of heuristics happens at regular intervals, the outcome of a selection in one iteration may influence the outcome of another in the next iteration, making the implementation of the BHH a _hidden Markov model_ (HMM) [35].

### Hyper-Parameters

The following hyper-parameters are implemented by the BHH: the _heuristic pool_ configures the type of heuristics included in the heuristic pool, the _population size_ specifies the number of entities in the entity pool, the _credit assignment strategy_ specifies which credit assignment strategy to use, the _reselection interval_ determines the frequency of heuristic reselection, the _replay window size_ determines the maximum size of the performance log, the _reanalysis interval_ determines the frequency at which Bayesian analysis is applied, the _burn in window size_ determines the size of an initial window where experience is simply gathered without reanalysis, and finally, the _discounted rewards_ and _normalisation_ flags toggle scaling modifiers on values assigned by the credit assignment strategies, backwards in the performance log.

## 7 Methodology

This section provides the details of the implementation of the empirical process. At a high level the experimental procedure consist of a comparison between the BHH and standalone low-level heuristics. A number of datasets, models and heuristics are specified. Throughout the empirical process, a BHH baseline configuration is used.

### BHH Baseline

The BHH baseline is a name given to a specific configuration of the BHH which has been found to provide a reasonable baseline performance. The baseline configuration is used as the cornerstone configuration from which all
other heuristics and their configurations are evaluated. The BHH baseline configuration is given in Table 2. In Table 2, the heuristic pool configuration, _all_, refers to a configuration where the heuristic pool contains all the low-level heuristics, including all gradient-based heuristics and MHs.

### BHH vs. Low-Level Heuristics

For the standalone heuristics experimental group, a number of low-level heuristics are used along with their specified hyper-parameters. Each of these standalone low-level heuristics is compared to that of the BHH baseline configuration, across all datasets. The intent of the standalone heuristics experimental group is to determine if the BHH baseline configuration can generalise to multiple problems in comparison to individual low-level heuristics.

Additional to the BHH baseline configuration, two more BHH configurations are included. These include BHH configurations where the heuristic pool only makes use of gradient-based heuristics, and a configuration where the heuristic pool only makes use of MHs. The intent behind the inclusion of these configurations is to determine the effectiveness of multi-method approaches in the heuristic pool applied to training FFNNs.

### Heuristics

Table 3 contains a list of all the standalone, low-level heuristics that are used as well as their hyper-parameter configurations. In some cases, parameters are changed dynamically throughout the training process using a _decay schedule_. Note from Table 3 that values that are configured to make use of a decay schedule are presented with the initial value first and the decay rate in brackets next to it.

The mapping of proxied heuristic state update operations implemented by the BHH in the empirical process is given in Figure 2. In Figure 2, cells containing \(\mathbf{x}\) indicate that the associated heuristic implements that particular state parameter explicitly, and cells containing \(\mathbf{o}\) indicate that the state parameter is implicitly implemented as part of the BHH algorithm.

\begin{table}
\begin{tabular}{c c c c c c c c} heuristic pool & population & burn in & credit & reselection & replay & reanalysis & normalise & discounted rewards \\ \hline all & 5 & 0 & ibest & 10 & 10 & 10 & false & false \\ \end{tabular}
\end{table}
Table 2: The BHH baseline configuration as it is used in the empirical study.
\begin{table}
\begin{tabular}{l l l l}
**heuristic** & **configuration** & **value** & **citation** \\ \hline sgd & learning rate & 0.1 (0.01) & [12] \\ momentum & learning rate & 0.1 (0.01) & [12] \\  & momentum & 0.9 & \\ nag & learning rate & 0.1 (0.01) & [12] \\  & momentum & 0.9 & \\ adagrad & learning rate & 0.1 (0.01) & [13] \\  & epsilon & 1E-07 & \\ rmsprop & learning rate & 0.1 (0.01) & [15] \\  & rho & 0.95 & \\  & epsilon & 1E-07 & \\ adadelta & learning rate & 1.0 (0.95) & [14] \\  & rho & 0.95 & \\  & epsilon & 1E-07 & \\ adam & learning rate & 0.1 (0.01) & [8] \\  & beta1 & 0.9 & \\  & beta2 & 0.999 & \\  & epsilon & 1E-07 & \\ pso & population size & 10 & [36] \\  & learning rate & 1.0 (0.9) & \\  & inertia weight (w) & 0.729844 & \\  & cognitive control (c1) & 1.49618 & \\  & social control (c2) & 1.49618 & \\  & velocity clip min & -1.0 & \\  & velocity clip max & 1.0 & \\ de & population size & 10 & [37] \\  & selection strategy & best & \\  & xo strategy & exp & \\  & recombination probability & 0.9 (0.1) & \\  & beta & 2.0 (0.1) & \\ ga & population size & 10 & [38] \\  & selection strategy & rand & \\  & xo strategy & bin & \\  & mutation rate & 0.2 (0.05) & \\ \end{tabular}
\end{table}
Table 3: Low-level heuristics and their hyper-parameter configurations.
Figure 2: Mapping of proxied heuristic state update operations as implemented by the BHH
### Datasets

In the context of training FFNNs, the underlying models are trained across a number of datasets. All the datasets used in the empirical process originate from the UCI Machine Learning Repository [39]. Datasets are grouped by problem type and include seven classification and seven regression datasets. The details around the datasets used can be found in Tables 4 and 5. Each dataset is split into a training set comprising 80% of the data, and a test set comprising 20% of the data.

A number of classification datasets contain an unbalanced representation of classes. This work does not apply mechanisms to cater for class balancing, in order to eliminate as many variables and factors in the empirical process as possible.

### Models

All models trained in the empirical process follow implementations of shallow FFNNs with only one hidden layer. The number of hidden units used were determined empirically. Weights are initialised by means of _Glorot uniform sampling_[54]. The models and their configuration, as it is used for each dataset, are given in Table 6.

\begin{table}
\begin{tabular}{c c c c c c c c c}
**dataset** & **output** & **types** & **attributes** & **classes** & **instances** & **batch** & **steps** & **citation** \\ \hline iris & multivariate & real & 4 & 3 & 150 & 16 & 10 & [40] \\ car & multivariate & categorical & 6 & 4 & 1728 & 128 & 14 & [41] \\ abalone & multivariate & categorical, integer, real & 8 & 28 & 4177 & 256 & 17 & [42] \\ wine quality & multivariate & real & 12 & 11 & 4898 & 256 & 20 & [43] \\ mushroom & multivariate & categorical & 22 & 2 & 8214 & 512 & 17 & [44] \\ bank & multivariate & real & 17 & 2 & 45211 & 512 & 89 & [45] \\ diabetic & multivariate & integer & 55 & 3 & 100000 & 1024 & 98 & [46] \\ \end{tabular}
\end{table}
Table 4: Classification datasets

\begin{table}
\begin{tabular}{c c c c c c c c}
**dataset** & **output** & **types** & **attributes** & **instances** & **batch** & **steps** & **citation** \\ \hline fish toxicity & multivariate & real & 7 & 908 & 64 & 15 & [47] \\ housing & univariate & real & 13 & 506 & 32 & 16 & [48] \\ forest fires & multivariate & real & 13 & 517 & 32 & 17 & [49] \\ student performance & multivariate & integer & 33 & 649 & 32 & 21 & [50] \\ parkinsons & multivariate & integer, real & 26 & 5875 & 256 & 23 & [51] \\ air quality & multivariate, time series & real & 15 & 9358 & 256 & 37 & [52] \\ bike & univariate & integer, real & 16 & 17389 & 256 & 68 & [53] \\ \end{tabular}
\end{table}
Table 5: Regression datasets
### Performance Measures

_Binary cross entropy_ (BinXE) is used for classification problems with two classes and _sparse categorical cross entropy_ (SparseCatXE) is used for classification problems with more than two classes. For the classification problems, accuracy is also measured. For regression problems, the _mean squared error_ (MSE) is used as a performance metric. After training has completed, the _average rank_, based on test loss, for all configurations, is calculated at each mini-batch step.

### Statistical Analysis

Each experiment and configuration is trained for a maximum of 30 epochs and is repeated over 30 independent runs, for each of the datasets. No early-stopping mechanism is used. Statistical analysis is executed on the results from the test datasets. An average rank is calculated across all 30 runs, for both experimental groups and configurations, at each step, for every epoch of training.

The Shapiro-Wilk test for normality [55] (\(\alpha\) = 0.001) is used to determine if the results are normally distributed. The Levene's test for equality of variance [56] (\(\alpha\) = 0.001) is used. For experiments with three or more configurations, the ANOVA statistical test [57] (\(\alpha\) = 0.001) is used. The Kruskal-Wallis ranked non-parametric test [58] for statistical significance (\(\alpha\) = 0.001) is used for cases where data is not normally distributed. Finally, a post-hoc Tukey honest significant difference test [59] (\(\alpha\) = 0.001) is used

\begin{table}
\begin{tabular}{r c c c c c c c c}
**dataset** & **inputs** & **hidden** & **output** & **biases** & **parameters** & **topology** & **l1 activation** & **l2 activation** \\ \hline fish toxicity & 6 & 3 & 1 & yes & 25 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ iris & 4 & 5 & 3 & yes & 43 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ air quality & 12 & 8 & 1 & yes & 113 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ housing & 13 & 8 & 1 & yes & 121 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ wine quality & 13 & 10 & 7 & yes & 217 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ parkinsons & 21 & 10 & 1 & yes & 231 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ car & 21 & 10 & 4 & yes & 264 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ forest fires & 43 & 16 & 1 & yes & 721 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ abalone & 10 & 36 & 28 & yes & 1432 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ bank & 51 & 32 & 1 & yes & 1697 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ bike & 61 & 32 & 1 & yes & 2017 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ student performance & 99 & 32 & 1 & yes & 3233 & dense & LReLU (\(\alpha=0.3\)) & sigmoid \\ adult & 108 & 64 & 1 & yes & 7041 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ mushroom & 117 & 64 & 1 & yes & 7617 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ diabetic & 2369 & 32 & 3 & yes & 75939 & dense & LReLU (\(\alpha=0.3\)) & softmax \\ \end{tabular}
\end{table}
Table 6: Model configurations
from which significant ranking is retrieved. Descriptive and critical difference plots are then retrieved from these results to provide visual aid.

### Implementation

All implementations are done from first principles in Python 3.9 using Tensorflow 2.7 and Tensorflow Probability 0.15.0. The source code and data for this research is provided and made public at [https://github.com/arneschreuder/masters](https://github.com/arneschreuder/masters).

## 8 Results

This section provides the results of the empirical process that has been conducted. Detailed discussions follow on the outcomes of each experiment.

### Overview

This section provides a brief discussion on the general outcome of the empirical process as a whole and identifies some key aspects to be kept in mind when interpreting the results of the experiments.

Firstly, the BHH applies a form of online learning. As such, the BHH applies the learning mechanism during training in a single run of the training process. The training process is not repeated iteratively as is the case with some HHs.

Most of the training progress is observed to occur within the first five epochs. As a result, the BHH should apply most learning at the early stages of the training process. After five epochs, the training of most of the underlying FFNNs converges and little performance gain is observed after that point. Since this empirical process does not apply early stopping of the training process, the BHH will continue to explore the heuristic space beyond the five epoch mark.

The BHH does not implement a type of move-acceptance strategy where the application of a heuristic to an entity is only accepted if it leads to a better solution. In some cases, the BHH then selects heuristics that yield sub-optimal results, but is shown to mostly return to optimal solutions over a number of steps.

Given the stochastic nature of the heuristic selection mechanism, sufficient samples of the performance of each heuristics-entity combination in the performance log are required for the BHH to learn. This requirement is further strengthened by the Bayesian nature of the probabilistic model
implemented by the BHH. The probabilistic model implements _probability distributions of heuristic selection probabilities_ and as such, insufficient samples in the performance log could render a form of random search.

By default, the BHH baseline configuration has a reanalysis interval of 10, and a replay window size of 10, which is a small window to learn from. Despite the small reanalysis interval and the small replay window size, it should be observed that the BHH exploits small performance biases and finds small performance gains throughout.

### BHH vs. Low-Level Heuristics

Tables 7 and 8 provide the empirical results in ranked format. The performance rank is calculated as the average rank produced by each heuristic, across all datasets, for all independent runs and all epochs. The average rank across all epochs produces a view on the performance of the heuristics as it relates to the entire training process. Finally, a normalised average rank is provided for the overall performance of all heuristics at the bottom of the table. The normalised average rank is calculated as a discrete normalisation of the average rank achieved across all datasets, for all independent runs and epochs.

Tables 7 and 8 show that the _bhh_gd_ configuration produced the best results of the BHH variants and managed to perform well, producing generally good results across all datasets. The _bhh_gd_ configuration managed to produce results that are comparable to the top three heuristics for each dataset, while the _bhh_all_ and _bhh_mh_ produced average results compared to all the heuristics.

The normalised average ranks provided in Tables 7 and 8 show that the _bhh_gd_ configuration ranked fourth, while the _bhh_al_ and _bhh_mh_ configurations ranked sixth and eighth amongst all thirteen heuristic implementations respectively. These results show that the BHH generally performs well, but is not able to outperform the best heuristic for each dataset.

Figure 3 provides an illustration showing a descriptive plot of the average ranks achieved over all independent runs, for each heuristic, per dataset. The heuristics are ordered according to the normalised ranks presented in Tables 7 and 8. Figure 3 shows that the _bhh_gd_ heuristic achieved the lowest variance in average rank across all datasets, compared to the other heuristics. The aforementioned shows the generalisation capabilities of the BHH to multiple problems.
Figure 4 provides an illustration of the overall critical difference plots that illustrate the statistically significant differences in ranked performance for each heuristic as it relates to all datasets, across all independent runs and epochs. Although the outcomes of the _bhh_al_ and _bhh_mh_ configurations seem to produce average performance results, it should be noted that the performance difference between all heuristics is very small. Furthermore, the best configuration of the BHH, namely the _bhh_gd_ configuration, is statisti

\begin{table}
\begin{tabular}{r|c c c|c|c|c}  & \multicolumn{5}{c||}{**BHH vs. Low-Level Heuristics - Average Rank (Part A)**} \\ \hline dataset & adagrad & adam & rmsprop & bhh_gd & mag & bhh_all \\ \hline abalone & 2.2215 (\(\pm\)15.91) & 2.2980 (\(\pm\)18.87) & 4.6172 (\(\pm\)2.65) & 4.7032 (\(\pm\)2.108) & 4.2731 (\(\pm\)1.542) & 5.9376 (\(\pm\)2.399) \\ air\_quality & 3.6109 (\(\pm\)2.259) & 5.4312 (\(\pm\)2.62) & 3.452 (\(\pm\)2.57) & 5.0817 (\(\pm\)2.762) & 3.8191 (\(\pm\)2.229) & 6.686 (\(\pm\)3.061) \\ bank & 2.5495 (\(\pm\)1.598) & 2.0796 (\(\pm\)1.587) & 3.4645 (\(\pm\)2.209) & 4.8828 (\(\pm\)1.702) & 4.2871 (\(\pm\)1.732) & 6.2419 (\(\pm\)2.157) \\ bike & **1.7204 (\(\pm\)1.384)** & 3.6925 (\(\pm\)4.003) & 4.2822 (\(\pm\)4.58) & 3.8441 (\(\pm\)4.398) & 6.4516 (\(\pm\)1.02) & 4.2151 (\(\pm\)4.361) \\ car & 4.7634 (\(\pm\)0.938) & 16.2228 (\(\pm\)1.405) & 2.2308 (\(\pm\)1.409) & 3.3432 (\(\pm\)1.35) & 6.0785 (\(\pm\)0.799) & 3.5624 (\(\pm\)3.185) \\ diabetic & **2.7966 (\(\pm\)1.569)** & 7.1484 (\(\pm\)2.227) & 6.7376 (\(\pm\)2.577) & 5.2269 (\(\pm\)2.186) & **1.8118 (\(\pm\)1.118)** & **9.3968 (\(\pm\)3.022) \\ fish\_toxicity & **4.2645 (\(\pm\)2.614)** & **3.0022 (\(\pm\)2.415)** & **3.3946 (\(\pm\)2.429)** & 5.4118 (\(\pm\)2.65) & 5.8914 (\(\pm\)2.629) & 5.829 (\(\pm\)2.856) \\ forest\_fires & 5.1559 (\(\pm\)2.922) & 4.2688 (\(\pm\)2.984) & 5.0355 (\(\pm\)3.143) & 4.6935 (\(\pm\)2.759) & 5.6882 (\(\pm\)2.215) & 5.4839 (\(\pm\)3.107) \\ housing & **3.4481 (\(\pm\)2.022)** & 3.3334 (\(\pm\)1.819) & 3.6946 (\(\pm\)2.166) & 4.4742 (\(\pm\)2.312) & 4.6839 (\(\pm\)2.658) & 4.3763 (\(\pm\)2.438) \\ iris & 6.3346 (\(\pm\)1.6) & 3.5839 (\(\pm\)2.511) & 6.2088 (\(\pm\)1.921) & 4.7473 (\(\pm\)2.275) & 3.5548 (\(\pm\)2.125) & 5.2204 (\(\pm\)1.301) \\ mushroom & 4.6565 (\(\pm\)1.053) & 2.1344 (\(\pm\)1.883) & 2.4656 (\(\pm\)1.530) & 3.8448 (\(\pm\)1.602) & 6.3323 (\(\pm\)0.891) & 3.6688 (\(\pm\)2.469) \\ parkinsons & 2.677 (\(\pm\)1.697) & 2.2333 (\(\pm\)1.74) & 3.5565 (\(\pm\)2.492) & 4.572 (\(\pm\)1.934) & 7.5355 (\(\pm\)1.44) & 4.3839 (\(\pm\)1.861) \\ student\_performance & 2.5634 (\(\pm\)1.912) & 11.3978 (\(\pm\)2.178) & 12.4312 (\(\pm\)1.344) & 5.6624 (\(\pm\)3.57) & 3.1895 (\(\pm\)2.12) & 5.8634 (\(\pm\)3.159) \\ wine\_quality & 3.2806 (\(\pm\)1.931) & 2.1118 (\(\pm\)1.666) & 3.6301 (\(\pm\)1.731) & 4.7882 (\(\pm\)2.105) & 4.1505 (\(\pm\)1.916) & 5.1925 (\(\pm\)1.951) \\ \hline avg rank & 3.5512 (\(\pm\)2.25) & 3.9314 (\(\pm\)3.423) & 4.5691 (\(\pm\)3.517) & 4.6346 (\(\pm\)2.364) & 4.8394 (\(\pm\)2.384) & 5.4327 (\(\pm\)2.9) \\ \hline normalised avg rank & \multicolumn{2}{c|}{1} & \multicolumn{2}{c|}{2} & \multicolumn{2}{c|}{3} & \multicolumn{2}{c|}{4} & \multicolumn{2}{c|}{5} & \multicolumn{2}{c|}{6} \\ \hline \end{tabular}
\end{table}
Table 7: Empirical results showing normalised average rank and statistics for the top six low-level heuristics and three heuristic pool variants of the BHH baseline configuration, across multiple datasets, for all independent runs and epochs.

\begin{table}
\begin{tabular}{r|c c|c c|c c c}  & \multicolumn{5}{c|}{BHH vs. Low-Level Heuristics - Average Rank (Part B)} \\ \hline dataset & adandelta & bhh\_mh & \multicolumn{1}{c}{gd} & \multicolumn{1}{c}{gd} & \multicolumn{1}{c}{gd} & \multicolumn{1}{c}{momentum} & \multicolumn{1}{c}{de} \\ \hline abalone & 5.3129 (\(\pm\)1.478) & 8.1882 (\(\pm\)1.95) & 11.1108 (\(\pm\)2.102) & 11.2599 (\(\pm\)1.820) & 8.628 (\(\pm\)1.019) & 9.8151 (\(\pm\)1.16) & 12.5706 (\(\pm\)1.29) \\ air\_quality & 5.2441 (\(\pm\)3.102) & 6.307 (\(\pm\)2.303) & 7.8204 (\(\pm\)2.260) & 9.9151 (\(\pm\)2.288) & 10.0410 (\(\pm\)1.090) & 11.5599 (\(\pm\)1.92) & 11.110 (\(\pm\)2.240) \\ bank & 5.672 (\(\pm\)1.214) & **1.9209** (\(\pm\)1.048) & **1.9691** (\(\pm\)1.216) & **1.8116** (\(\pm\)1.464) & 5.2774 (\(\pm\)1.03) & 8.4747 (\(\pm\)1.05) & 12.4696 (\(\pm\)1.294) \\ bike & 5.3092 (\(\pm\)1.155) & 4.7108 (\(\pm\)1.08) & 9.2203 (\(\pm\)1.18) & 9.3086 (\(\pm\)1.761) & 10.3035 (\(\pm\)1.419) & 10.7086 (\(\pm\)1.42) & 12.6414 (\(\pm\)1.46) \\ car & 7.7344 (\(\pm\)1.476) & 8.8305 (\(\pm\)1.413) & 10.7038 (\(\pm\)1.471) & 8.5613 (\(\pm\)1.622) & 10.2402 (\(\pm\)1.447) & 10.9226 (\(\pm\)1.310) & 12.408 (\(\pm\)1.92) \\ disable & **2.7811** (\(\pm\)1.692) & 8.507 (\(\pm\)1.471) & **1.0
cally outperformed overall by only Adagrad and Adam, yielding statistically comparable results to RMSProp and NAG. It should be noted that the standalone low-level heuristics already produce good results in general across all datasets. In this particular case, producing better performance outcomes can be hard to achieve. However, as mentioned previously, the BHH provides a generalisation capability across all datasets that is advantageous to the BHH.

Another observation that can be made is that the gradient-based heuristics generally performed much better than the MHs on all datasets. State of the art methods for training FFNNs, such as Adam, utilise gradient-based approaches that have been proven to work well on many occasions [8]. Exploration of the heuristic space leads the BHH to consider other heuristics during the training process, which could possibly result in worse performances at times. A suggestion to improve on these results is to include a move-acceptance strategy where heuristic progressions are discarded if they fail to produce better results.

Figure 3: Descriptive plots for the average ranks of all low-level heuristics compared to three heuristic pool variants of the BHH baseline configuration, per dataset, across all independent runs and epochs.
## 9 Conclusion

The research done in this study stems from the difficult and tedious process of selecting the best heuristic for training FFNNs. The research presented in this article identified the possibility of using a different approach, referred to as HHs, to automate the heuristic selection process.

This research set out to develop a novel high-level heuristic that utilises probability theory in an online learning setting to drive the automatic heuristic selection process.

For the experimental group that compares the BHH baseline with a number of low-level heuristics, it was found that the _bhh_gd_ configuration, which contains only gradient-based heuristics in the heuristic pool, performed the best out of the BHH variants, achieving an overall rank of fourth amongst thirteen heuristics that were implemented and executed on fourteen datasets. The _bhh_gd_ configuration produced performance results close to that of the best low-level heuristics and was statistically outperformed only by the top two low-level heuristics. The _bhh_all_ configuration, which contains only gradient-based heuristics and MHs in the heuristic pool, achieved an overall rank of sixth, and the _bhh_mh_ configuration, which contains only MHs in the heuristic pool, achieved an overall rank of eighth.

Although the _bhh_gd_ configuration produced performance results comparable to the best low-level heuristics, the _bhh_all_ and _bhh_mh_ configurations produced average results. It was found that, in general, gradient-based heuristics produced the best results, as such, it is understandable that the _bhh_gd_ yielded the best performance outcomes between the different BHH variants that were implemented. Although the BHH variants were not able

Figure 4: Critical difference plots for the average ranks of all low-level heuristics compared to three heuristic pool variants of the baseline BHH, across all datasets, runs and epochs.
to produce better results than the top low-level heuristics, the BHH variants still effectively trained the underlying FFNNs and produced good training outcomes overall. It was shown that the _bhh_gd_ configuration produced the lowest variance in rank between datasets out of all of the heuristics implemented, giving the BHH the ability to generalise well to other problems.

Finally, it was shown that the BHH provides a mechanism whereby prior expert knowledge can be injected, before training starts. Future research can exploit this knowledge and provide a significant bias towards heuristics that are known to perform well on particular problem types. Future research can also investigate the scalability and effectiveness of the BHH on other model architectures such as _deep neural networks_ (DNNs). Furthermore, the selection mechanism of the BHH can be extended to not just select heuristics from a heuristic pool, but also different model architectures from a model architecture pool.

## Appendix A Naive Bayes

This section aims to dissect the probabilistic model that is presented in Equation (1). The BHH implements a form of Naive Bayes classifier, and thus independence between events can be assumed. The following derived _probability mass functions_ (PMFs) are provided as fundamental building blocks to show the mechanism by which the BHH learns.

The independence between events for class label \(\mathbf{H}\), simply yields the PMF of the Multinomial distribution as presented below:

\[\begin{split} P(\mathbf{H}|\mathbf{\theta})&\propto \prod_{i=1}^{I}\prod_{k=1}^{K}P(h_{i,k}|\theta_{k})\\ &\propto\prod_{i=1}^{I}\prod_{k=1}^{K}\theta_{k}^{1_{1}(h_{i,k})} \\ &\propto\prod_{k=1}^{K}\theta_{k}^{\sum_{i=1}^{I}1_{1}(h_{i,k})} \\ &\propto\prod_{k=1}^{K}\theta_{k}^{N_{k}}\end{split} \tag{16}\]

where \(N_{k}\) is a summary variable such that \(N_{k}=\sum_{i=i}^{I}\mathbb{1}_{1}(h_{i,k})\), denoting
the count of occurrences of the event \(h_{i}\) taking on class \(k\) in \(I\) independent, identical runs.

The independence between events \(\mathbf{E}\), given class label \(\mathbf{H}\), is denoted by the likelihood of \(\mathbf{E}\), conditional to the occurrence of heuristic \(k\) and model parameter \(\mathbf{\phi}\) as follows:

\[\begin{split} P(\mathbf{E}|\mathbf{H};\mathbf{\phi})&\propto \prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}P(e_{i,j,k}|h_{i,k};\phi_{j,k})\\ &\propto\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\phi_{j,k}^{ \leavevmode\nobreak\ 1_{1}(e_{i,j,k})\leavevmode\nobreak\ 1_{1}(h_{i,k})}\\ &\propto\prod_{j=1}^{J}\prod_{k=1}^{K}\phi_{j,k}^{\leavevmode \nobreak\ 1_{i}\big{[}\leavevmode\nobreak\ 1_{1}(e_{i,j,k})\leavevmode\nobreak\ 1_{1}(h_{i,k})\big{]}}\\ &\propto\prod_{j=1}^{J}\prod_{k=1}^{K}\phi_{j,k}^{N_{j,k}}\end{split}\] (A.2)

where \(N_{j,k}\) is a summary variable such that \(N_{j,k}=\sum_{i=i}^{I}\mathbbm{1}_{1}(e_{i,j,k})\mathbbm{1}_{1}(h_{i,k})\), denoting the count of occurrences of the events \(e_{i}\) taking on class \(j\) and \(h_{i}\) taking on class \(k\), i.e. the count of occurrences of both entity \(j\) and heuristic \(k\) occurring together in \(I\) independent, identical runs.

Finally, the independence between events for the performance criteria \(\mathbf{C}\), given class label \(\mathbf{H}\), is denoted by the likelihood of \(\mathbf{C}\), conditional to the occurrence of heuristic \(k\) and model parameter \(\mathbf{\psi}\) as given below:
\[P(\mathbf{C}|\mathbf{H};\mathbf{\psi}) \propto\prod_{i=1}^{I}\prod_{k=1}^{K}P(c_{i,k}|h_{i,k};\psi_{k})\] (A.3) \[\propto\prod_{i=1}^{I}\prod_{k=1}^{K}\psi_{k}^{1_{1}(c_{i,k})1_{1} (h_{i,k})}(1-\psi_{k})^{1_{0}(c_{i,k})1_{1}(h_{i,k})}\] \[\propto\prod_{k=1}^{K}\psi_{k}^{\sum_{i=1}^{I}1_{1}(c_{i,k})1_{1} (h_{i,k})}(1-\psi_{k})^{\sum_{i=1}^{I}1_{0}(c_{i,k})1_{1}(h_{i,k})}\] \[\propto\prod_{k=1}^{K}\psi_{k}^{N_{1,k}}(1-\psi_{k})^{N_{0,k}}\] \[\propto\prod_{k=1}^{K}\psi_{k}^{N_{1,k}}(1-\psi_{k})^{(N_{k}-N_{1,k})}\]

where \(N_{k}\) is the same summary variable as described for Equation (A.1). \(N_{1,k}\) is a summary variable such that \(N_{1,k}=\sum_{i=1}^{I}\mathbbm{1}_{1}(c_{i,k})\mathbbm{1}_{1}(h_{i,k})\), denoting the count of occurrences of the events \(c_{i}\) taking on a success (i.e. \(c_{i}=1\)) and \(h_{i}\) taking on class \(k\), i.e. the count of occurrences of both succeeding in the performance criteria and heuristic \(k\) occurring together in \(I\) independent, identical runs. Similarly, \(N_{0,k}=N_{k}-N_{1,k}\) denotes the count of occurrences of the events \(c_{i}\) taking on a failure (i.e. \(c_{i}=0\)) and \(h_{i}\) taking on class \(k\).

Equations (A.1), (A.2) and (A.3) can be substituted into the proportional evaluation of the predictive model as given in Equation (1), resulting in

\[P(\mathbf{H}|\mathbf{E},\mathbf{C};\mathbf{\theta},\mathbf{\phi},\mathbf{\psi}) \propto P(\mathbf{E}|\mathbf{H};\mathbf{\phi})P(\mathbf{C}|\mathbf{H};\mathbf{\psi})P(\bm {H}|\mathbf{\theta})\] (A.4) \[\propto\left[\prod_{j=1}^{J}\prod_{k=1}^{K}\phi_{j,k}^{N_{j,k}} \right]\left[\prod_{k=1}^{K}\psi_{k}^{N_{1,k}}(1-\psi_{k})^{(N_{k}-N_{1,k})} \right]\left[\prod_{k=1}^{K}\theta_{k}^{N_{k}}\right]\]

Consider the practical implementation of the predictive model as shown in Equation (A.4). Computationally, the equation presented in Equation (A.4) will underflow on a real computer if the resulting probabilities are very small.

### Numerical Stability

When Equation (A.4) is evaluated, the numerical stability is shown to underflow if the resulting probabilities from its parts are very small. Multi
plication of multiple fractional parameters leads to an even smaller fractional number. Probabilities might be very low at some points during training. A solution to the aforementioned problem is to apply the _log-sum-exp_ trick. The transformation of Equation (A.4) using the log-sum-exp trick is given as

\[LSE(P(h_{k}|e_{j},c_{1};\mathbf{\theta},\mathbf{\phi},\mathbf{\psi}))=\ln(\exp(\phi_{j,k})+ \exp(\psi_{k})+\exp(\theta_{k}))\] (A.5)
