From Pointwise to Powerhouse: Initialising Neural Networks with
  Generative Models
Traditional initialisation methods, e.g. He and Xavier, have been effective
in avoiding the problem of vanishing or exploding gradients in neural networks.
However, they only use simple pointwise distributions, which model
one-dimensional variables. Moreover, they ignore most information about the
architecture and disregard past training experiences. These limitations can be
overcome by employing generative models for initialisation. In this paper, we
introduce two groups of new initialisation methods. First, we locally
initialise weight groups by employing variational autoencoders. Secondly, we
globally initialise full weight sets by employing graph hypernetworks. We
thoroughly evaluate the impact of the employed generative models on
state-of-the-art neural networks in terms of accuracy, convergence speed and
ensembling. Our results show that global initialisations result in higher
accuracy and faster initial convergence speed. However, the implementation
through graph hypernetworks leads to diminished ensemble performance on out of
distribution data. To counteract, we propose a modification called noise graph
hypernetwork, which encourages diversity in the produced ensemble members.
Furthermore, our approach might be able to transfer learned knowledge to
different image distributions. Our work provides insights into the potential,
the trade-offs and possible modifications of these new initialisation methods.
Christian Harder, Moritz Fuchs, Yuri Tolkach, Anirban Mukhopadhyay
# From Pointwise to Powerhouse: Initialising Neural Networks with Generative Models

###### Abstract

Traditional initialisation methods, e.g. He and Xavier, have been effective in avoiding the problem of vanishing or exploding gradients in neural networks. However, they only use simple pointwise distributions, which model one-dimensional variables. Moreover, they ignore most information about the architecture and disregard past training experiences. These limitations can be overcome by employing generative models for initialisation.

In this paper, we introduce two groups of new initialisation methods. First, we _locally_ initialise weight groups by employing variational autoencoders. Secondly, we _globally_ initialise full weight sets by employing graph hypernetworks. We thoroughly evaluate the impact of the employed generative models on state-of-the-art neural networks in terms of accuracy, convergence speed and ensembling. Our results show that _global_ initialisations result in higher accuracy and faster initial convergence speed. However, the implementation through graph hypernetworks leads to diminished ensemble performance on out of distribution data. To counteract, we propose a modification called _noise graph hypernetwork_, which encourages diversity in the produced ensemble members. Furthermore, our approach might be able to transfer learned knowledge to different image distributions. Our work provides insights into the potential, the trade-offs and possible modifications of these new initialisation methods.

## 1 Introduction

Neural networks have shown remarkable success in various computer vision tasks, such as image classification [5, 15, 35, 47], segmentation [4, 18, 41], and detection [10, 34]. Their performance depends on several factors, including their architecture, quality of data and available computing resources. Among these factors, the **choice of weight initialisation** plays a crucial role in the network's performance [43]. A proper initialisation greatly enhances training convergence, whereas a poor one can hinder it [15]. These findings have sparked entire fields of research dedicated to exploring innovative approaches. Techniques such as self-supervised learning [19], knowledge distillation [12], and transfer learning
[44] have emerged as notable approaches. In contrast, our work focuses on initialisations without training during the weight generation.

Traditional initialisation methods such as He [15] and Xavier [11] initialisation were designed to avoid the problem of vanishing or exploding gradients. Despite their effectiveness, these approaches have **two significant weaknesses**. Firstly, they rely on simple pointwise distributions, which model one-dimensional variables. Doing so overlooks the direct connections between neural network weights and results in suboptimal initialisations. Secondly, they disregard important architectural information and knowledge from past trainings of similar architectures. This practice amplifies the already substantial financial and environmental costs associated with training networks on large datasets [38].

In Bayesian Neural Networks (BNN) [2, 9, 13, 20, 24, 37, 48], where neural networks are combined with stochastic modelling, researchers assume arbitrary Gaussian distributions over the weights [39]. However, the deep-weight-prior [1] stands out as an exception. It employs generative models to learn a _local_ distribution of trained weights, which is then utilised for initialising small BNNs. Their findings indicate that leveraging generative models for neural network initialisation leads to improved convergence.

Building upon the concept of leveraging generative models for initialisation, we present **two groups of new initialisation methods**:_local_ and _global_ initialisations address the limitations of traditional methods. They employ complex weight distributions, incorporate comprehensive architectural information, and leverage past training experiences. While they share the common goal of improving initialisation, they differ in the scope of initialisation, the architectural information considered, and the specific generative models used to learn weight distributions.

Figure 1: Traditional initialisation methods only consider layer dimensions, while being simple pointwise distributions. Utilising generative models, we consider significantly more architecture information and model complete weight sets by employing graph hypernetworks (GHN).
With _local_** initialisations**, we sample small groups of weights from learned distributions using variational autoencoders (VAE) [21, 23, 45]. In contrast, _global_**initialisations** obtain full weight sets conditioned on the network's architecture, accomplished through graph hypernetworks (GHN) [25].

We evaluate these methods on state-of-the-art deep convolutional neural networks (CNN), focusing on convergence speed, accuracy and ensembling. Our findings reveal that _global_ initialisations achieve **faster initial convergence and higher accuracy**. However, these benefits come with a potential trade-off: **reduced generalisation ability** of ensembles.

To deepen our understanding of this trade-off, we explore the diversity of the different initialisation methods. We identify missing diversity in GHN initialisations as the cause for its reduced generalisation ability. Motivated by this, **we propose a modification called the Noise GHN**, which introduces diversity into GHNs by modelling a non-deterministic distribution. This is achieved by injecting noise into the GHN decoder and using a modified loss function to encourage the production of diverse weight sets. The noise injection also increases model robustness, enabling better learning of essential features and preventing overfitting. Consequently, the Noise GHN might be able to transfer learned knowledge to different image distributions.

## 2 Related Work and Preliminaries

Variational autoencoders and graph hypernetworks are the backbone of our approach. We offer a concise overview of these frameworks, highlighting their essential aspects and functionalities relevant to our work.

#### 2.0.1 Variational Autoencoders

VAEs are generative neural networks designed to learn and capture the underlying distribution of a given dataset. Once trained, these networks can be effectively utilised to generate diverse forms of data, such as images [42, 45] or natural language [7, 33].

It consists of two main components: an encoder model \(Q_{\boldsymbol{\Phi}}(\mathbf{Z}|\mathbf{X})\) and a decoder model \(P_{\boldsymbol{\theta}}(\mathbf{X}|\mathbf{Z})\). The encoder maps input data to a latent space, while the decoder model reconstructs the input data from the latent space. VAEs are optimised by maximising a lower bound on the log-likelihood of the data. This lower bound is called evidence lower bound (ELBO). Doing so, they learn a meaningful latent space representation that captures the underlying data structure.

For a given data point \(x\), a prior \(P(\mathbf{Z})\) on the latent variables \(\mathbf{Z}\) and the parameters \(\boldsymbol{\Phi}\) and \(\boldsymbol{\theta}\) of the VAE, the ELBO can be expressed as:

\[\mathcal{L}_{\boldsymbol{\theta},\boldsymbol{\Phi}}(\mathbf{x})=\mathbb{E}_{q \boldsymbol{\varphi}(\mathbf{z}|\mathbf{x})}\left[\log p_{\boldsymbol{\theta}} (\mathbf{x}|\mathbf{z})\right]-\mathrm{KL}[Q_{\boldsymbol{\Phi}}(\mathbf{Z}| \mathbf{x})||P(\mathbf{Z})]. \tag{1}\]

The VAE's optimisation is driven by the expected value of the log density. It promotes the generation of outputs similar to the input. Simultaneously, the KL Divergence between the encoder model and the prior acts as a regularisation
term for the latent space. By default, we assume simple Gaussian distributions for the prior, encoder, and decoder models conditioned on the input. This choice offers the advantage of closed-form KL Divergence computation for equation 1, without significantly limiting the model's expressive capacity.

Introducing a discrete latent space can encourage the model to learn a more structured representation, as it has to navigate the limited capacity of such a space. To achieve this, Vector Quantisation (VQ) is employed. VAEs utilising VQ are called Vector Quantised Variational Autoencoders (VQVAE) [45]. There, continuous outputs from the encoder network are mapped to discrete points in the latent space using a codebook. To enhance convergence, a codebook loss is incorporated. The authors also suggest employing an autoregressive model called PixelCNN [29] alongside the VQVAE, which we denote as VQVAE*.

Although there are various VAE variations based on different encoder and decoder assumptions [3, 21, 27, 40], this work does not delve into exploring them. We study the VQVAE's ability to initialise neural networks.

#### 2.0.1 Graph HyperNetworks

The GHN [25, 49] is a generative model that produces complete sets of network weights. Unlike VAEs, which are limited to fixed-size data, a single GHN can be used with various network architectures. The resulting weights are immediately effective, achieving an impressive 58.6% accuracy on CIFAR-10 for a ResNet-50 architecture that it has never seen before. The GHN combines two key concepts: the hypernetwork [14] and the graph network [36].

A hypernetwork is a neural network that takes input and produces weights for another neural network. The inputs can include various information such as network architecture or data points. The hypernetwork is trained by directly backpropagating the loss of the predicted parameters into the hypernetwork.

The second key component is the graph network, which is specifically designed to process graph-based inputs with varying shapes. During the forward pass, a graph network updates the states of the nodes by propagating information along the edges of the graph. By representing network architectures as a computational graphs, the GHN leverages the graph network to obtain states for each node. These states are then used to generate weights for each layer by feeding them into the hypernetwork. Finally, the produced weights are normalised and adjusted to match the dimensions of each layer using slicing and tiling techniques.

To optimise the GHN \(H_{\mathbf{\theta}}\) with parameters \(\mathbf{\theta}\), we employ mini-batch optimisation over two sets of data: batches \(b\) of images and batches \(b_{m}\) of architectures:

\[\mathcal{L}=\sum_{i=1}^{b}\sum_{j=1}^{b_{m}}L\left(f\left(\mathbf{x}_{i},a_{j},H_{\mathbf{\theta}}(a_{j})\right),\mathbf{y}_{i}\right), \tag{2}\]

where \(f^{(k)}(\mathbf{x},a,\mathbf{w})\) represents the forward-pass of input \(\mathbf{x}\) through the network architecture \(a\) with weights \(\mathbf{w}\). The function \(L\) denotes the loss function used in the optimisation process.

Our approach differs from the authors [25] work, as they focus on generating deterministic weights with high performance in a single forward pass.
## 3 Methods

VAEs and GHNs capture expressive distributions, enabling us to leverage them for initialising state-of-the-art convolutional classification networks. Their key strength lies in their ability to incorporate architecture information and leverage knowledge gained from past trainings of similar architectures. We now explain further how we utilise these generative models.

### Local Initialisations

Our focus on _local_ initialisations revolves around capturing patterns within the weights of CNNs. As convolutional networks progress, they transition from extracting edges and colour blobs in early layers to capturing higher-level features in later layers [32]. Motivated by the specific attributes of filters like edge detectors, we assume that the weights within a CNN layer follow an unknown distribution.

Given a neural network architecture, we train a set of variational autoencoders, one for each layer, to learn the unknown local distributions. Once trained, these generative models enable the production of weights that adhere to the learned distributions.

We evaluate our _local_ initialisations on the well known architecture of the ResNet-20. For every layer, we learn the distribution of the \(3\times 3\) weight slices that constitute the convolutional kernels. To this end, we separately train 100 ResNet-20s on the training datasets. Afterwards we take the parameters that perform best on the corresponding validation sets. As advised in [1], we remove the slices with a low \(l_{2}\), by disregarding the slices whose \(l_{2}\) is in the lowest 5% for every layer. We call these datasets of weight-slices the _Weight-Datasets_. Finally, we train VAEs and VQVAEs on it to capture the underlying distributions.

### The Global Approach

We introduce the Noise GHN to transfer the ability to produce instantly performing network weights to ensembles. Furthermore we adapt the training routine of the GHNs to reduce training resources.

#### 3.2.1 Reducing GHN Training Resources

The DEEPNETS1M [25] dataset trains the GHN, featuring a million network architectures and validation/test sets of 500/500 architectures. Every iteration samples 64 images and eight architectures. This training setup is conducted over 300 epochs of the image dataset, which exceeds our computational resources.

To conserve resources, we reduce the number of training architectures from eight to three and train for only 30 instead of 300 image dataset epochs. To ensure the effectiveness of training, we select training architectures similar to the evaluation architecture of a ResNet-20. Specifically, we choose a ResNet-32, a ResNet-44, and a ResNet-56 and use all three architectures in every iteration.
To compensate for the reduced amount of training and training network diversity, we initialise the graph networks of the GHNs by using already trained weights, which are provided by Knyazev [25]. As we modify the hypernetwork for the Noise GHN, we initialise all GHN hypernetworks from scratch in order to ensure a fair comparison.

#### 3.0.1 Modified GHN

Since the GHN is a deterministic model when given a fixed architecture, its employment results in more similar weights in the trained ensemble members. As ensembles benefit from the diversity of their members, this lack of diversity poses a potential problem.

To achieve different weights in every forward pass, we integrate noise into the GHN, as shown in Figure 2. Specifically, we sample and append a noise vector to every final hidden state of the graph network. This design allows the hypernetwork to generate varied outputs, while learning to produce effective weights based on architectures encodings.

Additionally, we encourage diversity in the loss function by including a similarity loss into the overall loss function. To measure similarity, we first perform two forward passes of the Noise GHN for the same architecture. Then we calculate the similarity of predictions on a batch of images. Doing so, we ensure that the two produced weight sets correspond to two different functions.

Given a Noise GHN \(H_{\mathbf{\theta}}\), with parameters \(\mathbf{\theta}\), a network architecture \(a\), a sample \((\mathbf{x},\mathbf{y})\) from the dataset \(D=\{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{N}\) and samples \(\xi_{1},\xi_{2}\) from a noise distribution, the similarity loss \(\mathcal{L}_{S}\) is calculated as:

\[\mathcal{L}_{S}\big{(}\mathbf{x},a,H_{\mathbf{\theta}},\xi_{1},\xi_{2}\big{)}= \text{CoSim}\Big{(}f^{(1)}\big{(}\mathbf{x},a,H_{\mathbf{\theta}}(a,\xi_{1}),f^{(2 )}\big{(}\mathbf{x},a,H_{\mathbf{\theta}}(a,\xi_{2})\big{)}\Big{)} \tag{3}\]

Figure 2: Functioning principle of the Noise GHN. The input, a network architecture, is expressed by a computational graph together with initial hidden states \(H^{0}\) for every node. The graph network propagates information through the computational graph resulting in final hidden states \(H^{T}\), encoding the function of each node. Every final hidden state \(h_{i}^{T}\) is then fed into a hypernetwork, together with a sampled noise vector. The hypernetwork is encouraged by the loss function to produce performant and diverse weights for every node. In a final step, the produced weights are normalised and fitted to the layer dimensions.
where \(f^{(k)}(\mathbf{x},a,\mathbf{w})\) represents the \(k\)-th forward-pass with input \(\mathbf{x}\) into the network architecture \(a\) with weights \(\mathbf{w}\), \(L\) some loss function and CosSim the _Cosine similarity_.

The overall loss function \(\mathcal{L}\) for a mini-batch \(b\) of images and the three training architectures can then be expressed as:

\[\mathcal{L}=\sum_{i=1}^{b}\sum_{j=1}^{3}\Bigg{[}\sum_{k=1}^{2}\Bigg{[}L\Big{(} f^{(k)}\big{(}\mathbf{x}_{i},a_{j},H_{\boldsymbol{\theta}}(a_{j},\xi_{k})\big{)}, \mathbf{y}_{i}\Big{)}\bigg{]}+\mathcal{L}_{S}\big{(}\mathbf{x}_{i},a_{j},H_{ \boldsymbol{\theta}},\xi_{1},\xi_{2}\big{)}\Bigg{]}. \tag{4}\]

The conceptual differences between the _local_ and the _global_ initialisations overall can be explained in two dimensions, as seen in Figure 1. First, _global_ initialisations utilise a distribution which enables initialisation of all weights in a meaningful connected way - as opposed to initialising all weights independently from another. As the GHN is trained to produce already working weights, the weights are already synchronised. On the contrary, the _local_ initialisations focus on small weight groups, who are initialised independently from another. Secondly, the _global_ initialisations condition the weights on the whole architecture, while the _local_ initialisations confine themselves to information about layer position.

## 4 Experiments and Results

We evaluate various network initialisations on CIFAR-10 [26] and the medical PatchCamelyon [46] (PCam) dataset to assess accuracy and convergence. Additionally, we evaluate ensemble accuracy and calibration on the out-of-distribution (OOD) CIFAR-C [17] dataset. Finally, we investigate the Noise GHN's generalisation ability from natural images to the medical domain. For more detailed information on the datasets and experimental setups, please refer to the supplementary material.

#### 4.0.1 Convergence Speed

We study convergence speed resulting from different initialisations by evaluating every 300 batches on PCam and every epoch on the CIFAR-10 dataset. Results are averaged over 25 trainings per initialisation. Comparing the convergence speed, we measure the steps to reach specified accuracy thresholds. PCam's thresholds are 0.80 and 0.85, while CIFAR-10's are 0.65 and 0.75. The visualisations can be found in Figure 3, while more detailed results, along with an explanation of our threshold selection, are available in Table 1 and Figure 1 in the supplementary material.

Our _global_ initialisation consistently outperforms other methods in early training. On the PCam dataset, GHN-initialised ResNet-20s reach the first threshold after the initial evaluation step, while other initialisations require five or more steps. The GHN initialisation consistently outperforms others, achieving the second thresholds almost twice as fast.

**The _global_ scope** for information and initialisation enhances the GHN's initial convergence, reaching thresholds **up to 5 times faster**. However, there is no
noticeable difference between pointwise standard initialisations and _local_ initialisations. Information about the target layer and initialising small weight groups shows no training improvement.

Since the learning rate scheduling is the same for all initialisations, we cannot assess how the accelerated initial convergence affects the overall training time. Despite not reaching state-of-the-art accuracies during training due to our training routine, this experiment yields promising results. Therefore, further research in this direction holds great potential.

#### 4.2.2 Accuracy

We assess test set accuracies of ResNet-20s with different initialisations, training 25 networks for each. Figure 4 shows the resulting accuracies. The _Global_ initialisation on PCam outperforms others, with **all GHN initialisations** achieving **higher accuracy**. On CIFAR-10, GHN initialisation achieves a higher median accuracy but by a smaller margin. However, no notable differences appear between traditional and _local_ initialisations.

The _global_ scope for information and initialisation leads to faster convergence and higher test set accuracy. Conversely, there are no significant performance differences between _local_ and standard initialisations. The payoff doesn't increase proportionally with expanded scope and information. This can be explained by a key advantage of our global GHN initialisation, which produces instantly performing and synchronised weight sets. The importance of this **synchronicity for fast initial convergence and higher accuracy** is a crucial insight, highlighting the potential benefits of employing generative models for initialisations.

#### 4.2.3 OOD Ensembling

Accurate initialisations and fast convergence are crucial in the context of ensembling. Ensembles provide higher accuracy and improved uncertainty estimation. These advantages are particularly important for OOD data. To examine the effect of different initialisations on ensembles, we evaluate their expected calibration error (ECE) and accuracy. Note that a lower ECE

Figure 3: Trajectories of validation accuracy during the initial training phase. GHN initialisations outperform other initialisation with a faster initial convergence.
indicates better calibration. We provide a detailed explanation of its calculation in the supplemental material.

The ensembles consist of the 25 trained ResNet-20 networks per initialisation. We sample 20 ensembles with 5 members each and calculate the ECE across all 5 corruption levels of the CIFAR-C dataset. We present the results in Figure 5. The corresponding accuracy results can be found in Figure 2 in the supplementary material, showing similar trends.

The VAE and VQVAE* based initialisations yield the lowest ECE values. Surprisingly, the _global_ GHN initialisation shows the highest ECE value, despite its faster convergence and higher accuracy. High accuracy does not necessarily guarantee good OOD performance. Nevertheless, the magnitude of the performance shift is surprising. This discrepancy is further investigated in the next section.

#### 4.2.2 Prediction Similarities

Ensemble uncertainty estimation and accuracy rely on member diversity [8]. We analyse this diversity by examining prediction similarity and cosine similarity of logits on the CIFAR-10 test set. Figure 6 illustrates pairwise similarities, and average scores, computed over the strict upper diagonal matrix. Lower similarity scores are preferable, as they indicate higher diversity.

**Standard** and _local_ initialisations exhibit similar diversities in both measures, while **GHN** initialisations show significantly lower diversity. The GHN produces the same set of weights for a fixed architecture, which results in more similar trained weights, reducing diversity.

To address this, our Noise GHN introduces diversification. We input a sampled vector into the Noise GHN decoder, and the modified loss function encourages low cosine similarity logits, ensuring diverse weights in each forward pass.

The effect is evident on the right-hand side of Figure 6. The Noise GHN initialisation yields the lowest cosine similarity and slightly improves on the GHN in

Figure 4: Boxplots displaying the resulting test set accuracies of 25 ResNet-20s per initialisation. The GHN results the highest median accuracy on both datasets, especially on the PCam dataset where all 25 GHN initialisations outperform every other initialisation.
prediction similarity. Notably, only the Noise GHN initialisation achieves cosine similarity scores below 0.8.

The Noise GHN significantly enhances diversity, leading to a clear impact on the OOD ECE, as shown in Figure 7. It consistently outperforms the GHN across all five corruption levels.

Furthermore, the Noise GHN retains the advantages of global initialisation, surpassing the convergence speed achieved by GHN initialisation, as seen in Table 1 in the supplementary material. This improvement is due to the sampled noise, which enhances weight robustness, akin to techniques like Monte-Carlo dropout.

#### 4.2.2 Knowledge Transfer

We explore the ability of our methods to transfer knowledge between image distributions, particularly between CIFAR-10's natural images and PCam's medical data. Such knowledge transfer can be especially beneficial for low-data tasks. To this end, we consider a dataset of 1000 training patches from PCam, with additional 400 validation patches for model selection and evaluate on a test dataset of a separate 10.000 patches. We call this downscaled dataset PCam Small. All splits underlie a strict 50%/50% split of tumorous and benign patches. After initialisation the ResNet-20s are trained for 40 epochs using our standard procedure. To account for the different dataset sizes, we adapt the learning rate schedule: After 20 epochs, the learning rate is halved, and after 30 epochs, the learning rate is reduced by a factor of five.

We compare the results to two types of baselines: He initialisation of a ResNet-20 and pretraining on CIFAR-10 for different amounts of epochs. We compare the performance of these baselines to a GHN and Noise GHN, who have been trained on the CIFAR-10 dataset.

Figure 5: Boxplots of the resulting ECE of 20 ensembles consisting of each five members on the OOD CIFAR-C dataset.
Figure 7 displays the results, showing overall weaker performance compared to the training on PCam due to the smaller dataset and distribution shift. The Noise GHN outperforms other initialisations, supported by a significant t-test result (p\(<\) 0.0001) against all other initialisations. This result can be attributed to the noise injection, making it more robust and preventing overfitting. This experiment indicates Noise GHN's potential for knowledge transfer between different image distributions. For a comparison of all initialisation methods in this work, please refer to Figure 3 in the supplementary material.

Our results align with Knyazev's findings [25], showing GHN parameters can be fine-tuned on small, different datasets. However, their experiments include a less severe distribution shift. Our experiment's weaker GHN performance might be attributed to the changes we made in the training setup. Nevertheless, further research is required to fully understand the reasons behind it.

#### 4.2.2 Limitations

Our work is focused on evaluating different generative models with regard to their potential for the initialisation of networks. As a first step, we evaluate these new initialisation methods on in-distribution data. In comparison to the training of a single network, this setup of training and employing on the same dataset is very costly. However, the potential savings when applying our methods to various datasets would justify these costs.

Due to the immense computational power required for the training of the GHN by Knyazev [25], we downscaled the training setup. Thus, a meaningful comparison to the original network is not possible.

We train and evaluate our methods on two in-distribution datasets, one OOD dataset and one architecture. Additional experiments are needed to verify, if the advantages extend to different datasets and network architectures.

Currently, the Noise GHN falls short of the CIFAR-C ensemble performance

Figure 6: Pairwise similarities of trained weights for different initialisations. Lower similarity score (dark blue) is better, as it implies a higher diversity. The GHN obtains the highest similarity measures, indicating missing diversity. The Noise GHN performs better than the GHN in terms of both similarity measures and all other initialisations in terms of their cosine similarity.
of _local_ and standard initialisations. Nevertheless, we believe future large-scale experiments can further unlock its potential.

## 5 Conclusion

We explore incorporating additional knowledge into initialisations. Our study demonstrates that initialisations based on learned weight distributions with a _global_** scope can offer significant advantages**. We show these advantages in terms of convergence speed and accuracy for feed-forward convolutional neural networks. We identify the key factor for the benefits of the global initialisations to be the **synchronicity of the produced weights**. In contrast, _local_ initialisations are missing synchronous produced weights.

This work also reveals that deterministic _global_ initialisations result in weaker OOD ensembling accuracy and calibration. However, the **introduction of the Noise GHN improves diversity and performance**. Further generalisation of our approach is especially intriguing due to the potential environmental and economic cost savings. We provide the groundwork for utilising _global_** initialisations in the area of ensembling** and show that the Noise GHN might be able to transfer learned knowledge to different image distributions. We showcase their strengths on single networks and introduce non-deterministic global initialisations.

Figure 7: Comparison of the GHN and Noise GHN. Left: Due to the diversity of the Noise GHN it improves upon the GHN over all levels of corruption. Right: The Noise GHN is able to transfer relevant knowledge from CIFAR-10 to the PCam Small dataset and outperform all other initialisations. We abbreviate pretraining on CIFAR-10 for five epochs as ”PT 5” and similarly for other epochs.