# Aligned Weight Regularizers for Pruning Pretrained Neural Networks

James O’ Neill  Sourav Dutta  Haytham Assem
Huawei Research Center, Dublin, Ireland
<EMAIL_ADDRESS>

###### Abstract

While various avenues of research have been explored for iterative pruning,
little is known what effect pruning has on zero-shot test performance and its
potential implications on the choice of pruning criteria. This pruning setup
is particularly important for cross-lingual models that implicitly learn
alignment between language representations during pretraining, which if
distorted via pruning, not only leads to poorer performance on language data
used for retraining but also on zero-shot languages that are evaluated. In
this work, we show that there is a clear performance discrepancy in magnitude-
based pruning when comparing standard supervised learning to the zero-shot
setting. From this finding, we propose two weight regularizers that aim to
maximize the alignment between units of pruned and unpruned networks to
mitigate alignment distortion in pruned cross-lingual models and perform well
for both non zero-shot and zero-shot settings. We provide experimental results
on cross-lingual tasks for the zero-shot setting using XLM-RoBERTaBase, where
we also find that pruning has varying degrees of representational degradation
depending on the language corresponding to the zero-shot test set. This is
also the first study that focuses on cross-lingual language model compression.

##  1 Introduction

Deep neural networks (DNNs) have grown increasingly large in the recent years.
This has led to models requiring more storage requirements, more resources for
training and inference (e.g., GPUs and TPUs), longer compute times and larger
carbon footprints. This is largely due to the rise of masked self-supervised
learning (SSL) which trains DNNs (e.g., Transformers in NLP) on a large
collection of samples that do not have task labels but instead use a subset of
the inputs as labels. Given the aforementioned challenges, it has become more
difficult for machine learning practitioners to use these SSL pretrained
models for fine-tuning on downstream tasks. While training tricks such as
effective batch sizes, gradient accumulation and dynamic learning rate
schedules Howard and Ruder (2018) have improved the efficiency of fine-tuning
DNNs under resource constraints, it can still come at a cost, e.g. gradient
accumulation leads to less updates.

Pruning LeCun et al. (1990); Reed (1993) is a type of model compression method
Buciluǎ et al. (2006) that aims to address these shortcomings by zeroing out a
subset of weights in the DNN, while maintaining performance close to the
original model. Retraining is often carried out directly after each pruning
step to recover from pruning induced performance drops. This process is
referred to as _iterative pruning_. Although, iterative pruning has been
extensively studied in the SSL setting Hassibi and Stork (1993); Han et al.
(2016); Ding et al. (2018) and the transfer learning setting Molchanov et al.
(2016); Gordon et al. (2020); Sanh et al. (2020), little is known about
pruning DNNs in the zero-shot setting111Here, zero or one-shot is the
conventional usage of the meaning (i.e., number of samples per class), not
one-shot pruning Lee et al. (2018) which is the number of pruning steps used
during retraining. where a model is required to make predictions on a set of
samples from classes that are unobserved during training. One salient example
is pretrained cross-lingual language models (XLMs) Conneau and Lample (2019);
Conneau et al. (2020a) whereby the model is trained with a masked/translation
language model (MLM/TLM) objective to predict tokens for a large set of
different languages whereby the objective forces the XLM model to learn
similar representations for different languages. After cross-lingual
pretraining, the model is further fine-tuned to a downstream task in one
language (e.g., English) and then evaluated on different languages in the
zero-shot setting (e.g., Spanish, French, Chinese, etc.). In this context,
applying current pruning methods can damage the XLM cross-lingual alignment
that has been learned during pretraining. Ideally, we would aim to prune XLMs
in such a way that avoids this alignment distortion which effects the zero-
shot performance of pruned XLMs. Additionally, overfitting to the language
used for fine-tuning becomes more of an issue due to the progressive reduction
in parameters throughout iterative pruning as the remaining weights are
relatively large, moving away from an “aligned” XLM state.

This is an important problem to address as the application of large pretrained
models in the zero shot-setting for natural language and other modalities (e.g
images and audio) is of practical importance e.g., using XLMs in production
for multiple languages by only requiring annotations in a single language for
fine-tuning, making predictions on unseen classes at test time from pretrained
visual representations Bucher et al. (2017) using only semantic descriptions
(i.e., label similarity to known classes) or zero-shot predictions in
pretrained multi-modal models such as CLIP Radford et al. (2021).

Hence, this work addresses the _alignment distortion_ pruning problem by
introducing AlignReg, a class of weight regularizers for magnitude-based
pruning that force pruned models to have parameters that point in a similar
direction or have a similar distribution to the parameters of the original
pretrained network. To our knowledge, this is the first study on how
iteratively pruned models perform in the zero-shot setting and how the
solution differs from solutions found in the non-zero shot setting. We believe
our findings have a strong practical implication as well-established pruning
criteria may not be suitable given the observed discrepancy between zero-shot
performance and the typically reported non-zero shot performance. Moreover,
our proposed weight regularizer improves overall pruning generalization in
zero-shot cross-lingual transfer. Below, we summarize our contributions.

  * •

The first analysis of pruning cross-lingual models, how this effects zero-shot
cross-lingual transfer and performance differences to pruning in the SSL
setup.

  * •

A weight regularizer that mitigates alignment distortion by minimizing the
layer-wise Frobenius norm or unit similarity between the pruned model and
unpruned model, avoiding overfitting to single language task fine-tuning.

  * •

A post-analysis of weight distributions after pruning and how they differ
across module types in Transformers.

##  2 Related Work

Below we describe regularization-based pruning, other non-magnitude based
pruning and how masked language modeling (MLM) implicitly learns to align
cross-lingual representations.

Regularization-based pruning. Pruning can be achieved by using a weight
regularizer that encourages network sparsity. Three well-established
regularizers are $L_{0}$ Louizos et al. (2018), $L_{1}$ regularization Liu et
al. (2017); Ye et al. (2018) and the commonly used $L_{2}$ regularization for
weight sparsity Han et al. (2015, 2016). Wang et al. have proposed an $L_{2}$
regularizer that increases in influence throughout retraining and shows the
increasing regularization improves pruning performance. For structured pruning
where whole blocks of weights are removed, Group-wise Brain Damage Lebedev and
Lempitsky (2016) and SSL Wen et al. (2016) propose to use Group LASSO Yuan and
Lin (2006) to learn structured solutions.

Importance-based pruning. Magnitude-based pruning (MBP) relies on the
assumption that weight or gradient magnitudes have correlation with its
importance to the overall output of the network. Mozer and Smolensky instead
use a learnable gating mechanism that approximates layer importance, finding
that weight magnitudes reflect importance statistics. To measure weight
importance as the difference in loss between pruned and unpruned network,
LeCun et al. approximate this difference with a Taylor series up to the second
order. This involves the product of the gradient and weight magnitude in the
1st term and an approximation of the Hessian and the square of the weight
magnitude for the second term. However, computing the Hessian and even its
approximations LeCun et al. (1990); Hassibi and Stork (1993); Dong et al.
(2017); Wang et al. (2019); Singh and Alistarh (2020) can significantly slow
down retraining. In our work, we avoid the requirement of computing the
Hessian or approximations thereof, as it is not scalable for models such as
XLM-R Conneau et al. (2020a). Park et al. have extended MBP to block
approximations to avoid pruning lowest weight magnitudes that may be connected
to weights in adjacent layers that have high weight magnitude. Lee et al. have
provided a method to automatically choose the sparsity of layers by using the
rescaled version of weight magnitude to incorporates the model-level
distortion incurred by pruning.

#### Implicit Alignment in Pretrained MLMs

In context of multi-task learning, Chen et al. (2020) minimize the mean
squared error between pretrained weights and weights being learned for a set
of different source tasks to avoid catastrophic forgetting in the continual
learning setting. Conneau et al. (2020b) have found that multilingual MLM (i.e
training with an MLM objective with concatenated text for multiple languages)
naturally leads to models with strong cross-lingual transfer capabilities.
Additionally, they find that this is also found for monolingual models that do
not share vocabulary across monolingual corpora and the only requirement is
that weight sharing is used in the top layers of the multi-lingual encoder. In
the context of our work, we want to bias our fine-tuned and iteratively pruned
model to have similar geometric properties and symmetries to these pretrained
MLMs to preserve zero-shot cross-lingual transfer.

##  3 Methodology

In this section, we describe how our proposed AlignReg weight regularizers can
improve pruning performance in both supervised learning and zero-shot pruning
settings. We focus on two regularizers, namely, a neuron correlation-based
regularizer ($\mathrm{cosine}$-$\mathrm{MBP}$) and Frobenius layer-norm
regularizer ($\mathrm{frobenius}$-$\mathrm{MBP}$).

Let $\mathcal{D}:=\\{X_{i},y_{i}\\}^{D}_{i=1}$ where each $X_{i}$ of $D$
training samples consists of a sequence of vectors
$X_{i}:=(\bm{x}_{1},\ldots,\bm{x}_{n})$ and $\bm{x}_{i}\in\mathbb{R}^{d}$
(e.g., $d=512$). For structured prediction (e.g., NER and POS),
$y_{i}\in\mathbb{R}^{n\times c}$ and for single and pairwise sentence
classification, $y_{i}\in\mathbb{R}^{c}$ where $c$ is the number of classes.
Let $\theta=(\theta_{1},\ldots,\theta_{L})$ be the parameters of a pretrained
network $f$ with $L$ layers, where $\theta_{l}$ refers to the parameters,
including weight matrix $\text{\bf W}_{l}$ and bias $b_{l}$, at layer $l$. Let
$f_{\tilde{\theta}}$ be a network with parameters $\tilde{\mathbf{\theta}}$
consisting of weights $\tilde{\text{\bf W}}_{l}\in\mathbb{R}^{N_{l-1}\times
N_{l}}$ and bias $\bm{\tilde{b}}_{l}\in\mathbb{R}^{N_{l}}$ where $N_{l}$ is
the number of units in the $l$-th layer. Here,
$\tilde{\mathbf{W}}_{l}:=\text{\bf W}_{l}\text{\bf M}_{l}$ where M is the
pruned mask. For MBP Karnin (1990) we remove weights of $\text{\bf W}_{l}$,
$\forall l\in L$ with the smallest absolute weight magnitude until a specified
percentage $p$ of weights are removed. Note that this is a layer-wise process
and requires the pruned weights to be masked with $\text{\bf M}_{l}$ which has
0 entries corresponding to weights to be pruned and 1 entries for unpruned
weights $\text{\bf W}_{l}$. Global MBP can also be used whereby the weights
$\\{\text{\bf W}_{l}\\}^{L}_{l=1}$ are first vectorized and concatenated prior
to choosing $p$ lowest weight magnitudes. Unlike layer-wise MBP, the
percentage of weights removed in each layer can vary for global-MBP.
Typically, weight regularization is used with MBP to encourage weight
sparsity. Thus the objective for iterative pruning can be expressed as,

$\mathcal{L}_{\theta}:=\frac{1}{D}\sum^{D}_{i=1}\ell_{ce}\big{(}f_{\tilde{\mathbf{\theta}}}(\text{\bf
X}_{i}),\bm{y}_{i}\big{)}+\lambda||\tilde{\mathbf{\theta}}||_{0}$ (1)

where $\lambda$ controls the influence of the weight magnitude regularization.
We now describe our proposed AlignReg.

###  3.1 _AlignReg_ \- Pruning-Aware Regularization

AlignReg can be used to align weights unit-wise or layer-wise between unpruned
and pruned networks. We initially discuss the $\mathrm{cosine}$-$\mathrm{MBP}$
regularizer.

$\mathrm{cosine}$-$\mathrm{MBP}$ aims to preserve the inherent cross-lingual
alignment, during iterative pruning, by minimizing the angle between parameter
vectors of the same unit in the pruned and unpruned network. The intuition is
that cross-lingual alignment relies more on parameter vector direction than
vector magnitudes. Moreover, as the network is being pruned, the weights will
consequently change weight magnitude to account for the information loss. To
apply _AlignReg_ to linear layers within Transformers, we compute the pairwise
cosine similarity between pairs of pruned weights
$\mathbf{\tilde{W}}_{l}\subset\tilde{f}$ and unpruned weights $\text{\bf
W}\subset f$ for all $l$-th layers. For $\text{\bf
W}_{l}\in\mathbb{R}^{N_{l-1}\times N_{l}}$ of the $l$-th layer, the average
weight correlation is

$\begin{gathered}\rho(\tilde{\mathbf{W}}_{l},\text{\bf
W}_{l})=\frac{1}{N_{l}}\sum_{i=1}^{N_{l}}\frac{|\text{\bf
W}^{\top}_{li}\tilde{\mathbf{W}}_{li}|}{||\text{\bf
W}_{li}||_{2}||\tilde{\mathbf{W}}_{li}||_{2}}\end{gathered}$ (2)

where $\text{\bf W}_{li}$ is $i$-th column of the matrix corresponding to the
$i$-th unit of the $l$-th layer. Intuitively, $\rho(\text{\bf
W}_{l},\mathbf{\tilde{W}}_{l})$ is the average cosine similarity between
weight vectors of the same unit at the $l$-th layer of the pruned and unpruned
network. Adding AlignReg to the objective results in Equation (3),

$\mathcal{L}_{\theta}:=\ell_{ce}\big{(}f_{\tilde{\theta}}(\text{\bf
X}),\bm{y}\big{)}-\frac{\lambda}{L}\sum_{l}^{L}\rho\big{(}\tilde{\mathbf{W}}_{l},\text{\bf
W}_{l}\big{)}$ (3)

where $\lambda\in[0,\infty)$ controls the importance of AlignReg relative to
the main cross-entropy loss $\ell_{ce}(\cdot,\cdot)$. The gradient of the loss
w.r.t to $\theta$ is then expressed as equation (4),

$\displaystyle\nabla_{\theta}\mathcal{L}_{\theta}:=\nabla_{\tilde{\theta}}\ell_{ce}(f_{\tilde{\theta}}(\text{\bf
X}),\bm{y})-\frac{\lambda}{L}\sum_{l}^{L}\frac{\partial\rho\big{(}\mathbf{\tilde{W}}_{l},\text{\bf
W}_{l}\big{)}}{\partial\tilde{\mathbf{W}}_{l}}$

(4)

where $\frac{\partial\rho(\mathbf{\tilde{W}}_{l},\text{\bf
W}_{l})}{\partial\tilde{\mathbf{W}}_{l}}$ is a function of the ‘2-norm of the
matrices in $\text{\bf W}_{l}$. For the element $\text{\bf W}_{l,(i,j)}$ of
$i$-th row and $j$-th column in $\text{\bf W}_{l}$, we have

$\begin{gathered}\scalebox{0.85}{\mbox{$\displaystyle\frac{\partial\rho(\mathbf{\tilde{W}}_{l},\text{\bf
W}_{l})}{\partial\mathbf{\tilde{W}}_{l,(i,j)}}=\frac{1}{N_{l}-1}\sum_{j=1}^{N_{l}}\Big{(}\textrm{sign}(\text{\bf
W}^{\top}_{l,(,j)}\mathbf{\tilde{W}}_{l,(,j))}$}}\\\
\scalebox{0.85}{\mbox{$\displaystyle\Big{[}\frac{\mathbf{\tilde{W}}_{l,(i,j)}}{||\text{\bf
W}_{l,(,j)}||_{2}||\mathbf{\tilde{W}}_{l,(,j)}||_{2}}\scalebox{1.75}[1.0]{-}\frac{\text{\bf
W}_{l,(i,j)}\text{\bf
W}^{\top}_{l,(,j)}\mathbf{\tilde{W}}_{l,(,j)}}{||\text{\bf
W}_{l,(,j)}||^{3}_{2}||\mathbf{\tilde{W}}_{l,(,j)}||_{2}}\Big{]}\Big{)}$}}\end{gathered}$
(5)

where $\text{\bf W}_{l,(,j)}$ and $\tilde{\mathbf{W}}_{l,(,j)}$ are $j$-th
column in $\text{\bf W}_{l}$ and $\tilde{\mathbf{W}}_{l}$, respectively. Thus,
this regularization favors solutions with high cosine similarity between units
of pruned and unpruned networks. We also consider a layer-wise $\rho(\text{\bf
W},\tilde{\mathbf{W}})$ that relaxes the unit-level alignment to whole layers.
This is partially motivated due to the fact neural networks can exhibit
similar output activation behavior even when neuron weights have been permuted
within the layer Brea et al. (2019). To perform this we simply apply Equation
(2) with vectorized weights
$\rho(\mathrm{vec}(\tilde{\mathbf{W}}_{l}),\mathrm{vec}(\text{\bf W}_{l}))$
and the subsequent partial derivatives in Equations (4) and (5) are applied
for updating $\tilde{\mathbf{W}}_{l}$. In our experiments we did not see a
significant difference using vectorized weights and thus use unit-wise cosine
similarity.

Algorithm 1 shows how AlignReg is applied for a single mini-batch update
during an iterative pruning epoch.

Algorithm 1 AlignReg Pruning

1:Input: Weight tensors $\text{\bf W}_{1},\dots,\text{\bf W}_{L}$ of a fine-
tuned network, $p$ percentage of weights to remove per layer

2:Output: Pruned weight tensors
$\tilde{\mathbf{W}}_{1},\dots\tilde{\mathbf{W}}_{L}$

3:for $l=1,\dots,L$ do

4: Compute $\rho(\tilde{\mathbf{W}}_{l},\mathbf{W}_{l})$ with Eq.2.

5: Set $\tilde{\mathbf{W}}_{s_{i}}$ as $s_{l}$-th smallest element of
$\tilde{\mathbf{W}}$

6: Set
$\mathbf{M}_{l}\leftarrow\mathbbm{1}\\{\mathbf{W}_{l}-\tilde{w}_{s_{l}}\geq
0\\}$

7: Set $\tilde{\mathbf{W}}_{l}\leftarrow\mathbf{M}_{l}\odot\mathbf{W}_{l}$

8:end for

9:Compute $\mathcal{L}_{\theta}$ according to Eq.3

#### Relaxing Unit-Wise AlignReg To A Layer-Wise Frobenius Distortion
Formulation

Thus far we have described the application of cosine similarity as a measure
of similarity between unpruned and pruned weights of the same units. However,
this may be a strict constraint, particularly at high compression rates where
the remaining weights for a unit are forced to have higher norms to allow
zeroed weights. Hence, an alternative measure is the layer-wise Frobenius norm
(Frobenius-MBP) regularizer based on the difference between weights
$||\text{\bf W}-\tilde{\mathbf{W}}||_{F}$. MBP itself can be viewed in terms
of minimizing the Frobenius distortion Han et al. (2016); Dong et al. (2017)
as $\min_{\text{\bf M}:||\text{\bf M}||_{0}=p}||\text{\bf W}-\text{\bf
M}\odot\text{\bf W}||_{F}$ where $\odot$ is the Hadamard product,
$||\cdot||_{0}$ denotes the entrywise 0-norm, and $p$ is a constraint of the
number of weights to remove as a percentage of the total number of weights for
that layer. In the zero-shot setting, we need to account for out-of-
distribution Frobenius distortions, such as _alignment distortion_ in XLM due
to pruning and overfitting to a single language. Taking the view of Frobenius
distortion minimization when using our weight regularizer, we reformulate it
to include Frobenius-MBP as,

$\displaystyle\min_{\text{\bf M}:||\text{\bf M}||_{0}=p}\Big{[}||\text{\bf
W}\text{-}\text{\bf M}\odot\text{\bf W}||^{2}_{F}+\lambda||\text{\bf
W}^{T}\text{-}\text{\bf M}\odot\text{\bf W}||^{2}_{F}\Big{]}$

(6)

where $\text{\bf W}^{T}$ are the weights from the pretrained model prior to
fine-tuning that is cross-lingually aligned from the masked language modeling
(MLM) pretraining objective. In our experiments, $\lambda=5\times 10^{-4}$.

####  _frobenius_ -MBP Implicitly Aligns Eigenvectors

To explicitly show that the Frobenius distortion minimization aligns fine-
pruned and pretrained parameter vectors we expect their eigenvectors to also
be close. We can use the Eckart-Young-Mirsky Theorem (Golub et al., 1987) to
express Frobenius distortion minimization as Equation 7,

$||\text{\bf W}^{T}-\text{\bf M}\odot\text{\bf
W}||^{2}_{F}=||\bm{\Sigma}-\text{\bf U}^{\top}\text{\bf M}\odot\text{\bf
W}\text{\bf V}||^{2}_{F}$ (7)

where the unitary invariance under the 2-norm that U,V vanishes and singular
value matrix is left to approximate $\text{\bf W}^{T}$, hence the inclusion of
$\bm{\Sigma}$. We express $\text{\bf X}=\text{\bf U}_{k}\Sigma^{12}_{k}$,
$\text{\bf Y}=\Sigma^{12}_{k}\text{\bf V}^{\top}_{k}$ and $\text{\bf
X}\text{\bf Y}=\text{\bf A}_{k}$. Hence, we can further describe the
minimization as $||\Sigma-\text{\bf U}^{\top}\text{\bf W}^{T}_{k}\text{\bf
V}||^{2}_{F}$ and since X, Y are unitary, $||\Sigma-\Sigma_{k}||^{2}_{F}$.

Figure 1: English and Zero-Shot Test Accuracy on News Classification.

###  3.2 Connections to Knowledge Distillation

Knowledge distillation (KD) works by using outputs of the last layer Hinton et
al. (2015) or intermediate layers Romero et al. (2015) as additional soft
targets. AlignReg regularizers instead operate directly on minimizing a
divergence or distance between weight tensors as opposed to their
corresponding output activations. Hence, AlignReg does not necessarily need
training data as it operates directly on aligning weight tensors. Since the
networks that are used for alignment are architecturally identical, we can
show that maximizing weight similarity is equivalent to minimizing distance
between their corresponding output activations Romero et al. (2015) when the
norm of input $Z$ is smaller than the output range of $\sigma$. For our
experiments, we use XLM-RoBERTaBase which contain Gaussian Linear Error Unit
(GeLU) activation functions, which can be formulated as $\sigma(\text{\bf
Z}_{li}):=\text{\bf Z}_{li}/2(1.0+\mathrm{erf}(\text{\bf Z}_{li}/\sqrt{2.0}))$
where $\mathrm{erf}$ is an error function, $\sigma(\cdot)$ is a monotonic
activation function and $\bm{Z}_{li}$ is the input vector. The GELU activation
has the properties that for $\text{\bf Z}_{li}>0$ it is equivalent to the ReLU
activation and $\text{\bf Z}_{li}\leq 0$ it tends to -1. For $\text{\bf
Z}_{li}>0$, $||\text{\bf Z}_{li}||_{2}\leq 1$ and a monotonic piecewise linear
function $\sigma(\cdot)$, the inequality in Equation 8 holds.

$\displaystyle||\text{\bf W}_{li}\text{-}\text{\bf M}_{li}\odot\text{\bf
W}_{li}||_{F}\leq$ (8) $\displaystyle||\sigma(\text{\bf Z}_{l}\text{\bf
W}_{li})-\sigma(\text{\bf Z}_{li}\text{\bf M}_{li}\odot\text{\bf
W}_{li})||_{F}$

Layer normalization leads to features having zero mean and unit variance and
hence $||\text{\bf Z}_{li}||_{2}\leq 1$. Hence, minimizing the Frobenius
distortion of pruned and unpruned weights is equivalent to minimizing the mean
squared error (MSE) between output activations, as is the knowledge
distillation method used for FitNets Romero et al. (2015). In contrast, KD
using FitNets encourages the student network to have activation outputs that
are the same as the teacher with permutation invariance on the units incoming
weights, not restricting the weights to be similar. Unlike KD, this
minimization can be performance without any data.

##  4 Experimental Setup

#### Datasets.

We perform experiments on multilingual tasks from the XGLUE benchmark Liang et
al. (2020) with pretrained XLM-RBase. This covers pairwise classification
(QAM, QADSM, WPR, XNLI), sentence classification (NC) and structured
prediction (NER and POS) tasks.

#### Iterative Pruning Details.

Texts are tokenized using the SentencePiece BPE tokenizer Sennrich et al.
(2016) with a vocabulary of 250K tokens. For structured prediction tasks (POS
and NER), a single layer feed-forward (SLFF) token-level classifier is used on
top of XLM-RBase and for sentence-level task a SLFF sentence-level classifier
is used. The batch size is 32, the learning rate is $5\cdot 10^{-6}$ and the
maximum sequence length is set to 256 for all tasks, except for POS in which
we use a learning rate of $2\cdot 10^{-5}$ with the $\mathtt{adam}$ optimizer
Kingma and Ba (2015) with weight decay (AdamW) and a max sequence length of
128. We carry out a pruning step after each 15 training epochs, uniformly
pruning 10% of the parameters at each pruning step. We omit the pruning of
embedding layers, layer normalization parameters and the classification layer
as they account for a relatively small number of the total parameter count
($<1$%) and play an important role in XLM generalization. Although prior work
has suggested non-uniform pruning schedules (e.g., cubic schedule Zhu and
Gupta (2017)), we did not see major differences to uniform pruning in
preliminary experiments. Each task is trained with English data only and
evaluated on all available languages for that task. Hence, we expect the
percentage of achievable compression to be lower as performance in the zero-
shot cross-lingual setting to be more difficult than the monolingual setting
(e.g., GLUE tasks).

Figure 2: Zero-Shot Test F1 on Named Entity Recognition.  Figure 3: Question
Answer Matching Test Accuracy.

#### Pruning Baselines.

Below lists our pruning baselines. Random Pruning Stepniewski and Keane (1997)
\- weights are pruned uniformly at random across all layers to a chosen
fraction.Layer-wise Magnitude Pruning (MBP) Janowsky (1989); Mozer and
Smolensky (1989) \- for each layer, weights with the lowest absolute value
(LAV) are pruned. Layer-wise Gradient Magnitude Pruning Sun et al. (2017) \-
for each layer, prunes the weights with LAV of the accumulated gradients
evaluated on a batch of inputs. Global Magnitude Pruning (Global-MBP) Karnin
(1990) \- prunes weights with LAV anywhere in the DNN. $L_{0}$ norm MBP
Louizos et al. (2018) \- uses non-negative stochastic gates that choose which
weights are set to zero as a smooth approximation to the non-differentiable
$L_{0}$-norm. Lookahead pruning (LAP) Park et al. (2019) \- prunes paths that
have smallest weight magnitude across blocks of layers, unlike MBP which
treats layers independently. Layer-Adaptive Magnitude Pruning (LAMP) Lee et
al. (2020) adaptively sets the pruning ratio of each layer.

Figure 4: Part of Speech Tagging Test Accuracy.

##  5 Empirical Results

We now discuss results on the XGLUE tasks.

#### News Classification (NC)

Figure 1 shows the results on news classification where a category for news
article is predicted and evaluated in 5 languages and trained and iteratively
pruned on English text. Firstly, we observe the trend in iterative pruning
performance degradation is somewhat volatile. From preliminary experiments we
found news classification to require only 3 epochs to converge for standard
fine-tuning on XLM-RoBERTaBase. We find that this task is relatively “similar”
to the pretraining task and therefore able to easier recover from pruning
steps. Overall, both Cosine-MBP and Frobenius-MBP consistently lead to the
best zero-shot test performance across both pruning steps and languages.

Figure 5: Web-Page Ranking Test Matthew’s Correlation Coefficient.

#### Question Answer Matching (QAM)

Figure 3 shows the test accuracy on English and the zero-shot test accuracy on
French and German for Question-Answer Matching (QAM). This involves predicting
whether a question is answered correctly or not given a question-answer pair.
We find that Frobenius-MBP and Cosine-MBP maintain higher accuracy across
multiple pruning steps, outperforming baselines. More generally, we see there
is close to 2% drop in average test accuracy drop in French and German when
compared to testing on samples from the same language used in training.

Figure 6: Zero-Shot XNLI Results Per Language After Iteratively Fine-Pruning
XLM-RoBERTaBase

#### Named Entity Recognition (NER)

The Named Entity Recognition (NER) cross-lingual dataset is made up of
CoNLL-2002 NER and CoNLL-2003 NER Sang and De Meulder (2003), covering
English, Dutch, German and Spanish with 4 named entities. From Figure 2 we
find that cross-lingual transfer of pruned models is most difficult in German
and Dutch, which both come from the same language family, sharing
commonalities such as word order and having similar vocabularies. The primary
reason for the difficulty in maintaining performance in high compression rates
for this NER dataset is that there is only 15k training samples, being
significantly lower than the remaining XGLUE tasks (the majority contains 100k
training samples). Thus, not only is there less training data to recover
directly after each pruning step, but the pruning step interval itself is
shorter. In contrast, English test performance is close to the original
performance up until 25% of remaining weights, unlike the remaining languages.
We find that gradient-MBP eventually overtakes MBP approaches past 20%
remaining weights. However accuracy has reduced too much at this compression
level. We find that Cosine-MBP and Frobenius-MBP weight regularizers achieve
the best performing pruned model performance above 20% remaining weights, with
Lookahead pruning and $L_{0}$ regularized MBP being competitive in zero-shot
performance.

#### Part of Speech Tagging (POS)

The Part of Speech (PoS) tagging dataset consists of a subset of the Universal
Dependencies treebank Nivre et al. (2020) and covers 18 languages. In Figure
4, we see both Cosine-MBP and Frobenius-MBP tend to outperform baselines,
although $L_{0}$-based pruning Louizos et al. (2018) has similar performance
to Cosine-MBP for zero-shot accuracy. There is also a clear discrepancy
between SSL accuracy (English) versus zero-shot accuracy (Average), the latter
following closer to linear decay after 40-50% of weights remaining. Generally,
both Cosine-MBP and Frobenius-MBP outperform baselines with the exception of
Thai and Urdu at higher compression rates ($<40$%), both being some of the
most under-resourced languages of all 18 languages.

#### Web Page Ranking

aims to predict whether a web page is relevant (1-5 ratings, “bad” to
“perfect”) to an input query and it is evaluated for 7 languages using the
Normalized Discounted Cumulative Gain (nDCG). From Figure 5, we see that
between the 15% - 45% region the average zero-shot performance degrades faster
than the English language used for training. In contrast, semantically and
syntactically different languages from English, such as Chinese, already
suffer from loss of alignment due to pruning as the performance gap between
proposed methods (and baselines) and random pruning is shortened.

#### Cross-Lingual Natural Language Inference (XNLI)

Figure 6 shows the zero-shot cross-lingual transfer for various unstructured
pruning methods. We find that both the accuracy on the English test (i.e SSL
generalization) and the average zero-shot test accuracy are consistently
improved using Cosine-MBP and Frobenius-MBP, outperforming $L_{0}$ pruning,
Lookahead pruning and LAMP. We find that morphologically rich languages such
as Arabic, Swahili and Turkish degrade in performance linearly once
performance begins to drop after 60% of the remaining weights are pruned. This
trend is roughly followed for all MBP-based pruning methods. Additionally,
test accuracy on English can be maintained within 10% accuracy drop of the
original test accuracy up to 20% of remaining weights for MBP, while Swahili
can only be within a 10% accuracy drop up to 55% of the remaining weights.
Hence, iterative pruning in the zero-shot setting leads to faster performance
degradation for languages that are typologically or etymologically further
from the language used for fine-tuning.

When comparing, English and the average zero-shot test accuracy we see that
the slope is steeper after the inflection point222The point which the
performance slope significantly steepens and drops are relatively large to
previous pruning steps. for all pruning methods, not to mention the greater
than 10% accuracy drop across pruning steps.

#### XGLUE Average Result

Finally, in Table 1 we show the overall and average task understanding scores
on the XGLUE benchmark for our proposed AlignReg weight regularizer and the
pruning baselines. We find that the use of AlignReg Cosine-MBP and Frobenius-
MBP better preserves cross-lingual alignment during model pruning, thereby
outperform other MBP baselines, including LAMP and Lookahead pruning, based on
improved zero-shot cross-lingual performance.

Prune Method | XNLI | NC | NER | PAWSX | POS | QAM | QADSM | WPR | Avg.
---|---|---|---|---|---|---|---|---|---
No Pruning | 73.48 | 80.10 | 82.60 | 89.24 | 80.34 | 68.56 | 68.06 | 73.32 | 76.96
Random | 51.22 | 70.19 | 38.19 | 57.37 | 52.57 | 53.85 | 52.34 | 70.69 | 55.80
Global-Random | 50.97 | 69.88 | 38.30 | 56.74 | 53.02 | 54.02 | 53.49 | 69.11 | 55.69
$L_{0}$-MBP | 64.75 | 78.98 | 56.22 | 72.09 | 71.38 | 59.31 | 53.35 | 71.70 | 65.97
$L_{2}$-MBP | 64.30 | 78.79 | 54.43 | 77.99 | 70.68 | 59.24 | 60.33 | 71.52 | 67.16
$L_{2}$-Global-MBP | 64.17 | 78.64 | 54.47 | 75.51 | 72.27 | 59.26 | 60.10 | 71.50 | 66.99
$L_{2}$-Gradient-MBP | 61.11 | 73.77 | 53.25 | 79.56 | 65.89 | 57.35 | 59.33 | 71.59 | 65.23
Lookahead | 60.84 | 79.18 | 54.44 | 71.05 | 68.76 | 55.94 | 53.41 | 71.26 | 64.36
LAMP | 58.04 | 63.64 | 51.92 | 66.05 | 67.43 | 55.36 | 52.42 | 71.09 | 60.74
Cosine-MBP | 66.20 | 79.15 | 55.62 | 78.45 | 71.62 | 57.56 | 61.37 | 72.51 | 67.81
Frobenius-MBP | 65.71 | 79.84 | 55.61 | 78.78 | 71.62 | 61.62 | 61.37 | 71.48 |  68.25†

Table 1: Overall XGLUE Score for Iterative Pruning of XLM-RBase @ 31%
Remaining Weights.

#### Discussion

Figure 7: Pruned Model Weight Norms Per Layer

From our experiments, we found that layer-wise pruning tends to outperform
global pruning. This can be explained by the clear discrepancy between weight
norms of different layer types within each self-attention block. Global
pruning chooses the majority of weights to prune from the layer type that has
the smallest norm, leading to an information bottleneck, or layer collapse Lee
et al. (2018) for very high compression rates. This effect is due to layer
normalization being applied after query, key and value (QKV) parameters,
rescaling features such that weight magnitudes remain low. Hence, this
motivates why we have focused on the application of AlignReg to layer-wise
MBP. This is reflected in Figure 7 which shows the weight norm by layer type
for each layer for MBP. We see that QKV weight values are distinctly higher
than the remaining fully-connected layers (attention output layer,
intermediate position-wise feedforward layer and the blocks output layer),
with the exception that the output attention layer norm becomes higher between
layer 3-8.

Figure 8: Class Separability Between Class Representations At Each Iterative
Pruning Step on PAWSX.

For the majority of tasks, the rate of performance drop for zero-shot test
performance occurs close to 30% of remaining weights. This is consistent for
all pruning methods and therefore the focus of our analysis has been around
this operating region.

We also note that the effect of MBP (including our AlignReg regularization-
based MBP) on zero-shot performance for different languages heavily depends on
the semantic distance of evaluated language to the single language used for
training. For example, in Figure 6 Arabic, Bulgarian, Swahili and Hindi have
the largest drops in test accuracy around 20-60% remaining weights. Similarly
Arabic, Thai and Hindi suffer most around 20% - 60% for PoS tagging in Figure
4. However, we also acknowledge this is partly reliant on the proportion of
training data per language used during pretraining the underlying language
model, in our case XLM-RBase.

Lastly, to show the representational degradation of pruned networks, in Figure
8 we visualize the class separability via a t-SNE plot of two principal
components of the last hidden representation corresponding to the [CLS] token
of an iteratively pruned XLM-RBase for PAWSX. Even from only two principal
components of a single token input, we clearly see a change in class
separability from 31% to 28% remaining weights, reflecting the lack of linear
separation.

##  6 Conclusion

In this paper, we analysed iterative pruning in the zero-shot setting where a
pretrained masked language model uses self-supervised learning on text from
various languages but can only use a single language for downstream task fine-
tuning. We find that some languages degrade in iterative pruning performance
faster than others for some tasks (NER and XNLI) and propose a weight
regularizer that biases the iteratively pruned model towards learning weight
distributions close to the cross-lingually aligned pretrained state. This
improves over well-established weight regularization methods for magnitude-
based pruning in both the standard supervised learning setting and the zero-
shot setting.

## References

  * Brea et al. (2019) Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. 2019.  Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape.  _arXiv preprint arXiv:1907.02911_.
  * Bucher et al. (2017) Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. 2017.  Generating visual representations for zero-shot classification.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , pages 2666–2673.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 535–541.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020\.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7870–7881, Online. Association for Computational Linguistics.
  * Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in neural information processing systems_ , volume 32.
  * Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b.  Emerging cross-lingual structure in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6022–6034, Online. Association for Computational Linguistics.
  * Ding et al. (2018) Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. 2018.  Auto-balanced filter pruning for efficient convolutional neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Dong et al. (2017) Xin Dong, Shangyu Chen, and Sinno Pan. 2017.  Learning to prune deep neural networks via layer-wise optimal brain surgeon.  _Advances in Neural Information Processing Systems_ , 30.
  * Golub et al. (1987) G.H. Golub, Alan Hoffman, and G.W. Stewart. 1987.  A generalization of the eckart-young-mirsky matrix approximation theorem.  _Linear Algebra and its Applications_ , 88-89:317–327.
  * Gordon et al. (2020) Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. 2020.  Compressing bert: Studying the effects of weight pruning on transfer learning.  _arXiv preprint arXiv:2002.08307_.
  * Han et al. (2016) S Han, H Mao, and WJ Dally. 2016.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  In _International Conference on Learning Representations_.
  * Han et al. (2015) Song Han, Jeff Pool, John Tran, and William Dally. 2015.  Learning both weights and connections for efficient neural network.  _Advances in neural information processing systems_ , 28.
  * Hassibi and Stork (1993) Babak Hassibi and David G Stork. 1993.  _Second order derivatives for network pruning: Optimal brain surgeon_.  Morgan Kaufmann.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Janowsky (1989) Steven A Janowsky. 1989.  Pruning versus clipping in neural networks.  _Physical Review A_ , 39(12):6600.
  * Karnin (1990) Ehud D Karnin. 1990.  A simple procedure for pruning back-propagation trained neural networks.  _IEEE transactions on neural networks_ , 1(2):239–242.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Lebedev and Lempitsky (2016) Vadim Lebedev and Victor Lempitsky. 2016.  Fast convnets using group-wise brain damage.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2554–2564.
  * LeCun et al. (1990) Yann LeCun, John S Denker, and Sara A Solla. 1990.  Optimal brain damage.  In _Advances in neural information processing systems_ , pages 598–605.
  * Lee et al. (2020) Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. 2020.  Layer-adaptive sparsity for the magnitude-based pruning.  In _International Conference on Learning Representations_.
  * Lee et al. (2018) Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018.  Snip: Single-shot network pruning based on connection sensitivity.  In _International Conference on Learning Representations_.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020.  Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018.
  * Liu et al. (2017) Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017.  Learning efficient convolutional networks through network slimming.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2736–2744.
  * Louizos et al. (2018) Christos Louizos, Max Welling, and Diederik P Kingma. 2018.  Learning sparse neural networks through l_0 regularization.  In _International Conference on Learning Representations_.
  * Molchanov et al. (2016) Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016.  Pruning convolutional neural networks for resource efficient inference.  _arXiv preprint arXiv:1611.06440_.
  * Mozer and Smolensky (1989) Michael C Mozer and Paul Smolensky. 1989.  Skeletonization: A technique for trimming the fat from a network via relevance assessment.  In _Advances in neural information processing systems_ , pages 107–115.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Park et al. (2019) Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. 2019.  Lookahead: A far-sighted alternative of magnitude-based pruning.  In _International Conference on Learning Representations_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Reed (1993) Russell Reed. 1993.  Pruning algorithms-a survey.  _IEEE transactions on Neural Networks_ , 4(5):740–747.
  * Romero et al. (2015) Adriana Romero, Samira Ebrahimi Kahou, Polytechnique Montréal, Y. Bengio, Université De Montréal, Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015.  Fitnets: Hints for thin deep nets.  In _in International Conference on Learning Representations (ICLR)_.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander M Rush. 2020.  Movement pruning: Adaptive sparsity by fine-tuning.  _arXiv preprint arXiv:2005.07683_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Singh and Alistarh (2020) Sidak Pal Singh and Dan Alistarh. 2020.  Woodfisher: Efficient second-order approximation for neural network compression.  _Advances in Neural Information Processing Systems_ , 33:18098–18109.
  * Stepniewski and Keane (1997) Slawomir W Stepniewski and Andy J Keane. 1997.  Pruning backpropagation neural networks using modern stochastic optimisation techniques.  _Neural Computing & Applications_, 5(2):76–98.
  * Sun et al. (2017) Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. 2017.  meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting.  In _International Conference on Machine Learning_ , pages 3299–3308. PMLR.
  * Wang et al. (2019) Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. 2019.  Eigendamage: Structured pruning in the kronecker-factored eigenbasis.  In _International Conference on Machine Learning_ , pages 6566–6575. PMLR.
  * Wang et al. (2020) Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. 2020.  Neural pruning via growing regularization.  In _International Conference on Learning Representations_.
  * Wen et al. (2016) Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016.  Learning structured sparsity in deep neural networks.  _Advances in neural information processing systems_ , 29.
  * Ye et al. (2018) Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. 2018.  Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers.  In _International Conference on Learning Representations_.
  * Yuan and Lin (2006) Ming Yuan and Yi Lin. 2006.  Model selection and estimation in regression with grouped variables.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 68(1):49–67.
  * Zhu and Gupta (2017) Michael Zhu and Suyog Gupta. 2017.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  _arXiv preprint arXiv:1710.01878_.


# The updated SANAE neutron monitor

R. D. Strauss<EMAIL_ADDRESS>C. van der Merwe,
<EMAIL_ADDRESS>C. Diedericks<EMAIL_ADDRESS>H. Krüger,
<EMAIL_ADDRESS>H. G. Krüger<EMAIL_ADDRESS>K. D.
Moloto<EMAIL_ADDRESS>S. Lotz<EMAIL_ADDRESS>G. M. Mosotho,
<EMAIL_ADDRESS>Center for Space Research, North-West
University, Potchefstroom Campus, Private Bag X6001, 2520 Potchefstroom, South
Africa  South African National Space Agency, PO BOX 32, 7200 Hermanus, South
Africa

###### Abstract

Neutron monitors have been the premier ground-based instruments for monitoring
the near-Earth cosmic ray flux for more than 70 years. It is essential to
continue with such measurements in order to extend this unique long-term time
series. Moreover, with the recent interest of the aviation industry to space
weather effects, and especially the radiation risk posed by solar energetic
particles and galactic cosmic rays, it is vital to extend the current neutron
monitor network in order to provide near-real-time measurements to the space
weather community. In this paper we discuss a new electronics system that was
retrofitted to the SANAE neutron monitor in Antarctica. We present initial
results from this system, featuring very high temporal resolution and discuss
the techniques applied to the data analysis. Based on these successful
upgrades, we are confident that this system can be used to rejuvenate the
aligning neutron monitor network, and even possibly to revive some of the
decommissioned instruments.

###### keywords:

cosmic rays , neutron monitor , space weather

††journal: Advances in Space Research

Fig. 1: The top panel shows the location of the SANAE IV base, along with a
selection of other Antarctic bases operating NMs. The SANAE NM, with its front
cover removed, is shown in the bottom panel. The middle panel, showing the
SANAE IV base, is a modified version of an image by Dr Ross Hofmeyr (CC BY-SA
3.0)

##  1 Introduction

Cosmic rays, of galactic and solar origin, are the main source of harmful
natural radiation at, and above, aviation altitudes. As such, their
intensities must be monitored in near-real-time and the most cost effective
instrument to do so, is the neutron monitor (NM). The near-real-time data from
the South African NM network is available at
https://fskbhe1.puk.ac.za/neutronmonitor/, and can be monitored live at
https://fskbhe1.puk.ac.za/spaceweather/. Of course, NM observations are also
used for theoretical studies, especially long-term cosmic ray studies, where
the quality of the data, and the stability of the monitor, are major concerns.

CRs can enter the Earth’s atmosphere if they have sufficient energy to pass
through the Earth’s geomagnetic field. This minimum energy is given in terms
of a cut-off rigidity (i.e. momentum per unit change), $P_{c}$, which depends
strongly on geomagnetic latitude (e.g. Dorman, 2004; Desorgher et al., 2009).
When CRs enter the atmosphere, they interact with nuclei of atmospheric gases,
leading to showers of secondary particles. These secondary protons and
neutrons are then detected on ground level by NMs.

These instruments, therefore, do not measure the cosmic radiation directly,
but rather the flux of secondary and tertiary neutrons produced in the
atmosphere, and in the detector itself. The count rate of a NM can thus be
written, in the most basic formulation, as

$N(P_{c},t)=\sum_{i}\int_{P_{c}}^{\infty}j_{i}(P,t)Y_{i}(P,t,\ldots)dP$ (1)

where $i$ represents the particle distribution under consideration (e.g.
protons and helium particles), $j_{i}(P,t)$ is the differential rigidity
spectrum of cosmic rays of type $i$, and $Y_{i}(P,t,\ldots)$ is the so-called
yield function that represents the response of the instrument to the flux of
primary cosmic rays. This parameter includes atmospheric and instrumental
effects and therefore it depends on various parameters uniquely to each
monitor (Clem & Dorman, 2000; Caballero-Lopez, 2016; Mishev et al., 2020)

The basic design of the NM has not changed much in 70 years (Simpson, 2000;
Bütikofer, 2018). The main part of the instrument is the lead producer where
secondary nucleons interact with lead nuclei, which in turn releases a number
of evaporated neutrons (Bieber et al., 2004). These suprathermal neutrons
moves through a moderator (made from a proton-rich material; usually
polyethylene in newer NMs, and paraffin wax in older types NMs), thermalizing
their energies, and thus leading to a larger cross section to react with the
BF3 gas in the detector tube. There, the following neutron capture process
takes place (Knoll, 2010)

${}^{10}\mathrm{B}+\mathrm{n}$ $\displaystyle\rightarrow$
$\displaystyle\left\\{\begin{array}[]{ccc}^{7}\mathrm{Li}\,\,\,+\alpha+2.78\,\mathrm{MeV}\\\
^{7}\mathrm{Li}^{*}+\alpha+2.30\,\mathrm{MeV}\end{array}\right.$ (4)

where the $\alpha$ particle ionizes the gas as it is accelerated by an applied
voltage, leading to a slight decrease in the applied voltage, and which is
registered by appropriate electronics as a pulse. Note that about 6% of the
reactions leave 7Li∗ in an excited state. 3He gas is also an efficient neutron
capture gas and is used in some detectors. Lastly, the NM is surrounded by a
reflector (made from the same material as the moderator), which stops thermal
neutrons from entering the monitor, while also trapping neutrons, produced in
the lead producer, inside of the NM. Neutrons produced inside the producer
from high energy protons tend to show a high level of multiplicity as a single
cosmic ray proton can produce multiple neutrons. However, suprathermal
neutrons, produced in the atmosphere, and neutrons formed in the producer by
low energy protons are registered through the same neutron capture process,
resulting in a low level of multiplicity. We will show how to differentiate
between these two populations in a later section.

South Africa has had a permanent presence on the continent of Antarctica since
1959/1960 when the first overwintering team, South African National Antarctic
Expedition (SANAE) 1, spent the winter in an abandoned Norwegian base. The
first South African base, SANAE I (note the Roman numeral), was opened in 1962
during the SANAE 2 expedition. The SANAE I base was constructed on the
Fimbulisen Ice Shelf in the Dronning Maud Land, located at 70∘18’S 2∘22’W. A
3NM64 (3 tubes of the 1964 super neutron monitor design) was installed at the
SANAE I based by the SANAE 5 expedition in 1963/1964. The SANAE I base,
however, suffered from snow accumulation, and the whole NM structure had to be
raised above the new snow level each year. The same design was employed for
subsequent bases, SANAE II (opened in 1971) and III (opened in 1979) at the
same location, where after it was decided to design and build a more permanent
structure. The SANAE IV base was opened in 1997 on the nunatak (a rocky
outcrop) Vesleskarvet located at 72∘40’22”S 2∘50’26”W. During the move to the
SANAE IV base, the NM was upgraded to a 6NM64.The top panel of Fig. 1 shows
the location of SANAE IV, together with a selection of contemporary Antarctic
research stations that, at some point, operated a NM. This selection is by no
means complete and does not included, for example, the new Syowa station (Kato
et al., 2021). The middle panel of the figure shows the SANAE IV base,
emphasizing “Block C” which houses the NM, shown in the bottom panel with its
front cover removed to show the newly installed electronics. The NM is located
on an elavated steel platform close to the building’s roof. A historical
account of the SANAE NM is given by Moraal et al. (2011).

In this paper we describe how the old NM tubes were retrofitted in February
2019 with a newly developed data acquisition system, which is a self-contained
unit, featuring an integrated high-voltage (HV) supply, and a temporal
resolution of 500ns. We describe the data reduction applied to the
measurements, and how this new system can be used to track the integrity of
the measurement through new proxies. We propose that the system described here
can also be used to rejuvenate and/or restart older NM stations that were
decommissioned due to either financial constraints or lack of replacement
parts.

##  2 New SANAE installation

Although the SANAE IV NM installation was more environmentally stable than
previous SANAE NM incarnations, there was still a major problem with noise in
the high-voltage (HV) supply during windy conditions due to static build-up.
Together with the ailing 50 year old electronics, is was decided to perform a
complete update, beginning at the end of 2018. We decided to use the new
electronic heads, developed for our mini-NM project, and described in detail
by Strauss et al. (2020).

###  2.1 Hardware changes

The SANAE BP28 NM tubes are BP28F type (Fowler, 1963) used extensively in the
International Quiet Sun Year (IQSY) NMs, also know as “super-NMs”, in the
1960’s (Hatton & Carimichael, 1964). All older electronics, including the
attached pre-amplifiers, were completely replaced. The polarity of the tubes
were also changed to a positive central wire and grounding the walls of the
tubes to both the HV generator, but also the main ground pin of the base. The
BP28’s guard rings, which surrounds the central wire near the walls of the
tube, were connected to the central wire through a 5$\Omega$ resistor (Krüger
et al., 2017). The central wire and walls were connected to a normal HV
connector, and the electronics head used in the mini-NMs were screwed onto
each tube. The tubes are operated at a potential difference of 2850V (Fowler,
1963). The finished set-up is shown in the bottom panel of Fig. 1.

When we started examining the 10 BP28 tubes housed at SANAE (for both the
standard 6NM64 and the neutron moderated detector, 4NMD), it became clear that
the tubes all suffered from different levels of degradation (see also the
discussion in Sec. 3.6 and A), with some of the older tubes not producing
reliable pulse forms. After testing, 6 tubes were deemed to be in a good
condition and we ended up with a 3NM64 (consisting of lead tubes 4,5,6) and a
3NMD (tubes 1,2,3). In this paper we discuss only the 3NM64. Based on the
different levels of degradation, tubes 4,5, and 6 have different levels of
amplification; the implication thereof discussed in Sec.3.1.

###  2.2 Software changes

All the hardware described above uses newly developed software, similar to
what is implemented for our mini-NM programme and described in detail by
Strauss et al. (2020). Raw data (see Sec. 3.1) is saved on solid state drives
(SSDs) in Antarctica, while the (uncorrected) minute-averaged pressure,
temperature, and count rate from each tube is sent to a sever in
Potchefstroom, South Africa. There, the data is pressure and temperature
corrected (see Sec. 3.2). SANAE raw and corrected data published at
https://fskbhe1.puk.ac.za/neutronmonitor/SANAE/. Due to the limited bandwidth,
not all raw data can be sent back to Potchefstroom in real-time, and only a
selection is available to validate the instrument. SSDs are switched at the
end of every year during the annual South African relief voyage in December,
and then manually copied onto our servers.

##  3 Data processing

###  3.1 Raw data and multiplicities

Fig. 2: The time profile of 100 pulses, with two overlapping pulses coloured
red, is shown. The horizontal dashed line shows the ADC threshold level. Note
that the time axis is given in units of 500ns; the native sampling rate of the
ADC.

Fig. 3: The different panels show the pulse amplitude-width diagrams (number
of pulses, in a logarithmic scale, registered with a certain duration (width;
in units of 500ns) and pulse amplitude (height) for the 3NM64 tubes presently
under consideration (data from tubes 4 (panel a) and 5 (panel b) are shown in
the top left and right panels, respectively, and results for tube 6 (panel c)
in the bottom panel). The vertical and horizontal dashed lines show examples
of how software filters can, in future, be applied.

Fig. 4: The left panel shows the distribution of waiting times (blue
histogram) with the waiting time defined as the time registered between
subsequent pulses. The solid red curve shows a fitted Poisson distribution.
Green and gray dashed lines show the 1 and 10 count limits. The right panel
shows the total integrated count as a function of simulated dead time in the
detector.

The amplified voltage is continuously sampled by the analogue-to-digital
converter (ADC) at a rate of 2MHz (i.e. time interval of 0.5$\mu$s). When this
voltage crosses a pre-defined software threshold (currently at 0.2V), the
start of the pulse is registered, assigned a relative time-stamp, and these
voltage values are saved into the ADC’s buffer until the voltage again drops
below the threshold. The buffer is then readout to a Raspberry Pi micro-
computer, along with the absolute and relative time-stamps. We can therefore
digitize the shape (i.e. temporal profile) of each pulse registered. As far as
we are aware, this is currently the only large-scale NM that has this ability.
Fig. 2 shows 100 of these pulses (in gray) and the ADC threshold level (blue
dashed line). The red curves show two occurrences of the interesting scenario
where pulses overlap. These events are, however, rare for the current counting
rate ($<1\%$). We can also apply further software filters on the digitized
pulses when the ADC’s buffer is read out; something that is not implemented at
the moment.

Shown in Fig. 3 are pulse amplitude-width diagrams constructed from the
individual pulses for each tube. The pulse width (length or duration) is the
time (here, in units of 500ns; the native sampling rate of the ADC) from which
a registered pulse first crosses the ADC threshold to where the signal again
decreasing below this threshold, while the pulse amplitude is the maximum
voltage of each pulse. The amplifying stage can only amplify pulses up to 3.3V
and therefore the amplification and shaping stages are chosen so that the
pulses from cosmic ray interactions and detector noise (low amplitude, low
width pulses) can be distinguished (see the general discussion in e.g. Knoll,
2010, for a discussion on how these two distributions can be separated). In
order to accomplish this, we had to adjust the amplification of each tube;
newer tubes have lower amplification. This leads to different pulse width-
height diagrams for each tube. These 2D distribution are also projected onto
both the pulse amplitude and width planes, shown as the black distributions.
Currently we do not apply any additional data selections on the distributions
and simply count all the pulses shown here. However, in future, we can apply
more complex data selections to better constrain instrumental noise. As an
example, we show a simple rectangular selection (indicated by the dashed
lines), with data in this selection shown as the red histograms.

In the bottom panel (c) we also indicate two effects: The black rectangle
emphasizes a portion of the distribution that indicates two peaks produced by
neutron capture in the BF3 tube, which we attribute to the two reactions in
Eq. 4, with the ${}^{7}\mathrm{Li}^{*}$ reaction producing a peak with a
slightly lower amplitude. The red rectangle shows a selection of pulse which
we believe is the population of overlapping pulses (the red pulse in the top
left panel). We currently do not correct for these scarce over-lapping pulses.
However, with even high counting rates, during e.g. ground level enhancements,
the effect from these overlapping pulses may become significant and will be
investigated further in future. A recent study of the different pulse shapes,
using a similar electronics set-up for a mini-neutron monitor, is presented by
Similä et al. (2021).

The distributions in Fig. 3 are from an hour’s worth of data. Due to the low
bandwidth connection to SANAE, we cannot copy all the raw data, and only a
selection of pulses are copy every day from which such a distribution can be
reconstructed (at present only the first pulse registered at the start of
every second is sent back in near real-time). This is our primary check for
data quality on the different tubes. The most recent (“live”) plot can be
found at https://fskbhe1.puk.ac.za/spaceweather/sanae_tube_4.html for tube 4,
and at similar URLs for the other tubes.

Having access to the raw data, we can also calculated the waiting-time (or
delay) distribution of the pulses. This is done by calculating the delay
between the onset of subsequent pulses, where $j$ labels the pulse number, as
$\Delta t_{j}=\,^{\mathrm{o}}t_{j+1}-\,^{\mathrm{o}}t_{j}$ where the
superscript $\,{}^{\mathrm{o}}$ refers to the onset of each pulse. We use the
(more accurate) relative time-stamp of the ADC as we are interested in
accurate time differences, and not in the absolute timing of the pulses. The
left panel of Fig. 4 shows this distribution as a blue histogram for an hour
worth of data from tube 4 and is similar to the previous results of Krüger et
al. (2017). The gray and green dashed lines show the 1 and 10 count limits.
The system implemented continuously samples at 2MHz and does not have a longer
hardware-define dead-time; the dead-time is determined by the length of the
pulses, and, as shown before, different pulses can overlap and be counted as a
single detection. The blue histogram in the left panel cuts off below $\sim
50-100\mu$s, not due to any dead-time effects, but rather because the average
pulse length is reached.

Two distinct distributions are visible in the waiting-time distribution: The
second distribution, with the larger waiting times, is the population of low-
multiplicity neutrons produced both in the atmosphere, and neutrons produced
in the lead producer from low energy protons (Bieber et al., 2004; Balabin et
al., 2011; Mangeard et al., 2016; Sáiz et al., 2019). The first distribution,
with shorter waiting times, are high-multiplicity evaporated neutrons produced
in the lead producer during nuclear interactions with high-energy incident
protons. Our current system can calculate the multiplicity spectrum to much
shorter time delays as compared to contemporary instruments (e.g. Ruffolo et
al., 2016). The low-multiplicity (large delays between pulses) distribution
can be well described by a Poisson distribution

$\mathcal{D}(\Delta t)=\mathcal{D}_{0}\exp\left(\frac{\Delta
t-\,^{\mathrm{o}}\Delta t}{\tau}\right),$ (5)

with $\Delta t$ the binned waiting times (time between subsequent pulses) and
$\,{}^{\mathrm{o}}\Delta t$ a constant. This distribution is shown as the
solid red line, and the resulting $\tau$ indicated in the legend. The gray
shaded area indicate the data used to constrain the fitted the distribution.

In the right panel of Fig. 4, we calculate, using measurements presented in
the left panel, the total number of counts measured by this detector as a
function of simulated dead time of the detector, $\Delta T$,

$\mathcal{C}(\Delta T)=\int_{\Delta T}^{\infty}\mathcal{D}(\Delta t)d(\Delta
T).$ (6)

The blue vertical line at $\sim 1$ms, indicates the time scale separating the
two multiplicity distributions discussed earlier, consistent with the estimate
of $\sim 4$ms of Bieber et al. (2004). We estimate that at least 20% of the
neutrons detected in the SANAE NM is produced in the lead producer via high-
multiplicity evaporated neutrons. This is, of course, expected to be dependent
on the incident particle spectrum, and will therefore be time dependent, and
dependent on the cut-off rigidity of any NM. The vertical red line, at
40$\mu$s shows the minimum dead time in the detector needed to count 99% of
incident particles. A typical NM64 set-up has a dead time $\sim 20\mu$s
depending on the electronics used (see the discussion by e.g. Bieber et al.,
2004)..

###  3.2 Processed data

Fig. 5: The top panel shows the uncorrected count rate from the three working
NM tubes, the middle panel the pressure from each tube, and the bottom panel
the temperature. Both the temperature and pressure is measured by a
temperature probe inserted next to each BF3 tube. The data extends from 2019
DOY 60 to 2020 DOY 55. All measurements have a minute cadence.

The main space weather data product from the SANAE NM is the minute-averaged
count rate, and we use these measurements from March 2019 to February 2020 in
this paper to perform the data corrections described in this section. Fig. 5
shows these measurements for times when all three tubes provided data. The top
panel of the figure shows the count rates, the middle panel the pressures, and
the bottom panel the temperatures. The data gaps correspond mostly to times
when the instrument was off-line for hardware testing and software upgrades.
As expected, the pressure values from the different tubes are comparable, and
reflect the accuracy of the cost-effective BME-280 pressure sensors used here
(see again the description in Strauss et al., 2020)

The temperatures of the different tubes are, however, significantly different
and there is a clear temperature gradient in the monitor with tube 6 being
consistently colder. Also note that the temperature of the monitor does change
significantly throughout a year, and a temperature correction will have to be
performed as discussed later. Although the base housing the NM is heated,
block C is actively used as a storeroom for the base’s vehicles, and the
outside roller-doors opened occasionally. Also note that during extreme storms
(e.g. 2019 DOY 150), the temperature inside block C can drop significantly.

###  3.3 Pressure correction

Fig. 6: The top rows show the pressure correction calculations, and the bottom
panel the temperature corrections. The left panels are for tube 4, the middle
panels for tube 5, and the right panel for tube 6.

Correction for atmospheric changes is the primary correction on the NM count
rate (Carmichael et al., 1968; Paschalis et al., 2013). The count rate of each
tube $i$ varies according to the relationship,

$N_{i}=\langle N_{i}\rangle\exp\left(-\beta_{i}\Delta P_{i}\right),$ (7)

where $\Delta P_{i}=P_{i}-\langle P_{i}\rangle$ is the pressure change,
$\langle P_{i}\rangle$ is a long-term average pressure value, and $\beta_{i}$
is the barometric coefficient for each tube, labelled by $i$. To calculate the
barometric coefficient of each tube, we plot $\ln(N_{i}/\langle N_{i}\rangle)$
as a function of $-\Delta P_{i}$, and perform a linear regression, with the
barometric coefficient being the gradient thereof. This is shown in the top
panel of Fig. 6 for the different tubes; the left panels are for tube 4, the
middle panels for tube 5, and the right panel for tube 6. The data points show
minute averages, while the linear regression are indicated by the straight
lines. The calculated barometric coefficients, with their associated errors
are indicated in the title of each figure. There is a statistically
significant difference in the barometric coefficient of each tube which is
presumably related to slightly different responses to the incident particles.

Once these coefficients are known, we apply the pressure correction by
calculating

$N_{i}^{\prime}=N_{i}\exp\left(\beta_{i}\Delta P_{i}\right),$ (8)

where $N_{i}^{\prime}$ is the pressure corrected count rate of each tube. At
present we use only the internal pressure of the base and do not correct for
possible pressure changes due to changing environmental conditions (see e.g.
Malan & Moraal, 2002, for a discussion of such effects).

###  3.4 Temperature correction

After applying the pressure correction, we test for any possible temperature
dependence in the count rate. The bottom panel of Fig. 6 shows
$N_{i}^{\prime}/\langle N_{i}^{\prime}\rangle$ as a function of the
temperature change $\Delta T_{i}=T_{i}-\langle T_{i}\rangle$ for the different
tubes. The correction for the temperature changes can be implemented as

${}^{T}N_{i}^{\prime}=N_{i}^{\prime}-{{}^{T}}\beta_{i}\Delta T_{i},$ (9)

where ${{}^{T}}N_{i}$ is now the temperature and pressure corrected count
rate, and ${{}^{T}}\beta_{i}$ the so-called temperature coefficient (i.e. the
gradient of the linear regression curve). These coefficients are indicated on
the graphs for each tube. Again, they are slightly different for each tube,
which is again related to internal differences inside the NM (Krüger et al.,
2008; Evenson et al., 2005). Note that, as the temperature of the tubes do not
vary significantly on short timescales, we can apply a linear temperature
correction and not the normal exponential form (e.g. Krüger et al., 2008).
These forms are, however, consistent as long as ${{}^{T}}\beta_{i}\Delta
T_{i}\ll 1$.

###  3.5 Tube ratios

Fig. 7: The top panel shows binned averages of the calculated tube ratios, for
each tube, along with a fitted Gaussian distribution. The bottom panel shows
the tube ratio (offset with constant factors) as a function of time.

In order to combine the data from different tubes into a single count rate, we
make use of so-called tube ratios (Medina et al., 2013). We calculate this
quantity for each tube $i$, out of a total of $M$ tubes, as

$\mu^{-1}_{i}=\left\langle\frac{1}{{{}^{T}}N_{i}^{\prime}}\frac{1}{M}\sum_{i=1}^{M}{{}^{T}}N_{i}^{\prime}\right\rangle,$
(10)

This is the inverse of the average contribution of each tube to the average
count rate, per tube, of the detector. Usually, in large NMs with identical
tubes, we see that the tubes at the edges have a lower count rate than
centrally located tubes. This edge effect is due to the central tubes being
surrounded by a larger volume of neutron-producing lead. The tube ratios have
the useful relationship

$\sum_{i=1}^{M}\mu_{i}=M.$ (11)

The top panel of Fig. 7 shows the tube ratios, as calculated for each tube,
for the entire dataset presented here. Minute averages are used, binned into
histograms, and fitted by a Gaussian distribution, indicating that any
variations are due to random fluctuations. Each panel is labelled by the
average value of the Gaussian distribution (this is the tube ratio according
to Eq. 10), along with the variance of the distribution ($\sigma$) given an
indication of the error associated with $\mu$.

Plotted in the bottom panel is a test quantity

$\left\langle\frac{{{}^{T}}N_{i}^{\prime}}{\mu_{i}}\left(\frac{1}{M}\sum_{i=1}^{M}{{}^{T}}N_{i}^{\prime}\right)^{-1}\right\rangle=1.$
(12)

These values (related to the contribution of each tube to the total count
rate) should, of course, be constant over time, serving as a measure of the
validity of the measurements and the stability of the monitor over longer time
scales. The near real-time SANAE data, including the tube ratios, can be
monitored at this page: https://fskbhe1.puk.ac.za/spaceweather/sanae.html.

###  3.6 Tube efficiency

Fig. 8: The top panels show normalized pulse height distributions, for each,
tube, taken 12 months apart. The bottom panels show the difference between
these distributions.

As already mentioned, our new hardware and software allow us to examine the
pulse height distribution on a daily, and even hourly, basis. After running
the new system for 12 months, we can also compare, and search for, any
possible time dependent degradation of any tubes. In Fig 8 we show, for each
tube, a normalized histogram of the observed pulse height for two days (the
first of March) in 2019 and 2020. The 2019 distribution, $n_{\mathrm{2019}}$,
is shown in red, and the 2020 distribution, $n_{\mathrm{2020}}$, in blue.
Although on a very shorter time scale, these results seem compatible with
those presented by Bieber et al. (2007) showing tube degradation on a yearly
basis, where the peak of the distribution moves to lower voltages, while the
distribution becomes wider. As discussed by Bieber et al. (2007), the
degradation is probably due to uneven plating of the tube’s central wire,
decreasing its counting efficiency. We present an analysis of several older
(and discarded) tubes in A. In the bottom panels of the figure, we show
$n_{\mathrm{2019}}-n_{\mathrm{2020}}$, illustrating the effect discussed above
in more detail. At present, even though the distribution has shifted to lower
peak values, all pulses, of cosmic ray origin, still seem to be within the
range where they will be counted. As such, we do not yet perform any tube
efficiency calculations to account for any degradation.

###  3.7 Combined count rate

Fig. 9: The top panel shows the combined SANAE NM count rate from March 2019
to March 2020 in different temporal resolutions, alongside hourly data from
the Oulu NM. The bottom panel shows the long term SANAE NM observations from
1965 – present. The symbols are hourly averages and the solid lines are
monthly averages. Note the data gaps around 1997 and 2018, corresponding to
moving the NM to its current positions, and performing the upgrades presented
here, respectively.

Using the tube ratios, determined in Sec. 3.5, we can now combine the counts
from the different tubes into a single count rate from the monitor. However,
we need to account for the fact that not all the tubes will produce valid data
at any given time: A tube might be switched off for repairs, there might be
spikes from a given tube, or it may be malfunctioning. With $M$ the total
number of tubes, and $M^{\prime}$ the number of tubes active at any given
time, we propose the following expression to calculate the total count rate

$C=\epsilon\left(\frac{M}{M^{\prime}}\right)\sum_{i=1}^{M^{\prime}}\zeta_{i}\frac{{{}^{T}}N_{i}^{\prime}}{\mu_{i}}.$
(13)

With the tube ratios correctly calculated, each tube contributes exactly a
fraction of $1/M^{\prime}$ to the total count rate, and the factor
$M/M^{\prime}$ corrects for the possibility that $M^{\prime}$ might be time
dependent. For a real-time visualization of the number of active SANAE NM
tubes, see https://fskbhe1.puk.ac.za/spaceweather/sanae.html. We introduce the
tube efficiency via the parameter $\zeta_{i}$, with is currently set to unity
for each tube, but could increase/decrease as the degradation of the tubes
becomes measurable. The detector efficiency, $\epsilon$, which is also
currently set to unity, can be used, in future, to account for changes in the
NM’s environment, i.e. changes that will affect the counting rate of all tubes
in a similar fashion.

The combined SANAE count rate is shown in the top panel of Fig. 9 in minute,
hourly, and daily resolution as the gray, blue, and red curves. The horizontal
lines show the yearly average (solid black line) and possible 5% variations
(dashed black lines). The Oulu NM count rate is shown in green for comparison.
During the calibration time, March 2019 – March 2020, there were very little
cosmic ray modulation observed, and this seems to be also reflected in the
SANAE observations. There seems, however, to be a small discrepancy in the
SANAE NM data around September 2019 (2019.75), which is still under
investigations. We currently believe that this might be due to extreme
environmental conditions, and this is discussed in more detail in Sec. 4.

###  3.8 Long-term normalization

With the recent upgrades and changes to the SANAE NM set-up completed, the
current count rate must be normalized to older measurements to create a
consistent long-term dataset. In order to do so, we use the Oulu NM as a
calibrator: The Oulu NM count rate is normalized to the SANAE dataset (using
monthly averages) between January 2009 – December 2009, where after the new
SANAE measurement (monthly averages between March 2019 – March 2020) is again
normalized to the Oulu measurements. Both time periods represent quiet-time
solar minimum conditions, with very few temporal variations. The results are
shown in the bottom panel of Fig. 9, with the SANAE NM count rate normalized
to 100% in March 1987. The symbols show hourly averages from 1965 to the
present, while the solid lines show monthly averages. Note that the dataset is
divided into three periods, based on the evolution of SANAE NM, with data gaps
around 1997 (when the NM was moved to its current position) and 2018 (when the
upgrades presented here were implemented). Note that before 2019 efficiency
calculations were only computed on the monthly averages, while the hourly
averages were only corrected for pressure changes. This difference is apparent
on the figure as a time-dependent difference between the symbols and solid
lines, especially diverging towards the start of observations. These older
efficiency calculations were done by hand and, unfortunately, we do not have
access to the efficiency calculations pre-2019 and thus cannot correct the
minute data retroactively. A major aim of this paper is to also standardize
and document the process to remove any such uncertainties in future.

##  4 Summary and outlook

In this paper we have described the recent upgrades performed on the SANAE NM
in Antarctica. We have retrofitted the older BP28 tubes with newly developed
electronics, originally designed for use in our mini-NM programme. These
electronics feature, amongst other new developments, a self-contained high-
voltage supply, and a new digitizing system allowing us to digitise individual
pulses with a 500ns time resolution. This allows us to validate, on an hourly
basis, the measured pulses via a pulse width-amplitude plot, and calculate
every tube’s multiplicity down to the low $\mu$s level. We can also use this
to monitor, over longer time-scales, any possible tube degradation. In
addition, we have described, in detail, the methodology we applied to correct
and combine the data from the different tubes. This is in an effort to
standardize the approach used at different NM stations and we will implement
these changes to the South African NM network in due course.

We are currently looking at possible environmental issues in more detail. We
suspect the slight enhancement (up to 5%) in the SANAE count rate in September
2020 (2020.75), visible in Fig. 9 is partially due to a long-lasting, and very
severe, storm influencing the detector. These storms at the SANAE IV base
usually lead to a large drop in temperature and pressure, and we will need to
make sure our current approach is valid for such extreme conditions. In
addition to changing environmental conditions, the large wind gusts associated
with the Antarctic storms can lead to significant mechanical vibration in the
base, and this might contribute to mechanical noise in the detector
electronics. We will report on any such issues in future publications. As a
preliminary quality flag the number of tubes registering counts at a specific
time can be used as excessive vibrations usually lead to saturation of at
least one tube’s ADC. The near real-time calculated tube ratios are also a
good indication of the data quality for detailed studies.

The upgrading process described here can be used on any NM using the older
BP28 tubes, as well as the even-older IGY (International Geophysical Year)
tubes. We have started testing and installing the new hardware on the
Potchefstroom IGY NM, and have not encountered any major issues. Based on our
experience, we are therefore confident that the upgrades described in this
paper can be used to rejuvenate the current ailing NM network, and even bring
older, decommissioned, NMs back on-line. This can help with efforts to expand
the current world-wide network of NMs (see e.g. Mishev & Usoskin, 2020).

A 1NMD (neutron moderated detector, i.e. a NM without the lead producer, also
referred to as a “bare” NM) was installed in at SANAE in 1971 and upgraded to
a 4NMD in 1973. Currently, this detector is operated as a 3NMD, upgraded with
the same electronics as discussed in this paper in early 2020. We are
currently gathering calibration data and expect measurement to become publicly
available later in 2021. Live data from the different tubes are, however,
already available at e.g.
https://fskbhe1.puk.ac.za/spaceweather/sanae_tube_1.html for tube 1, with
similar links to tubes 2 and 3.

## Acknowledgement

Oulu data obtained from http://cosmicrays.oulu.fi, courtesy of the Sodankylä
Geophysical Observatory. Figures prepared with Matplotlib Hunter (2007) and
certain calculations done with NumPy (Harris et al., 2020). This work is based
on the research supported in part by the National Research Foundation of South
Africa (NRF grant numbers: 119424, 120847, and 120345). Support from the NRF’s
South African National Antarctic Programme (SANAP; grant number 110741) is
especially acknowledged. Opinions expressed and conclusions arrived at are
those of the authors and are not necessarily to be attributed to the NRF.

##  Appendix A Tube degradation

As mentioned in the main text, during the upgrading of the SANAE NM, several
of the older NM tubes were discarded as they either provided no, or very weak
pulses. The tubes were transported back to Potchefstroom and investigated in
more detail. Of course, there are several reason for a tube to malfunction and
completely stop registering pulse, including physical damage to the central
wire (the integrity of the central wires can be confirmed by measuring the
impedance of the tube) or a broken gas seal, leading to escape of the BF3 gas.
However, based on the results of Sec. 3.6, we are also interested in finding
possible reasons for the steady decline of the count rate of tube with
apparently no physical damage. The central wire of the broken/inefficient
tubes were removed and examined under a scanning electron microscope for any
discrepancies. We noted several that the central wires were non-uniformly
encrusted with crystal structures. An example is shown in Fig. 10 under
different levels of magnification. In the middle panel, we identified three
possible different crystal structures; a very coarse structure (labelled 1), a
finer structure (labelled 2), and a thin layer of plating (labelled 3).

Fig. 10: An example of the crystal structures found on several central wires.

To identify the crystal structures, the samples were analyzed using energy-
dispersive X-ray spectroscopy (EDS). However, this method cannot readily
identify light elements in a sample due to the lack of characteristic X-ray
emission. Results of the EDS analysis is presented in Fig. 11 for three
different tubes (other samples were very similar), with the composition
results shown in the left panels, and the magnified samples in the right
panels.

Fig. 11: Results of the composition analysis of the crystal structures found
on several central wires.

For Tube #1 (top panels), spectrum #5 shows the results of a relatively
“clean” central wire consisting of $>80\%$ Tungsten (the green curve).
Analysis of the larger crystal structures covering parts of both Tube #1 and
#3 (middle panel) showed high concentrations of B and O. BF3 gas is extremely
reactive with water and/or moisture, leading to the following reaction

$4\mathrm{BF}_{3}+2\mathrm{H}_{2}\mathrm{O}\rightarrow
3\mathrm{HBF}_{4}+\mathrm{B(OH)}_{3}$ (14)

which forms two acid compounds. Visually, the BF3 gas turns white on contact
with moisture. The analysis for Tubes #1 and #3 seem to indicate the possible
presence of boric acid (B(OH)3) crystals on the central wire. We can therefore
conclude that these tubes malfunctioned due to a broken seal allowing BF3 gas
to escape and moisture to penetrate the tube.

The smaller crystal structure observed on the central wire of Tube #2 (bottom
panels), however, show large concentration of F, which is a by-product of
neutron capture process

${}^{10}\mathrm{BF}_{3}+\mathrm{n}\rightarrow\,^{7}\mathrm{Li}^{+}+\alpha^{+2}+3\mathrm{F}^{-}$
(15)

The finer crystal structures present in the samples could therefore be LiF
crystals formed during the neutron capture process, keeping in mind that the
EDS technique cannot measure the lighter Li element. However, this can also be
F compounds as F is very reactive and could bind with and form compounds with
a range of elements present in the tubes.

The results presented in this section seem to confirm the statement by Bieber
et al. (2007) that material formed during neutron detection process can “plate
out” onto the central wire and decrease the detection efficiency of the tube.
This is most likely BF3 compounds released in the cascade, with trace amounts
of material formed directly in the neutron capture reaction.

## References

  * Balabin et al. (2011) Balabin, Y. V., Gvozdevsk, B. B., Maurchev, E. A., Vashenyuk, E. V., & Dzhappuev, D. D. (2011).  Fine structure of neutron multiplicity on neutron monitors.  Astrophysics and Space Sciences Transactions, 7(3), 283–286. doi:10.5194/astra-7-283-2011.
  * Bieber et al. (2007) Bieber, J. W., Clem, J., Desilets, D., Evenson, P., Lal, D., Lopate, C., & Pyle, R. (2007).  Long-term decline of South Pole neutron rates.  Journal of Geophysical Research (Space Physics), 112(A12), A12102. doi:10.1029/2006JA011894.
  * Bieber et al. (2004) Bieber, J. W., Clem, J. M., Duldig, M. L., Evenson, P. A., Humble, J. E., & Pyle, R. (2004).  Latitude survey observations of neutron monitor multiplicity.  Journal of Geophysical Research (Space Physics), 109(A12), A12106. doi:10.1029/2004JA010493.
  * Bütikofer (2018) Bütikofer, R. (2018).  Ground-based measurements of energetic particles by neutron monitors.  In O. E. Malandraki, & N. B. Crosby (Eds.), Solar Particle Radiation Storms Forecasting and Analysis: The HESPERIA HORIZON 2020 Project and Beyond (pp. 95–111).  Springer International Publishing.
  * Caballero-Lopez (2016) Caballero-Lopez, R. A. (2016).  An estimation of the yield and response functions for the mini neutron monitor.  Journal of Geophysical Research (Space Physics), 121(8), 7461–7469. doi:10.1002/2016JA022690.
  * Carmichael et al. (1968) Carmichael, H., Bercovitch, M., Shea, M. A., Magidin, M., & Peterson, R. W. (1968).  Attenuation of neutron monitor radiation in the atmosphere.  Canadian Journal of Physics Supplement, 46, S1006–S1013. doi:10.1139/p68-405.
  * Clem & Dorman (2000) Clem, J. M., & Dorman, L. I. (2000).  Neutron Monitor Response Functions.  Space Science Reviews, 93, 335–359. doi:10.1023/A:1026508915269.
  * Desorgher et al. (2009) Desorgher, L., Kudela, K., Flückiger, E., Bütikofer, R., Storini, M., & Kalegaev, V. (2009).  Comparison of Earth’s magnetospheric magnetic field models in the context of cosmic ray physics.  Acta Geophysica, 57(1), 75–87. doi:10.2478/s11600-008-0065-3.
  * Dorman (2004) Dorman, L. I. (2004).  Cosmic Rays in the Earth’s Atmosphere and Underground.  Netherlands: Springer.
  * Evenson et al. (2005) Evenson, P., Bieber, J. W., Clem, J., & Pyle, R. (2005).  Neutron Monitor Temperature Coefficients: Measurements for BF3 and 3He Counter Tubes.  In 29th International Cosmic Ray Conference (ICRC29), Volume 2 (p. 485).  volume 2 of International Cosmic Ray Conference.
  * Fowler (1963) Fowler, I. L. (1963).  Very Large Boron Trifluoride Proportional Counters.  Review of Scientific Instruments, 34(7), 731–739. doi:10.1063/1.1718559.
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., & Oliphant, T. E. (2020).  Array programming with NumPy.  Nature, 585, 357–362. doi:10.1038/s41586-020-2649-2.
  * Hatton & Carimichael (1964) Hatton, C. J., & Carimichael, H. (1964).  Experimental Investigation of the NM-64 Neutron Monitor.  Canadian Journal of Physics, 42(12), 2443–2472. doi:10.1139/p64-222.
  * Hunter (2007) Hunter, J. D. (2007).  Matplotlib: A 2d graphics environment.  Computing in Science & Engineering, 9(3), 90–95. doi:10.1109/MCSE.2007.55.
  * Kato et al. (2021) Kato, C., Kihara, W., Ko, Y., Kadokura, A., Kataoka, R., Evenson, P., Uchida, S., Kaimi, S., Nakamura, Y., Uchida, H. A., Murase, K., & Munakata, K. (2021).  New cosmic ray observations at Syowa Station in the Antarctic for space weather study.  arXiv e-prints, (p. arXiv:2101.09887). arXiv:2101.09887.
  * Knoll (2010) Knoll, G. F. (2010).  Radiation Detection and Measurement, 4th ed..  New York, NY: Wiley.
  * Krüger et al. (2008) Krüger, H., Moraal, H., Bieber, J. W., Clem, J. M., Evenson, P. A., Pyle, K. R., Duldig, M. L., & Humble, J. E. (2008).  A calibration neutron monitor: Energy response and instrumental temperature sensitivity.  Journal of Geophysical Research (Space Physics), 113(A8), A08101. doi:10.1029/2008JA013229.
  * Krüger et al. (2017) Krüger, P., Krüger, H., Krüger, H., Diedericks, C., & Malan, D. (2017).  New, affordable, open-hardware neutron monitor electronics.  In Proceedings of 35th International Cosmic Ray Conference — PoS(ICRC2017) (p. 054).  volume 301.  doi:10.22323/1.301.0054.
  * Malan & Moraal (2002) Malan, N., & Moraal, H. (2002).  The effect of wind on pressure correction of the SANAE neutron monitor counting rate.  South African Journal of Science, 98(5), 278--281.
  * Mangeard et al. (2016) Mangeard, P. S., Ruffolo, D., Sáiz, A., Nuntiyakul, W., Bieber, J. W., Clem, J., Evenson, P., Pyle, R., Duldig, M. L., & Humble, J. E. (2016).  Dependence of the neutron monitor count rate and time delay distribution on the rigidity spectrum of primary cosmic rays.  Journal of Geophysical Research (Space Physics), 121(12), 11,620--11,636. doi:10.1002/2016JA023515.
  * Medina et al. (2013) Medina, J., Blanco, J. J., García, O., Gómez-Herrero, R., Catalán, E. J., García, I., Hidalgo, M. A., Meziat, D., Prieto, M., Rodríguez-Pacheco, J., & Sánchez, S. (2013).  Castilla-La Mancha neutron monitor.  Nuclear Instruments and Methods in Physics Research A, 727, 97--103. doi:10.1016/j.nima.2013.06.028.
  * Mishev & Usoskin (2020) Mishev, A., & Usoskin, I. (2020).  Current status and possible extension of the global neutron monitor network.  Journal of Space Weather and Space Climate, 10, 17. doi:10.1051/swsc/2020020. arXiv:2005.12621.
  * Mishev et al. (2020) Mishev, A. L., Koldobskiy, S. A., Kovaltsov, G. A., Gil, A., & Usoskin, I. G. (2020).  Updated Neutron-Monitor Yield Function: Bridging Between In Situ and Ground-Based Cosmic Ray Measurements.  Journal of Geophysical Research (Space Physics), 125(2), e27433. doi:10.1029/2019JA027433.
  * Moraal et al. (2011) Moraal, H., Stoker, P., Humble, J., & Mans, A. (2011).  Long-term Data Records of South African Neutron Monitors.  In International Cosmic Ray Conference (p. 187).  volume 11 of International Cosmic Ray Conference.  doi:10.7529/ICRC2011/V11/0475.
  * Paschalis et al. (2013) Paschalis, P., Mavromichalaki, H., Yanke, V., Belov, A., Eroshenko, E., Gerontidou, M., & Koutroumpi, I. (2013).  Online application for the barometric coefficient calculation of the NMDB stations.  New Astronomy, 19, 10--18. doi:10.1016/j.newast.2012.08.003.
  * Ruffolo et al. (2016) Ruffolo, D., Sáiz, A., Mangeard, P. S., Kamyan, N., Muangha, P., Nutaro, T., Sumran, S., Chaiwattana, C., Gasiprong, N., Channok, C., Wuttiya, C., Rujiwarodom, M., Tooprakai, P., Asavapibhop, B., Bieber, J. W., Clem, J., Evenson, P., & Munakata, K. (2016).  Monitoring Short-term Cosmic-ray Spectral Variations Using Neutron Monitor Time-delay Measurements.  Astrophysical Journal, 817(1), 38. doi:10.3847/0004-637X/817/1/38.
  * Sáiz et al. (2019) Sáiz, A., Mitthumsiri, W., Ruffolo, D., Evenson, P., & Nutaro, T. (2019).  Detecting Single and Multiple Atmospheric Secondaries in an 18NM64.  In Proceedings of 36th International Cosmic Ray Conference — PoS(ICRC2019) (p. 1145).  volume 358.  doi:10.22323/1.358.1145.
  * Similä et al. (2021) Similä, M., Usoskin, I., Poluianov, S., Mishev, A., Kovaltsov, G. A., & Toit Strauss, D. (2021).  High-altitude polar NM with the new DAQ system as a tool to study details of the cosmic-ray induced nucleonic cascade.  arXiv e-prints, (p. arXiv:2104.04727). arXiv:2104.04727.
  * Simpson (2000) Simpson, J. A. (2000).  The Cosmic Ray Nucleonic Component: The Invention and Scientific Uses of the Neutron Monitor - (Keynote Lecture).  Space Science Reviews, 93, 11--32. doi:10.1023/A:1026567706183.
  * Strauss et al. (2020) Strauss, D. T., Poluianov, S., van der Merwe, C., Krüger, H., Diedericks, C., Krüger, H., Usoskin, I., Heber, B., Nndanganeni, R., Blanco-Ávalos, J., García-Tejedor, I., Herbst, K., Caballero-Lopez, R., Moloto, K., Lara, A., Walter, M., Giday, N. M., & Traversi, R. (2020).  The mini-neutron monitor: a new approach in neutron monitor design.  Journal of Space Weather and Space Climate, 10, 39. doi:10.1051/swsc/2020038.