# Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional
Encoder-Decoder

Jeffrey M. Ede

<EMAIL_ADDRESS>

Abstract: We present an atrous convolutional encoder- decoder trained to
denoise 512$\times$512 crops from electron micrographs. It consists of a
modified Xception backbone, atrous convoltional spatial pyramid pooling module
and a multi-stage decoder. Our neural network was trained end-to-end to remove
Poisson noise applied to low-dose ($\ll 300$ counts ppx) micrographs created
from a new dataset of 17267 2048$\times$2048 high-dose ($>$ 2500 counts ppx)
micrographs and then fine-tuned for ordinary doses (200-2500 counts ppx). Its
performance is benchmarked against bilateral, non-local means, total
variation, wavelet, Wiener and other restoration methods with their default
parameters. Our network outperforms their best mean squared error and
structural similarity index performances by 24.6% and 9.6% for low doses and
by 43.7% and 5.5% for ordinary doses. In both cases, our network’s mean
squared error has the lowest variance. Source code and links to our new high-
quality dataset and trained network have been made publicly available at
https://github.com/Jeffrey-Ede/Electron-Micrograph-Denoiser.

Keywords: deep learning, denoising, electron microscopy, low dose

##  1 Introduction

Every imaging mode in electron microscopy is limited by and has been shaped by
noise[1]. Increasingly, ever more sophisticated and expensive hardware and
software based methods are being developed to increase resolution, including
aberration correctors[2, 3], advanced cold field emission guns[4, 5],
holography[6, 7] and others[8, 9, 10]. However, these developments are all
fundamentally limited by the signal-to-noise ratios in the micrographs they
are being applied to. Low-dose applications such as single-particle cryogenic
microscopy[11] and real-time tomography[9] are also complicated, made
preventively difficult or limited by noise.

Moving towards higher resolution, a large number of general[12] and electron
microscopy-specific[1, 13] denoising algorithms have been developed. However,
most of these algorithms rely on laboriously hand-crafted filters and are
rarely; if ever, truly optimized for their target domains e.g. [14]. Neural
networks are universal approximators[15] that overcome these difficulties[16]
through representation learning[17]. As a result, networks are increasingly
being applied to noise removal[18, 19, 20, 21] and other applications in
electron microscopy[22, 23, 24, 25] and other areas of science.

The recent success of large neural networks in computer vision is attributed
to the advent of graphical processing unit (GPU) acceleration[26, 27]. In
particular, the GPU acceleration of large convolutional neural networks[28,
29] (CNNs) in distributed settings[30, 31]. Large networks that surpass human
performance in image classification[32, 33], computer games[34, 35, 36],
speech recognition[37, 38], relational reasoning[39] and in many other
applications[23, 40, 41, 42, 43] have all been enabled by GPUs.

At the time of writing, there are no large neural networks for electron
micrograph denoising. Instead, most denoising networks act on many small
overlapping crops e.g. [20]. This makes them computationally inefficient and
unable to utilize all the information available. Some large denoising networks
have been trained as part of generative adversarial networks[44] and try to
generate micrographs resembling high-quality training data as closely as
possible. This can avoid the blurring effect of most filters; however, this is
achieved by generating features that might be in high-quality micrographs.
This means that they are prone to producing artifacts; something that is often
undesirable in scientific applications.

This paper presents a large CNN for electron micrograph denoising. Network
architecture is shown in section 2. The collation of a new high-quality
electron micrograph training dataset, training hyperparameters and learning
protocols are described in section 3. Performance comparison with other
methods’, error characterization and example usage are in section 4. Finally,
architecture and hyperparameter tuning experiments are presented in section 5.

Figure 1:  Architecture of our deep convolutional encoder-decoder for electron
micrograph denoising. The entry and middle flows develop high-level features
that are sampled at multiple scales by the atrous spatial pyramid pooling
module. This produces rich semantic information that is concatenated with low-
level entry flow features and resolved into denoised micrographs by the
decoder.

##  2 Architecture

Our highest performing denoising network was trained for 512$\times$512 inputs
and is shown in fig. 1. It consists of modified Xception[45] entry and middle
flows for feature extraction, an atrous spatial pyramid pooling (ASPP)
module[46, 47] that samples rich high-level semantics at multiple scales and a
multi-stage decoder that combines low-level entry flow features with ASPP
semantics to resolve them. The architecture is inspired by Google’s
DeepLab3[46], DeepLab3+[47] and other encoder-decoder architectures[18, 48,
20]. A guide for readers unfamiliar with convolution arithmetic for deep
learning is [49].

Convolutions all use 3$\times$3 or 1$\times$1 kernels and are followed by
batch normalization[50] before ReLU6[51] activation. Similar to DeepLab3+,
original Xception max pooling layers have been replaced with strided depthwise
separable convolutions, enabling the network to learn its own downsampling.
Extra batch normalization is added between the depthwise and pointwise
convolutions of every depthwise separable convolution like in MobileNet[52].

The following subsections correspond to the subsections of fig. 1. Training
details follow in section 3.

###  2.1 Entry Flow

Figure 2:  Start of an unmodified Xception entry flow[45].

Other than the modification of max pooling layers to strided depthwise
separable convolutions, most entry flow convolutions are similar to
Xception’s[45]. Common convolutions have the same number of features and are
arranged into Xception-style residual[53] blocks to reduce semantic decimation
during downsampling[46].

The main change is the replacement of the first two Xception convolutions with
an residual convolutional downsampling block. For reference, the start of the
original Xception entry flow is shown in fig. 2. This change was made so that
the lowest-level features concatenated in the decoder would be deeper and have
a larger feature space. An alternative solution is to apply additional pre-
concatenation 1$\times$1 convolutions to change the feature space size. This
was the approach taken in DeepLab3+[47] to prevent high-level ASPP semantics
being overwhelmed by low-level features in the decoder.

###  2.2 Middle Flow

Skip-3 residual blocks are repeated 12 times to develop high-level semantics
to flow into the ASPP module. This is more than the 8 skip-3 blocks used in
Xception[45] as the inflowing tensors are larger; 32$\times$32$\times$728
rather than 19$\times$19$\times$728\. Nevertheless, this is fewer than the 16
skip-3 residual blocks used in DeepLab3+ for similar tensors[47]. Middle flow
tensors are much smaller than those in other parts of the network so this
decision was a compromise between expressibility and training time.

###  2.3 Atrous Spatial Pyramid Pooling

This is the ASPP module Google developed for semantic image segmentation[46,
47] without a pre-pooling 1$\times$1 convolution. It is being used rather than
a fully connected layer; like the one in [18]’s denoiser, to sample semantics
at multiple scales as it requires fewer parameters and has almost identical
performance. The atrous rates of 6, 12 and 18 are the same as those used in
the original ASPP module[46]. Importantly, a bottleneck 1$\times$1 convolution
is used to reduce the number of output features from 3640 to 256, forcing the
network to work harder to develop high-level semantics.

###  2.4 Decoder

Rich ASPP semantics are bilinearly upsampled from 32$\times$32$\times$256 to
128$\times$128$\times$256 feature maps and concatenated with low-level entry
flow features to resolve them in the following convolutions. This is similar
to the approach used in other encoder-decoder architectures[46, 47, 18, 48,
20]. There are two residual concatenations with low-level features from the
entry flow; rather than one, so that semantics can be resolved from the first
concatenation before being resolved into fine spatial information after the
second.

To prevent rich semantic information from being overwhelmed by low-level
features, feature sizes are chosen so that, disregarding spatial dimensions,
the same number of low-level and high-level features enter the decoder. The
optimal ratio is unknown; however, this paper and Google’s work on semantic
segmentation[46, 47] establish that a 1:1 ratio works for multiple domains.
The resolved features are transpositionaly convoluted back to the size of the
original image; rather than resolved at that scale, to introduce another
bottleneck. This forces the network to work harder to develop meaningful
features to resolve recovered micrographs from in the final convolutions.

##  3 Training

Figure 3:  Mean squared error (MSE) losses of our neural network during
training on low dose ($\ll 300$ counts ppx) and fine-tuning for ordinary doses
(200-2500 counts ppx). Learning rates (LRs) and the freezing of batch
normalization are annotated. Validation losses were calculated using 1
validation example after every 5 training batches.

In this section, we discuss training with the TensorFlow[31] deep learning
framework. Training was performed using ADAM[54] optimized synchronous
stochastic gradient decent[30] with 1 replica network on each of 2 Nvidia GTX
1080 Ti GPUs.

###  3.1 Data Pipeline

A new dataset of 17267 2048$\times$2048 high-quality electron micrographs
saved to University of Warwick data servers over several years was collated
for training. Here, high-quality refers to 2048$\times$2048 micrographs with
mean counts per pixel above 2500. The dataset was collated from individual
micrographs made by dozens of scientists working on hundreds of projects and
therefore has a diverse constitution. It has been made publicly available as a
set of TIFFs[55] via https://github.com/Jeffrey-Ede/Electron-Micrograph-
Denoiser.

The dataset was split into 11350 training, 2431 validation and 3486 test
micrographs. For training, each micrograph was downsized by a factor of 2
using area interpolation to 1024$\times$1024\. This increased mean counts per
pixels above 10000, corresponding to signal-to-noise ratios above 100:1. Next,
512$\times$512 crops were taken at random positions and subject to a random
combination of flips and 90$\degree$ rotations to augment the dataset by a
factor of 8. Each crop was then linearly transformed to have values between
0.0 and 1.0.

To train the network for low doses, Poisson noise was applied to each crop
after scaling by a number sampled from an exponential distribution with
probability density function (PDF)

$f\left(x,\frac{1}{\beta}\right)=\frac{1}{\beta}\exp\left(-\frac{x}{\beta}\right).$
(1)

We chose $\beta=75.0$ and offset the numbers sampled by 25.0 before scaling
crops with them. These numbers and distribution are arbitrary and were chosen
to expose the network to a continuous range of noise levels where most are
very noisy. After noise application, ground truth crops were scaled to have
the same means as their noisy counterparts.

After being trained for low-dose applications, the network was fine-tuned for
ordinary doses by training it on crops scaled by numbers uniformly distributed
between 200 and 2500.

###  3.2 Learning Policy

In this subsection, we discuss our training hyperparameters and learning
protocol for the learning curve shown in fig. 3.

Loss metric: Our network was trained to minimize the Huberised[56] scaled mean
squared error (MSE) between denoised and high-quality crops:

$L=\begin{cases}1000\,\textit{MSE},&1000\,\textit{MSE}<1.0\\\
\left(1000\,\textit{MSE}\right)^{\frac{1}{2}},&1000\,\textit{MSE}\geq
1.0\end{cases}$ (2)

The loss is Huberized to prevent the network from being too disturbed by
batches with especially noisy crops. To surpass our low-dose performance
benchmarks, our network had to achieve a MSE lower than $7.5\times 10^{-4}$,
as tabulated in table 1. Consequently, MSEs were scaled by 1000 to limit
trainable parameter perturbations by MSEs larger than $1.0\times 10^{-3}$.
More subtly, this also increased our network’s effective learning rate by a
factor of 1000.

Our network was trained without clipping its outputs between 0.0 and 1.0.
Clipping is only applied optionally during inference. Nevertheless, as
clipping is desirable in most applications, all performance statistics;
including losses during training, are reported for clipped outputs.

Batch normalization: Batch normalization layers from [50] were trained with a
decay rate of 0.999 for 134108 batches. Batch normalization was then frozen
for the rest of training. Batch normalization significantly reduced grid-like
increases in error in our output images. As a result, it was not frozen until
the instability introduced by varying batch normalization parameters
noticeably limited convergence.

Optimizer: ADAM[54] optimization was used with a stepped learning rate. For
the low dose version of the network, we used a learning rate of
1.0$\times$10-3 for 134108 batches, 2.5$\times$10-4 for another 17713 batches
and then 1.0$\times$10-4 for 46690 batches. The network was then fine-tuned
for ordinary doses using a learning rate of 2.5$\times$10-4 for 16773 batches,
then 1.0$\times$10-4 for 17562 batches. The unusual numbers of batches are a
result of learning rates being adjusted after saving variables; something that
happened at wall clock times.

We found the recommended[54, 31] ADAM decay rate for the 1st moment of the
momentum, $\beta_{1}=0.9$, to be too high and chose $\beta_{1}=0.5$ instead.
This lower $\beta_{1}$ made training more resistant to varying noise levels in
batches. Our experiments with $\beta_{1}$ are discussed in section 5.

Regularization: L2 regularization[57] was applied by adding 5$\times$10-5
times the quadrature sum of all trainable variables to the loss function. This
prevented trainable parameters growing unbounded, decreasing their ability to
learn in proportion[58]. Importantly, this ensures that our network will
continue to learn effectively if it is fine-tuned or given additional
training. We did not perform an extensive search for our regularization rate
and think that 5$\times$10-5 may be too high.

Activation: All neurons are ReLU6[51] activated. Other activations are
discussed in section 5.

Initialization: All weights were Xavier[59] initialized. Biases were zero
initialized.

##  4 Performance

In this section, our network’s MSE and structural similarity index (SSIM)
performance is benchmarked other methods’ and its mean error per pixel is
mapped. We also present example applications of our network to noisy electron
microscopy and scanning transmission electron microscopy (STEM) images.

###  4.1 Benchmarking

Figure 4:  Gaussian kernel density estimated mean squared error (MSE) and
structural similarity index (SSIM) probability density functions (PDFs) for
denoising methods applied to 20000 instances of Poisson noise. To ease
comparison, the highest values in each MSE and SSIM PDF set has been scaled to
1.0. Only the starts of MSE PDFs are shown.

To benchmark our network’s performance, we applied it and 9 popular denoising
methods to 20000 instances of Poisson noise applied to 512$\times$512 crops
from test set micrographs using the method in section 3.1. This benchmarking
is done for the low-dose version of our network and the version fine-tuned for
ordinary doses. Implementation details for each denoising methods follow.

| Low Dose, $\ll$ 300 counts ppx | Ordinary Dose, 200-2500 counts ppx
---|---|---
| MSE $\left(\times 10^{-3}\right)$ | SSIM | MSE $\left(\times 10^{-3}\right)$ | SSIM
Method | Mean | Std Dev | Mean | Std Dev | Mean | Std Dev | Mean | Std Dev
Unfiltered | 4.357 | 2.558 | 0.454 | 0.208 | 0.508 | 0.682 | 0.850 | 0.123
Gaussian | 0.816 | 0.452 | 0.685 | 0.159 | 0.344 | 0.334 | 0.878 | 0.087
Bilateral | 1.025 | 1.152 | 0.574 | 0.261 | 1.243 | 1.392 | 0.600 | 0.271
Median | 1.083 | 0.618 | 0.618 | 0.171 | 0.507 | 0.512 | 0.821 | 0.126
Wiener | 1.068 | 0.546 | 0.681 | 0.137 | 0.402 | 0.389 | 0.870 | 0.085
Wavelet | 0.832 | 0.58 | 0.657 | 0.186 | 0.357 | 0.312 | 0.875 | 0.085
Chambolle TV | 0.746 | 0.725 | 0.686 | 0.192 | 0.901 | 0.909 | 0.674 | 0.217
Bregman TV | 1.109 | 1.031 | 0.544 | 0.268 | 4.074 | 3.025 | 0.348 | 0.312
NL means | 2.924 | 2.338 | 0.357 | 0.315 | 1.403 | 1.266 | 0.545 | 0.281
Neural network | 0.562 | 0.449 | 0.752 | 0.147 | 0.201 | 0.169 | 0.926 | 0.057
Table 1: Means and standard deviations of mean squared error (MSE) and
structural similarity index (SSIM) test set performances of denoising methods
for 20000 instances of Poisson noise. Noise removal methods were implemented
with default parameters, as described in the main text. TV - total variation.
NL - non-local.

Unfiltered: For reference, MSE and SSIM statistics were collected for noisy
crops without any denoising method being applied to recover them.

Gaussian filter: OpenCV[60] default implementation of Gaussian blurring for a
3$\times$3 kernel.

Bilateral filter: OpenCV[60] implemented bilateral filtering[61]. We used a
default 9 pixel neighborhoods with radiometric and spatial scales of 75. These
scales are a compromise between small scales; less than 10, that have little
effect and large scales; more than 150, that cartoonize images.

Median filter: OpenCV[60] implemented 3$\times$3 median filter.

Wiener: Scipy[62] implementation of Wiener filtering.

Wavelet denoising: Scikit-image[63] implementation of BayesShrink adaptive
wavelet soft-thresholding[64]. The noise standard deviation used to compute
wavelet detail coefficient thresholds was estimated from the data using the
method in [65].

Chambolle total variation denoising: Scikit-image[63] implementation of
Chambolle’s iterative total-variation denoising algorithm[66]. We used the
scikit-image default denoising weight of 0.1 and ran the algorithm until
either the fractional change in the algorithm’s cost function was less than
$2.0\times 10^{-4}$ or it reached 200 iterations.

Bregman total variation denoising: Scikit-image[63] implementation of split
Bregman anisotropic total variation denoising[67, 68]. We used a denoising
weight of 0.1 and ran the algorithm until either the fractional change in the
algorithm’s cost function was less than $2.0\times 10^{-4}$ or it reached 200
iterations

Non-local means: Scikit-image[63] implementation of texture-preserving non-
local means denoising[69].

MSE and SSIM performances were divided into 200 equispaced bins in [0.0, 1.2]
$\times$ $10^{-3}$ and [0.0, 1.0], respectively, for both low and ordinary
doses. Performance PDFs shown in fig. 4 are Gaussian kernel density
estimated[70, 71] (KDE) from these. KDE bandwidths were found using Scott’s
Rule[72]. To ease comparison, PDFs for each set of performance statistics are
scaled so the highest value in each set is 1.0. The scale factors are
6.07$\times$10-4 for low dose MSE, 2.99$\times$10-1 for low dose SSIM,
3.03$\times$10-4 for ordinary dose MSE and 9.17$\times$10-2 for ordinary dose
SSIM.

Performance statistics are also summarized in table 1. This shows that our
network outperforms the mean MSE and SSIM performance of every other method
for both low and ordinary doses. Its MSE variance is also lower for both low
and ordinary doses.

Figure 5:  Mean absolute errors of our low and ordinary dose networks’
512$\times$512 outputs for 20000 instances of Poisson noise. Contrast limited
adaptive histogram equalization[73] has been used to massively increase
contrast, revealing grid-like error variation. Subplots show the top-left
16$\times$16 pixels’ mean absolute errors unadjusted. Variations are small and
errors are close to the minimum everywhere, except at the edges where they are
higher. Low dose errors are in [0.0169, 0.0320]; ordinary dose errors are in
[0.0098, 0.0272].

###  4.2 Network Error

Mean absolute errors for each pixel of our network’s output are shown for low
and ordinary doses in fig. 5. The errors are uniform almost everywhere, except
at the edges where they are higher. For low and ordinary doses, the mean
absolute errors per pixel are 0.0177 and 0.0102, respectively.

Small, grid-like variations in absolute error are revealed by contrasted
limited adaptive histogram equalization[73] in fig. 5. These variations are
common in deep learning and are often associated with transpositional
convolutions. Consequently, some authors[74] have recommended their
replacement with bilinear upsampling then convolution. We tried this; however,
it only made the errors less grid-like.

Instead, we found batch normalization to be the best way to reduce structured
error variation. This is demonstrated by errors being more grid-like for the
ordinary dose version of our network, which was trained for longer after batch
normalization was frozen. Consequently, batch normalization was not frozen
until the instability introduced by its trainable variables noticeably limited
convergence.

###  4.3 Example Usage

We provide several example applications of our low-dose network to
512$\times$512 crops from high-quality electron micrographs that we applied
noise to in fig 6 and fig. 7. Several example applications of our low-dose
network to 512$\times$512 crops from scanning transmission electron
micrography (STEM) images are also presented in fig. 8. Our network has not
been trained for STEM so this demonstrates its ability to generalize to other
domains.

Figure 6:  Example applications of the noise-removal network to instances of
Poisson noise applied to 512$\times$512 crops from high-quality micrographs.
Enlarged 64$\times$64 regions from the top left of each crop are shown to ease
comparison.  Figure 7:  More example applications of the noise-removal network
to instances of Poisson noise applied to 512$\times$512 crops from high-
quality micrographs. Enlarged 64$\times$64 regions from the top left of each
crop are shown to ease comparison.  Figure 8:  Example applications of our
neural network to 512$\times$512 crops from scanning transmission electron
micrographs. Enlarged 64$\times$64 regions from the top left of each crop are
shown to ease comparison. Our network has not been trained for this domain.

Our neural network is designed to be simple and easy to use. An example of our
network being loaded once and used for inference multiple times in python is

⬇

>>> from denoiser import Denoiser

>>> noise_remover = Denoiser()

>>> restored_img1 = noise_remover.denoise(img1)

>>> restored_img2 = noise_remover.denoise(img2)

Under the hood, our program divides images larger than 512$\times$512 into
slightly overlapping 512$\times$512 crops that can be processed by our
network. This gives higher accuracy than using non-overlapping crops as our
network has much higher errors for the couple of pixels near the edges of its
outputs. Reflection padding is applied to images before cropping to reduce
errors at their edges after restoration. Users can customize the amount
overlap, padding and many other options or use default values we have chosen.

We speed-tested our network by applying it to 20000 512$\times$512 images with
1 external GTX 1080 Ti GPU and 1 thread of a i7-6700 processor. It has a mean
batch size 1 (worst case) inference time of 77.0 ms. It also takes a few
seconds to load before it is ready for repeated use.

##  5 Additional Experiments

As part of development, we experimented with multiple architectures and their
learning policies. Initially, we experimented with shallower architectures
similar to [18, 20] and [21]; however, these struggled to meet Chambolle’s
low-dose benchmark in table 1. Consequently, we switched to the deeper
Xception-based architecture presented in this paper. In this section we
present some of the experiments we performed to fine-tune it.

Figure 9: Batch size 2 mean squared error (MSE) learning curves for various
training hyperparameters and activation functions. Training is most stable for
ADAM[54] $\beta_{1}=0.2$ without batch renormalization[75] and with ReLU6[51];
rather than ReLU[76], activation. Figure 10: Batch size 10 mean squared error
(MSE) learning curves for training with and without clipping, extra structural
similarity index[77] (SSIM) based losses and RMSProp[78] rather than ADAM [54]
optimization.

###  5.1 Batch Size 2

Initial experiments with a batch size of 2 are summarized in fig. 9. This
includes experiments with different decay rates for the 1st moment of the
momentum, $\beta_{1}$, used by the ADAM solver. Our results show that training
converges faster and more stably for $\beta_{1}=0.2$ than the recommended[54,
31] $\beta_{1}=0.9$. $\beta_{1}=0.9$ being too high seems to be a theme in
high resolution, low batch size applications e.g. [79].

Our batch size 2 experiments also revealed that training is faster and more
stable with ReLU6[51] than with ReLU[76] activation. We think this is because
ReLU6 is especially effective in the decoder where it does not allow sparse
activations to grow unbounded. Only ReLU-based activation functions were
considered as the non-saturation of their gradients accelerates the
convergence of stochastic gradient descent[80].

Batch renormalization[75] was trailed as our limited hardware made large batch
sizes prohibitive. We also wanted to test how well it would work with batch
size 2 as we were unable to find quantitative information for this batch size.
It is ineffective: Fig. 9 shows that the extra trainable variables it
introduces destabilize training and reduce convergence. This is not surprising
as it is not designed for batches this small and can take time to become
effective. Additionally, instability introduced by applying different amounts
noise to training examples may be adding to the inherent batch renormalization
instability, making it much worse than it might be for other optimizations.

###  5.2 Batch Size 10

We tried adding multiples of the distance from the maximum SSIM,
$1.0-\textit{SSIM}$, to the training loss to optimize our network for an
additional metric. As shown by fig. 10, adding different multiples does not
have a significant effect on the rate of MSE convergence. Regardless, we
decided against adding this loss as SSIMs are weighted to measure a human
perception of quality and we did not want to introduce this slight bias.
Nevertheless, we have shown that additionally optimizing the SSIM does not
significantly affect training.

Outputs were clipped between 0.0 and 1.0 before calculating training losses in
all of our batch size 2 and the first of our batch size 10 experiments shown
in fig. 10. Removing this clipping significantly decreases the rate of
convergence. Nonetheless, we decided to remove clipping as the network was
especially prone to producing artifacts close to 0.0 and 1.0. This ensured
that these artifacts were not present in the fully trained network’s outputs.

Following the success of lower $\beta_{1}$ with the ADAM optimizer, we tried
other momentum-based optimizers. The start of a learning curve for simple
RMSProp[78] is shown in fig. 10. It shows that training is significantly less
stable and has a lower rate of convergence. We also experimented with
Nesterov-accelerated gradient descent[81, 82] with similar results.

##  6 Summary

  * •

We have developed a deep neural network for electron micrograph denoising and
shown that it outperforms existing methods for low and ordinary electron
doses.

  * •

Fully trained versions of our network have been made available for low and
ordinary doses with example usage: https://github.com/Jeffrey-Ede/Electron-
Micrograph-Denoiser

  * •

A new dataset of 17267 2048$\times$2048 high-quality micrographs collected to
train our network has been made publicly available.

  * •

Example applications of our network to noisy electron micrographs are
presented. We also present example application to STEM images to show that our
network can generalize to other domains.

  * •

Our network architecture, training hyperparameters and learning protocols are
detailed. In addition, details of several of our initial architecture and
learning policy experiments are presented.

## References

  * [1] E. Oho, N. Ichise, W. H. Martin, and K.-R. Peters, “Practical method for noise removal in scanning electron microscopy,” Scanning, vol. 18, no. 1, pp. 50–54, 1996.
  * [2] S. J. Pennycook, “The impact of stem aberration correction on materials science,” Ultramicroscopy, vol. 180, pp. 22–33, 2017.
  * [3] M. Linck, P. Hartel, S. Uhlemann, F. Kahl, H. Müller, J. Zach, M. Haider, M. Niestadt, M. Bischoff, J. Biskupek, et al., “Chromatic aberration correction for atomic resolution tem imaging from 20 to 80 kv,” Physical review letters, vol. 117, no. 7, p. 076101, 2016.
  * [4] F. Houdellier, L. De Knoop, C. Gatel, A. Masseboeuf, S. Mamishin, Y. Taniguchi, M. Delmas, M. Monthioux, M. Hÿtch, and E. Snoeck, “Development of tem and sem high brightness electron guns using cold-field emission from a carbon nanotip,” Ultramicroscopy, vol. 151, pp. 107–115, 2015.
  * [5] T. Akashi, Y. Takahashi, T. Tanigaki, T. Shimakura, T. Kawasaki, T. Furutsu, H. Shinada, H. Müller, M. Haider, N. Osakabe, et al., “Aberration corrected 1.2-mv cold field-emission transmission electron microscope with a sub-50-pm resolution,” Applied Physics Letters, vol. 106, no. 7, p. 074101, 2015.
  * [6] H. Adaniya, M. Cheung, C. Cassidy, M. Yamashita, and T. Shintake, “Development of a sem-based low-energy in-line electron holography microscope for individual particle imaging,” Ultramicroscopy, vol. 188, pp. 31–40, 2018\.
  * [7] C. T. Koch, “Towards full-resolution inline electron holography,” Micron, vol. 63, pp. 69–75, 2014.
  * [8] A. Feist, N. Bach, N. R. da Silva, T. Danz, M. Möller, K. E. Priebe, T. Domröse, J. G. Gatzmann, S. Rost, J. Schauss, et al., “Ultrafast transmission electron microscopy using a laser-driven field emitter: Femtosecond resolution with a high coherence electron beam,” Ultramicroscopy, vol. 176, pp. 63–73, 2017.
  * [9] V. Migunov, H. Ryll, X. Zhuge, M. Simson, L. Strüder, K. J. Batenburg, L. Houben, and R. E. Dunin-Borkowski, “Rapid low dose electron tomography using a direct electron detection camera,” Scientific reports, vol. 5, p. 14516, 2015.
  * [10] Y. Jiang, Z. Chen, Y. Han, P. Deb, H. Gao, S. Xie, P. Purohit, M. W. Tate, J. Park, S. M. Gruner, et al., “Electron ptychography of 2d materials to deep sub-ångström resolution,” Nature, vol. 559, no. 7714, p. 343, 2018.
  * [11] J. Hattne, D. Shi, C. Glynn, C.-T. Zee, M. Gallagher-Jones, M. W. Martynowycz, J. A. Rodriguez, and T. Gonen, “Analysis of global and site-specific radiation damage in cryo-em,” Structure, 2018.
  * [12] M. C. Motwani, M. C. Gadiya, R. C. Motwani, and F. C. Harris, “Survey of image denoising techniques,” in Proceedings of GSPX, pp. 27–30, 2004.
  * [13] Q. Zhang and C. L. Bajaj, “Cryo-electron microscopy data denoising based on the generalized digitized total variation method,” Far east journal of applied mathematics, vol. 45, no. 2, p. 83, 2010.
  * [14] H. S. Kushwaha, S. Tanwar, K. Rathore, and S. Srivastava, “De-noising filters for tem (transmission electron microscopy) image of nanomaterials,” in Advanced Computing & Communication Technologies (ACCT), 2012 Second International Conference on, pp. 276–281, IEEE, 2012.
  * [15] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural networks, vol. 2, no. 5, pp. 359–366, 1989.
  * [16] H. W. Lin, M. Tegmark, and D. Rolnick, “Why does deep and cheap learning work so well?,” Journal of Statistical Physics, vol. 168, no. 6, pp. 1223–1247, 2017.
  * [17] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013.
  * [18] X. Yang, V. De Andrade, W. Scullin, E. L. Dyer, N. Kasthuri, F. De Carlo, and D. Gürsoy, “Low-dose x-ray tomography through a deep convolutional neural network,” Scientific reports, vol. 8, no. 1, p. 2575, 2018.
  * [19] T. Remez, O. Litany, R. Giryes, and A. M. Bronstein, “Deep convolutional denoising of low-light images,” arXiv preprint arXiv:1701.01687, 2017.
  * [20] X.-J. Mao, C. Shen, and Y.-B. Yang, “Image restoration using convolutional auto-encoders with symmetric skip connections,” arXiv preprint arXiv:1606.08921, 2016.
  * [21] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017.
  * [22] W. Xu and J. M. LeBeau, “A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns,” arXiv preprint arXiv:1708.00855, 2017.
  * [23] K. Lee, J. Zung, P. Li, V. Jain, and H. S. Seung, “Superhuman accuracy on the snemi3d connectomics challenge,” arXiv preprint arXiv:1706.00120, 2017\.
  * [24] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Deep neural networks segment neuronal membranes in electron microscopy images,” in Advances in neural information processing systems, pp. 2843–2851, 2012.
  * [25] Y. Zhu, Q. Ouyang, and Y. Mao, “A deep convolutional neural network approach to single-particle recognition in cryo-electron microscopy,” BMC bioinformatics, vol. 18, no. 1, p. 348, 2017.
  * [26] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural networks, vol. 61, pp. 85–117, 2015.
  * [27] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014.
  * [28] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neural network architectures and their applications,” Neurocomputing, vol. 234, pp. 11–26, 2017.
  * [29] M. T. McCann, K. H. Jin, and M. Unser, “A review of convolutional neural networks for inverse problems in imaging,” arXiv preprint arXiv:1710.04011, 2017.
  * [30] J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting distributed synchronous sgd,” arXiv preprint arXiv:1604.00981, 2016.
  * [31] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., “Tensorflow: A system for large-scale machine learning.,” in OSDI, vol. 16, pp. 265–283, 2016.
  * [32] E. M. Christiansen, S. J. Yang, D. M. Ando, A. Javaherian, G. Skibinski, S. Lipnick, E. Mount, A. O’Neil, K. Shah, A. K. Lee, et al., “In silico labeling: Predicting fluorescent labels in unlabeled images,” Cell, vol. 173, no. 3, pp. 792–803, 2018.
  * [33] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015.
  * [34] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
  * [35] G. Lample and D. S. Chaplot, “Playing fps games with deep reinforcement learning.,” in AAAI, pp. 2140–2146, 2017.
  * [36] V. Firoiu, W. F. Whitney, and J. B. Tenenbaum, “Beating the world’s best at super smash bros. with deep reinforcement learning,” arXiv preprint arXiv:1702.06230, 2017.
  * [37] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Achieving human parity in conversational speech recognition,” arXiv preprint arXiv:1610.05256, 2016.
  * [38] C. Weng, D. Yu, M. L. Seltzer, and J. Droppo, “Single-channel mixed speech recognition using deep neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pp. 5632–5636, IEEE, 2014.
  * [39] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, “A simple neural network module for relational reasoning,” in Advances in neural information processing systems, pp. 4974–4983, 2017.
  * [40] Y. Wang and M. Kosinski, “Deep neural networks are more accurate than humans at detecting sexual orientation from facial images.,” Journal of personality and social psychology, vol. 114, no. 2, p. 246, 2018.
  * [41] Q. Yu, Y. Yang, F. Liu, Y.-Z. Song, T. Xiang, and T. M. Hospedales, “Sketch-a-net: A deep neural network that beats humans,” International Journal of Computer Vision, vol. 122, no. 3, pp. 411–425, 2017.
  * [42] S. S. Han, G. H. Park, W. Lim, M. S. Kim, J. Im Na, I. Park, and S. E. Chang, “Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network,” PloS one, vol. 13, no. 1, p. e0191493, 2018.
  * [43] T. Weyand, I. Kostrikov, and J. Philbin, “Planet-photo geolocation with convolutional neural networks,” in European Conference on Computer Vision, pp. 37–55, Springer, 2016.
  * [44] Q. Yang, P. Yan, Y. Zhang, H. Yu, Y. Shi, X. Mou, M. K. Kalra, Y. Zhang, L. Sun, and G. Wang, “Low dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss,” IEEE Transactions on Medical Imaging, 2018.
  * [45] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” arXiv preprint, 2016.
  * [46] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv preprint arXiv:1706.05587, 2017.
  * [47] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” arXiv preprint arXiv:1802.02611, 2018.
  * [48] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481–2495, 2017.
  * [49] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv preprint arXiv:1603.07285, 2016.
  * [50] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
  * [51] A. Krizhevsky and G. Hinton, “Convolutional deep belief networks on cifar-10,” Unpublished manuscript, vol. 40, p. 7, 2010.
  * [52] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  * [53] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [54] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [55] A. D. Association et al., “Tiff revision 6.0,” Internet publication: http://www. adobe. com/Support/TechNotes. html, 1992.
  * [56] P. J. Huber, “Robust estimation of a location parameter,” The annals of mathematical statistics, pp. 73–101, 1964.
  * [57] J. Kukačka, V. Golkov, and D. Cremers, “Regularization for deep learning: A taxonomy,” arXiv preprint arXiv:1710.10686, 2017.
  * [58] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in Advances in Neural Information Processing Systems, pp. 901–909, 2016.
  * [59] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256, 2010.
  * [60] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
  * [61] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Computer Vision, 1998. Sixth International Conference on, pp. 839–846, IEEE, 1998.
  * [62] E. Jones, T. Oliphant, P. Peterson, et al., “SciPy: Open source scientific tools for Python,” 2001–.  [Online; accessed ¡today¿].
  * [63] S. Van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu, “scikit-image: image processing in python,” PeerJ, vol. 2, p. e453, 2014.
  * [64] S. G. Chang, B. Yu, and M. Vetterli, “Adaptive wavelet thresholding for image denoising and compression,” IEEE transactions on image processing, vol. 9, no. 9, pp. 1532–1546, 2000.
  * [65] D. L. Donoho and J. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” biometrika, vol. 81, no. 3, pp. 425–455, 1994.
  * [66] A. Chambolle, “An algorithm for total variation minimization and applications,” Journal of Mathematical imaging and vision, vol. 20, no. 1-2, pp. 89–97, 2004.
  * [67] T. Goldstein and S. Osher, “The split bregman method for l1-regularized problems,” SIAM journal on imaging sciences, vol. 2, no. 2, pp. 323–343, 2009.
  * [68] P. Getreuer, “Rudin-osher-fatemi total variation denoising using split bregman,” Image Processing On Line, vol. 2, pp. 74–95, 2012.
  * [69] A. Buades, B. Coll, and J.-m. Morel, “Jm: A non-local algorithm for image denoising,” in In: Computer Vision and Pattern Recognition, 2005. CVPR 2005\. IEEE Computer Society Conference on, Citeseer, 2005.
  * [70] B. A. Turlach, “Bandwidth selection in kernel density estimation: A review,” in CORE and Institut de Statistique, Citeseer, 1993.
  * [71] D. M. Bashtannyk and R. J. Hyndman, “Bandwidth selection for kernel conditional density estimation,” Computational Statistics & Data Analysis, vol. 36, no. 3, pp. 279–298, 2001.
  * [72] D. W. Scott, Multivariate density estimation: theory, practice, and visualization.  John Wiley & Sons, 2015.
  * [73] K. Zuiderveld, “Contrast limited adaptive histogram equalization,” Graphics gems, pp. 474–485, 1994.
  * [74] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” Distill, vol. 1, no. 10, p. e3, 2016.
  * [75] S. Ioffe, “Batch renormalization: Towards reducing minibatch dependence in batch-normalized models,” in Advances in Neural Information Processing Systems, pp. 1942–1950, 2017.
  * [76] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010.
  * [77] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [78] G. Hinton, N. Srivastava, and K. Swersky, “Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,” Cited on, p. 14, 2012.
  * [79] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” arXiv preprint arXiv:1711.11585, 2017.
  * [80] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.
  * [81] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in International conference on machine learning, pp. 1139–1147, 2013.
  * [82] Y. Nesterov, “A method of solving a convex programming problem with convergence rate o (1/k2),” in Soviet Mathematics Doklady, vol. 27, pp. 372–376, 1983.


# On the nonexistence of certain orthogonal arrays of strength four

Rebeka Kiss  Bolyai Institute
University of Szeged
Aradi vértanúk tere 1
H-6720 Szeged, Hungary<EMAIL_ADDRESS>and  Gábor P. Nagy
Department of Algebra
Budapest University of Technology and Economics
Egry József utca 1
H-1111 Budapest, Hungary  Bolyai Institute
University of Szeged
Aradi vértanúk tere 1
H-6720 Szeged, Hungary<EMAIL_ADDRESS>
###### Abstract.

We show that no orthogonal arrays $OA(16\lambda,11,2,4)$ exist with
$\lambda=6$ and $\lambda=7$. This solves an open problem of the NSUCRYPTO
Olympiad 2018. Our result allows us to determine the minimum weights of
certain higher-order correlation-immune Boolean functions.

###### Key words and phrases:

Orthogonal array, NSUCRYPTO

###### 2010 Mathematics Subject Classification:

05B15

Support provided from the National Research, Development and Innovation Fund
of Hungary, financed under the 2018-1.2.1-NKP funding scheme, within the SETIT
Project 2018-1.2.1-NKP-2018-00004. Partially supported by NKFIH-OTKA Grants
119687, 115288 and SNN 132625.

##  1\. Introduction

In the Fifth International Students’ Olympiad in Cryptography NSUCRYPTO’2018
[7, 10] the following problem was stated. Given three positive integers $n$,
$t$, and $\lambda$ such that $t<n$, we call a $\lambda 2^{t}\times n$ binary
array (i.e., matrix over the two-element field) a $t-(2,n,\lambda)$ orthogonal
array if in every subset of $t$ columns of the array, every (binary) $t$-tuple
appears in exactly $\lambda$ rows. $t$ is called the strength of this
orthogonal array. Find a $4-(2,11,\lambda)$ orthogonal array with minimal
value of $\lambda$. So far, the best known answer to this question is
$\lambda=8$. Delsarte’s Linear Programming Bound [8, Theorem 4.15 and Table
4.19] implies $\lambda\geq 6$.

In this short note, we use the terminology of the monograph [8] and we denote
a $t-(2,n,\lambda)$ orthogonal array by $OA(2^{t}\lambda,n,2,t)$. The integers
$N=2^{t}\lambda$ and $n$ are called the number of runs and the number of
factors of the array. In an orthogonal array, the same row can occur multiple
times. The orthogonal array is simple, if each row occurs exactly once.

Our solution to the problem is stated in the following theorem.

######  Theorem 1.

No orthogonal arrays $OA(16\lambda,11,2,4)$ exist with $\lambda=6$ and
$\lambda=7$.

A Boolean function $f\colon\mathbb{F}_{2}^{n}\to\mathbb{F}_{2}$ is
correlation-immune of some order $t<n$ (in brief, $t$-CI) if fixing at most
$t$ of the $n$ input variables $x_{1},\ldots,x_{n}$ does not change the output
distribution of the function, whatever are the positions chosen for the fixed
variables and the values chosen for them. Equivalently, the support of the
function must be a simple binary orthogonal array of strength $t$, see [3, 4].
The weight of a Boolean function is the size of its support. Low weight $t$-CI
Boolean functions have practical importance in cryptography, since they resist
the Siegenthaler attack. Furthermore, $t$-CI Boolean functions allow reducing
the overhead while keeping the same resistance to side channel attacks. See
[4] and the references therein.

Theorem 1 allows us to determine the minimum weights of $t$th-order
correlation-immune Boolean functions in $n$ variables,

$n\in\\{11,12,13\\},\quad t\in\\{4,5\\}.$

These values were marked as unknown in [3, Table 2] and [4, Table 2].

We would like to thank Claude Carlet (Paris, France and Bergen, Norway) for
his detailed comments on the previous version of this paper.

##  2\. Proof of the theorem

Our proof uses the results by Bulutoglu and Margot [2], by Schoen, Eendebak
and Nguyen [11, 12] and also by Eendebak [5], where orthogonal arrays with
small parameters are classified. Bulutoglu and Margot [2] used integer linear
programming (ILP) methods, while the algorithms of Schoen, Eendebak and Nguyen
[11, 12] are based on the systematic study of the extensions of orthogonal
arrays by new columns. Moreover, both approaches must deal with the
isomorphism problem of orthogonal arrays.

###### Proof of the theorem.

From [2, Table 1], [11, Table III] and also [5] we can see that no
$OA(96,8,2,4)$ and no $OA(112,7,2,4)$ exist. We explain the relevant rows of
the two tables. In [2, Table 1] there are 4 columns with the following
meanings:

  * •

OA gives the parameters of the classified orthogonal array.

  * •

$m^{\prime}$ is the number of linearly independent equality constraints of the
generated ILP problem.

  * •

$p_{\max}$ is an upper bound on the maximum number of times a run can appear
in an $OA(2^{t}\lambda,n,2,t)$.

  * •

$h$ is the number of non-isomorphic orthogonal arrays with the given
parameters.

OA | $m^{\prime}$ | $p_{\max}$ | $h$
---|---|---|---
$OA(96,8,2,4)$ | $163$ | $2$ | $0$
$OA(112,7,2,4)$ | $99$ | $3$ | $0$

From this table we can see that if $\lambda=6$ then no orthogonal array exists
with $n=8$, which implies that no OA exists with $n\geq{8}$. Similarly if
$\lambda=7$ then no orthogonal array exists with $n=7$, thus no OA exists with
$n\geq{7}$.

In [11, Table III] orthogonal arrays with strength 4 are included, where

  * •

$N$ gives the run-size of the classified orthogonal array.

  * •

The notation $2^{a}$ for the factor set means a binary array with $a$ factors.

  * •

$a_{\max}$ is the maximum number $a$, such that there exists an $OA$ with $N$
runs and $a$ factors.

  * •

The numbers $m_{a}$, $a\in\\{t+1,\ldots,a_{\max}\\}$, in the last column
denote the number of isomorphism classes of arrays with $N$ runs and $a$
factors.

$N$ | Factor set | $a_{\max}$ | Isomorphism classes
---|---|---|---
$96$ | $2^{a}$ | $7$ | $4,9,4$
$112$ | $2^{a}$ | $6$ | $4,3$

This means that with run-size $96$ the maximum number $a$ such that an
$OA(96,a,2,4)$ exists is $7$, and with run-size $112$ the maximum number $a$
with an existing $OA(112,a,2,4)$ is 6. ∎

######  Remark 2.

According to [12], the number of isomorphism classes of binary orthogonal
arrays with run-size $N=128$, factor-size $n=11$, and strength $t=4$ is $477$.
The papers [2, 11] claim to achieve the above results within a few seconds.
Using SageMath [14], the GLPK package [9] and the integer linear programming
solver SCIP [6], a straightforward implementation of the formulas of [2] used
$51\,630$ sec and $481$ sec CPU time for the nonexistence of $OA(96,8,2,4)$
and $OA(112,7,2,4)$, respectively.

##  3\. Minimum weight of correlation-immune Boolean functions

Using the notation of [8], we denote by $F(n,2,t)$ the minimal number of runs
$N$ in any $OA(N,n,2,t)$ for given values $n$ and $t$. Theorem 1 says that
$F(11,2,4)\geq 128$, and in fact, equality holds. Let $\omega_{n,t}$ denote
the minimum weight of $t$-CI Boolean functions in $n$ variables. Equivalently,
$\omega_{n,t}$ is the minimum number of runs in a simple orthogonal array with
number of factors $n$ and strength $t$. Hence,

(1) $\displaystyle F(n,2,t)\leq\omega_{n,t}.$

Suppose $A$ is an $OA(N,n,s,t)$. As in [8, 5], one can construct an
$OA(N/s,n-1,s,t-1)$, say $A^{\prime}$. Clearly, if $A$ is simple then
$A^{\prime}$ is simple too. This implies

(2) $\displaystyle F(n-1,2,t-1)$ $\displaystyle\leq\frac{1}{2}F(n,2,t),$ (3)
$\displaystyle\omega_{n-1,t-1}$ $\displaystyle\leq\frac{1}{2}\omega_{n,t}.$

We are now able to fill some unknown values of [3, Table 2] and [4, Table 2].

######  Proposition 3.

For the minimum weight of $t$-CI Boolean functions in $n$ variables, we have

(4)
$\displaystyle\omega_{11,4}=\omega_{12,4}=\omega_{13,4}=\omega_{14,4}=\omega_{15,4}=128,$

and

(5)
$\displaystyle\omega_{11,5}=\omega_{12,5}=\omega_{13,5}=\omega_{14,5}=\omega_{15,5}=\omega_{16,5}=256.$

###### Proof.

The Nordstrom–Robinson code and also Sloane gives a simple $OA(256,16,2,5)$,
see [7, 1] and [13]. Straightforward computation shows that deleting the last
$5$ columns of it, the resulting orthogonal array is simple. Hence,
$\omega_{n,5}\leq 256$ for $n\in\\{11,\ldots,16\\}$. By (3), $\omega_{n,4}\leq
128$ for $n\in\\{10,\ldots,15\\}$. Theorem 1 implies $F(n,2,4)\geq 128$ for
$n\geq 11$. From (1) and (3) follow (4) and (5). ∎

## References

  * [1] Jürgen Bierbrauer  “Nordstrom–Robinson code and A7-geometry”  In _Finite Fields and Their Applications_ 13.1, 2007, pp. 158 –170  DOI: https://doi.org/10.1016/j.ffa.2005.05.004
  * [2] D. A. Bulutoglu and F. Margot  “Classification of orthogonal arrays by integer programming”  In _J. Statist. Plann. Inference_ 138.3, 2008, pp. 654–666  DOI: 10.1016/j.jspi.2006.12.003
  * [3] Claude Carlet and Xi Chen  “Constructing low-weight $d$th-order correlation-immune Boolean functions through the Fourier-Hadamard transform”  In _IEEE Trans. Inform. Theory_ 64.4, part 2, 2018, pp. 2969–2978  DOI: 10.1109/TIT.2017.2785775
  * [4] Claude Carlet and Sylvain Guilley  “Correlation-immune Boolean functions for easing counter measures to side-channel attacks”  In _Algebraic curves and finite fields_ 16, Radon Ser. Comput. Appl. Math.  De Gruyter, Berlin, 2014, pp. 41–70
  * [5] Pieter Eendebak  “Complete series of non-isomorphic orthogonal arrays”, 2020  URL: http://www.pietereendebak.nl/oapackage/series.html
  * [6] Gerald Gamrath et al.  “The SCIP Optimization Suite 7.0”, 2020  URL: http://www.optimization-online.org/DB_HTML/2020/03/7705.html
  * [7] Anastasiya Gorodilova et al.  “The Fifth International Students’ Olympiad in cryptography—NSUCRYPTO: Problems and their solutions”  In _Cryptologia_ 44.3 Taylor & Francis, 2020, pp. 223–256  DOI: 10.1080/01611194.2019.1670282
  * [8] A. S. Hedayat, N. J. A. Sloane and John Stufken  “Orthogonal arrays” Theory and applications; With a foreword by C. R. Rao, Springer Series in Statistics  Springer-Verlag, New York, 1999, pp. xxiv+416  DOI: 10.1007/978-1-4612-1478-6
  * [9] Andrew Makhorin  “GNU Linear Programming Kit”, 2020  URL: https://www.gnu.org/software/glpk/
  * [10] NSUCrypto2018  “Unsolved problems”, 2018  URL: https://nsucrypto.nsu.ru/archive/2018/round/2/section/0/task/2/
  * [11] Eric D. Schoen, Pieter T. Eendebak and Man V. M. Nguyen  “Complete enumeration of pure-level and mixed-level orthogonal arrays”  In _J. Combin. Des._ 18.2, 2010, pp. 123–140  DOI: 10.1002/jcd.20236
  * [12] Eric D. Schoen, Pieter T. Eendebak and Man V. M. Nguyen  “Correction to: Complete enumeration of pure-level and mixed-level orthogonal arrays [MR2604638]”  In _J. Combin. Des._ 18.6, 2010, pp. 488  DOI: 10.1002/jcd.20270
  * [13] N. J. A. Sloane  “A Library of Orthogonal Arrays”, 2020  URL: http://neilsloane.com/oadir/
  * [14] The Sage Developers  “SageMath, the Sage Mathematics Software System (Version 9.1)” https://www.sagemath.org, 2020