An implicit but pervasive hypothesis of modern computer vision research is that convolutional neural network (CNN) architectures which perform better on ImageNet will also perform better on other vision datasets. We challenge this hypothesis through an extensive empirical study for which we train $500$ sampled CNN architectures on ImageNet as well as $8$ other image classification datasets from a wide array of application domains. The relationship between architecture and performance varies wildly, between the datasets. For some of the datasets the performance correlation with ImageNet is even negative. Clearly, it is not enough to optimize architectures solely for ImageNet when aiming for progress that is relevant for all applications. We identify two dataset-specific performance indicators: the cumulative width across layers as well as the total depth of the network. Lastly, we show that the range of dataset variability covered by ImageNet can be significantly extended by adding ImageNet subsets restricted to fewer classes.

§ INTRODUCTION

Deep convolutional neural networks (CNNs) are the core building block for most modern visual recognition systems. In the past several years they have lead to major breakthroughs in many domains of computer perception. Since the inception of the field the community has been searching the high dimensional space of possible network architectures for models with desirable properties. Important milestones such as DanNet [1], AlexNet [2], VGG [3], HighwayNet [4], and ResNet [5] (a HighwayNet with open gates) can be seen as update steps in this stochastic optimization problem and stand testament that this manual architecture search works. But it is of great importance that the right metrics are used during the search for new neural network architectures. Only when we measure performance with a truly meaningful metric is it certain that a new high-scoring architecture is also fundamentally better.

It would be desirable to construct such a metric form a solid theoretical understanding of deep CNNs. Due to the absence of a solid theoretical basis novel neural network designs are tested in an empirical fashion. Traditionally model performance has been judged using accuracy point estimates [2, 6, 3]. This simple measure ignores important aspects such as model complexity and speed. Newer work addresses this issue by reporting a curve of the accuracy at different complexity settings of the model, highlighting how well a design deals with the accuracy versus complexity tradeoff [7, 8].

Very recent work strives to improve the quality of the empiric evaluation even further. There have been attempts to use extensive empirical studies to discover general rules on neural network design [9, 10, 11, 12], instead of simply showing the merits of a single neural network architecture. Another line of research aims to improve empiricism by sampling whole populations of models and comparing error distributions instead of individual scalar errors  [13].

Is a CNN architecture that performs well on ImageNet necessarily a good choice for a different vision dataset? This plot suggests otherwise: It displays the relative test errors of $500$ randomly sampled CNN architectures on three datasets (ImageNet, Powerline, and Insects) plotted against the test error of the same architectures on ImageNet. The architectures have been trained from scratch on all three datasets. Architectures with low errors on ImageNet also perform well on Insects, on Powerline the opposite the case.

We acknowledge the importance of the above-mentioned improvements in the empirical methods used to test neural networks, but identify a weak spot that runs trough the above-mentioned work: the heavy reliance on ImageNet [14] (and to some extent the very similar Cifar100 [15]). In 2011, Torralba and Efros already pointed out that visual recognition datasets that were built to represent the visual world tend to become a small world in themselves [16]. Objects are no longer in the dataset because they are important, they are important because they are in the dataset. A large part of computer vision research has lived in the ImageNet world for almost a decade now, and consequently worked under the implicit assumption that ImageNet is large and diverse enough to represent the visual world. In this paper we challenge this assumption and ask the question: Do neural network architecture improvements achieved on ImageNet generalize to other datasets?

We answer this question with an extensive empirical investigation. As a tool for this investigation we introduce the notion of architecture and performance relationship (APR). The  performance of a CNN architecture does not exist in a vacuum, it is only defined in relation to the dataset on which it is used. This dependency is what we call APR induced by a dataset.
We study the change in APRs between datasets by sampling $500$ neural network architectures and training all of them on a set of datasets[Since we only sample models in the complexity regime of $340$ mega flops (MF) to $400$MF (ResNet-152 has $11.5$GF) we could complete the necessary $7500$ model trainings within a moderate $85$ GPU days on Tesla V100-SXM2-32GB GPUs.]. We then compare errors of the same architectures across datasets, revealing the changes in APR (see Figure <ref>). This approach allows us to study the APRs induced by different datasets on a whole population of diverse network designs rather than just a family of similar architectures such as the ResNets [5] or MobileNets [17].

All of our code, data and trained models will be made publicly available to ensure reproducibility and facilitate future research.

§ RELATED WORK

Neural network design.
With the introduction of the first deep CNNs [1, 2] the design of neural networks immediately became an active research area. In the following years many improved architectures where introduced, such as VGG [3], Inception [18], HighwayNet [4], ResNet [5] (a HighwayNet with open gates), ResNeXt [7], or MobileNet [17]. These architectures are the result of manual search aimed at finding new design principles that improve performance, for example increased network depth and skip connections. More recently, reinforcement learning [8], evolutionary algorithms [19] or gradient descent [20] have been successfully used to find suitable network architectures automatically. Our work relates to manual and automatic architecture design because it adds perspective on how widely certain improvements are applicable.

Empirical studies.
In the absence of a solid theoretical understanding, large-scale empirical studies are the best tool at our disposal to gain insight into the nature of deep neural networks. These studies can aid network design [21, 22, 23] or be employed to show the merits of different approaches, for example that the classic LSTM [24] architecture can outperform more modern models [25], when it is properly regularised. More recently, empirical studies have been used to infer more general rules on the behaviour of neural networks such as a power-law describing the relationship between generalization error and dataset size [9] or scaling laws for neural language models [11].

Generalization in neural networks.
Despite their vast size have deep neural networks shown in practice that they can generalize extraordinarily well to unseen data stemming from the same distribution as the training data. Why neural networks generalize so well is still an open and very active research area [26, 27, 28].
This work is not concerned with the generalization of a trained network to new data, but with the generalization of the architecture design progress itself. Does an architecture designed for a certain dataset, e.g. natural photo classification using ImageNet, work just as well for medical imaging? There has been work investigating the generalization to a newly collected test set, but in this case the test set was designed to be of the same distribution as the original training data [29].

Neural network transferability
It is known that the best architecture for ImageNet is not necessarily the best  base architecture for other applications such as semantic segmentation [30] or object detection [31]. Researchers who computed a taxonomy of multiple visions tasks identified that the simmilarities between tasks did not depend on the used architecture [32].
Research that investigates the relation between model performance on ImageNet and new classification datasets in the context of transfer learning [33, 34] suggests that there is a strong correlation which is also heavily dependent on the training regime used [35].
Our work differs form the ones mentioned above in that we are not interested in the transfer of learned features but transfer of the architecture designs and therefore we train our networks from scratch on each dataset. Moreover do we not only test transferability on a few select architectures but on a whole network space.

Neural network design space analysis.
Radosavovic et al. [13] introduced network design spaces for visual recognition. They define a design space as a set of architectures defined in a parametric form with a fixed base structure and architectural hyperparameters that can be varied, similar to the search space definition in neural architecture search [8, 19, 20]. The error distribution of a given design space can be computed by randomly sampling model instances from it and computing their training error. We use a similar methodology but instead of comparing different design spaces, we compare the results of the same design space on different datasets.

§ DATASETS

To enable cross dataset comparison of APRs we assembled a corpus of datasets. We chose datasets according to the following principles: (a) include datasets from a wide spectrum of application areas, such that generalization is tested on a diverse set of datasets; (b) only use datasets that are publicly available to anyone to ensure easy reproducibility of our work. Table <ref> shows some meta-data of the chosen datasets. For more detailed dataset descriptions and example images please see chapter <ref> and Figure <ref> in the appendix.

Meta data of the used datasets.

Dataset     No. Images     No. Classes     Img. Size     Domain

Concrete       $40$K     $2$     $227 \times 227$     Maintenance

MLC2008        $43$K     $9$     $312 \times 312$     Biology

Imagenet       $1.3$M     $1000$     $256 \times 256$     Everyday objects

HAM10000       $10$K     $7$     $296 \times 296$     Medical

Powerline      $8$K      $2$     $128 \times 128$     Maintenance

Insects        $63$K     $291$     $296 \times 296$    Biology

Natural        $25$k     $6$     $150 \times 150$     Natural Scenes

Cifar10        $60$k     $10$     $32 \times 32$     Everyday objects

Cifar100       $60$k     $100$     $32 \times 32$    Everyday objects

§ EXPERIMENTS AND RESULTS

§.§ Experimental setup

The structure of models in the AnyNetX design space, with a fixed stem and a head, consisting of one fully-connected layer of size $c$, (where c is the number of classes). Each stage $i$ of the body is parametrised by $d_i,w_i,b_i,g_i$, the strides of the stages are fixed with $s_1=1$ and $s_i=2$ for the remainder.

We sample our architectures form the very general AnyNetX [36] parametric network space. The networks in AnyNetX consist of a stem, a body, and a head. The body performs the majority of the computation, stem and head are kept fixed across all sampled models. The body consists of four stages, each stage $i$ starts with a $1 \times 1$ convolution with stride $s_i$, the remainder is a sequence of $d_i$ identical blocks. The blocks are standard residual bottleneck blocks with group convolution [7], with a total block width $w_i$, bottleneck ratio $b_i$ and a group width $g_i$ (into how many parallel convolutions the total width is grouped into). Within a stage, all the block parameters are shared. See Figure <ref> for a comprehensive schematic.

The AnyNetX design space has a total of $16$ degrees of freedom, having $4$ stages with $4$ parameters each. We obtain our model instances by performing log-uniform sampling of $d_i \leq 16$, $w_i \leq 1024$ and divisible by $8$, $b_i \in {1,2,4}$, and $g_i \in  {1,2, ..., 32}$. The stride $s_i$ is fixed with a stride of $1$ for the first stage and a stride of $2$ for the rest. We repeatedly draw samples until we have obtained a total of $500$ architectures in our target complexity regime of $360$ mega flops (MF) to $400$ MF. We use a very basic training regime that consists of only flipping and cropping of the inputs in conjunction with SGD plus momentum and weight decay.
The same $500$ models are trained on each dataset until the loss is reasonably saturated. The exact number of epochs has been determined in preliminary experiments and depends on the dataset (see Table <ref>). For extensive ablation studies ensuring the empirical stability of our experiments with respect to Cifar10 performance, training duration, training variability, top-1 to top-5 error comparisons, overfitting and class distribution see chapters <ref> to <ref> in the appendix.

Dataset-specific experimental settings.

Dataset     No. training epochs     Eval. error

concrete       $20$     top-1

MLC2008       $20$     top-1

Imagenet       $10$     top-5

HAM10000       $30$     top-1

Powerline      $20$     top-1

Insects        $20$     top-5

Natural        $20$     top-1

Cifar10        $30$     top-1

Cifar100       $30$     top-5

§.§ Experimental results

$eCDF$ plots from the $500$ sampled architectures for each dataset. The y coordinate gives the share of models with less error than the corresponding x value.

As a preliminary analysis we compute the empirical cumulative error distribution function ($eCDF$) per dataset. The $eCDF$ of $n$ models with errors $e_i$ is defined as:

\begin{equation}
eCDF(e) = \frac{1}{n}\sum_{i=1}^{n}\textbf{1}[e_i < e],
\end{equation}

meaning $eCDF(x)$ returns the fraction of models with an error smaller than $x$. Figure <ref> shows the $eCDF$ per dataset. ImageNet exhibits a steady decrease in model performance and very large spread between the best and the worst architectures. Natural, Insects, Powerline and Cifar100 have a somewhat similarly shaped $eCDF$, with a relatively high density of good architectures and a long tail containing significantly worse architectures. Concrete, HAM10000, MLC2008 and Cifar10 show a different picture. The $eCDF$s of these datasets are narrow, indicating a low difference in performance for most of the sampled architectures. The error percentage for Concrete is below $0.5$ percent for almost all of the sampled architectures, indicating that this dataset is easy to learn for modern CNN architectures.

Test errors of all 500 sampled architectures on target datasets (y-axis) plotted against the test errors of the same architectures (trained and tested) on ImageNet (x-axis). The top 10 performances on the target datasets are plotted in orange and the worst 10 performances in red.

We analyze the architecture-performance relationship (APRs) in two ways. For every target dataset (datsets which are not ImageNet) we plot the test error of every sampled architecture against the test error of the same architecture (trained and tested) on ImageNet, visualizing the relationship of the target dataset's APR with the APR on ImageNet. Second, we compute Spearman's $\rho$ rank correlation coefficient [37]. It is a nonparametric measure for the strength of the relation between two variables (here the error on the target datasets with the error of the same architecture on ImageNet). Spearman's $\rho$ is defined on $[-1,1]$, where $0$ indicates no relationship and $-1$ or $1$ indicates that the relationship between the two variables can be fully described using only a monotonic function.

Figure <ref> contains the described scatterplots with the corresponding correlation coefficients in the title. The datasets plotted in the top row show a strong (Insects) or medium (MLC2008, HAM10000, Cifar100) error correlation with ImageNet. This confirms that many classification tasks have an APR similar to the one induced by ImageNet, which makes ImageNet performance a decent architecture selection indicator for these datasets. The errors on Concrete are independent of the their corresponding ImageNet counterparts since the accuracies are almost saturated with errors between $0$ and $0.5$. This has implications for practical settings, where in such cases suitable architectures should be chosen according to computational considerations rather than ImageNet performance, and reinforces the idea that practical problems may lie well outside of the ImageNet visual world [38]. The most important insight from Figure <ref>, however, is that some datasets have a slight (Cifar10) or even strong (Powerline, Natural) negative error correlation with ImageNet. Architectures which perform well on ImageNet tend perform sub-par on these datasets. A visual inspection shows that some of the very best architectures on ImageNet perform extraordinarily poor on these three datasets. With this evidence we can safely reject the hypothesis that increasing performance on ImageNet implies increased performance on other image classification tasks.

Errors of all 500 sampled architectures on ImageNet, Insects, HAM10000, Powerline, Natural, and Cifar100 (x-axis) plotted against the cumulative block depths (y-axis).

§.§ Identifying Drivers of Difference between Datasets

The block width and depth parameters of the top $15$ architectures for ImageNet (see Figure <ref> in the appendix) follow a clear structure: they consistently start with low values for both block depth and width in the first stage, then the values steadily increase across the stages for both parameters. The error relationships observed in Figure <ref> are consistent with how well these patterns are replicated by the other datasets. Insects shows a very similar pattern, MLC2008 and HAM10000 have the same trends but more noise. Powerline and Natural clearly break from this structure, having a flat or decreasing structure in the block with and showing a quite clear preference for a small block depth in the final stage. Cifar10 and Cifar100 are interesting cases, they have the same behaviour as ImageNet with respect to block width but a very different one when it comes to block depth.

We thus investigate the effect of the cumulative block depth (summation of the depth parameter for all four stages, yielding the total depth of the architecture) across the whole population of architectures by plotting the cumulative block depth against the test error for the six above-mentioned datasets. Additionally, we compute the corresponding correlation coefficients. Figure <ref> shows that the best models for ImageNet have a cumulative depth of at least $10$. Otherwise there is no apparent dependency between the ImageNet errors and cumulative block depth. The errors of Insects do not seem to be related to the cumulative block depth at all. HAM10000 has a slight right-leaning spread leading to a moderate correlation, but the visual inspection shows no strong pattern. The errors on Powerline, Natural, and Cifar100 on the other hand have a strong dependency with the cumulative block depth. The error increases with network depth for all three datasets. with the best models all having a cumulative depth smaller than $10$.

Errors of all 500 sampled architectures on ImageNet, Insects, HAM10000 and Cifar100 (x-axis) plotted against the cumulative block widths (y-axis).

We also plot the cumulative block widths against the errors and compute the corresponding correlation coefficients for the same six datasets (see Figure <ref>). We observe that the ImageNet errors are negatively correlated with the cumulative block width, and visual inspection shows that a cumulative block width of at least $250$ is required to achieve a decent performance. The errors on Insects and HAM10000 replicate this pattern to a lesser extent, analogous to the top $15$ architectures. Powerline and Natural have no significant error dependency with the cumulative block width, but Cifar100 has an extremely strong negative error dependency with the cumulative block width, showing that it is possible for a dataset to replicate the behaviour on ImageNet in one parameter but not the other. In the case of Cifar100 and ImageNet, low similarity in block depth and high similarity in block width yield a medium overall similarity of ARPs on Cifar100 and Imagenet. This is consistent with the overall relationship of the two datasets displayed in Figure <ref>.

Finally, we observe a very high variance for the bottleneck ratios and the group width parameter in Fig. <ref>. Individual parameter-performance scatterplots (see Figures <ref> to <ref> in the appendix) of bottleneck ratio and group width don't show tangible structures for any of the datasets either, such that we where unable to identify structural changes in these two parameter between the datasets. We thus conclude that optimal network depth and optimal network width can vary between datasets, making them both important influencing factors for architecture transferability.

§.§ Impact of the Number of Classes

ImageNet has by far the largest number of classes among all the datasets. Insects, which is the dataset with the second highest class count, also shows the strongest similarity in APR to ImageNet. This suggests that the number of classes might be an important property of a dataset with respect to optimal architecture design. We test this hypothesis by running an additional set of experiments on subsampled versions of ImageNet. We create new datasets by randomly choosing a varying number of classes from ImageNet and deleting the rest of the dataset. This allows us to isolate the impact of the number of classes while keeping all other aspects of the data itself identical. We create four subsampled ImageNet versions with $100$, $10$, $5$, and $2$ classes, which we call ImageNet-100, ImageNet-10, ImageNet-5, and ImageNet-2, respectively. We refer to the resulting group of datasets (including the original ImageNet) as the ImageNet-X family. The training regime for ImageNet-100 is kept identical to the one of ImageNet, for the other three datasets we switch to top-1 error and train for $40$ epochs, to account for the smaller dataset size. (see section <ref>  in the appendix for a control experiment that disentangles the effects of reduced dataset size and reduced number of classes)

Error of all $500$ sampled architectures on subsampled (by number of classes) versions of ImageNet (y-axis) plotted against the error of the same architectures on regular ImageNet (x-axis). The top $10$ performances on the target dataset are plotted in orange and the worst $10$ performances in red.

Figure <ref> shows the errors on the subsampled versions plotted against the errors on original ImageNet. APR on ImageNet-100 shows an extremely strong correlation with APR on ImageNet. This correlation significantly weakens as the class count gets smaller. ImageNet-2 is on the opposite end has errors which are practically independent from the ones on ImageNet. This confirms our hypothesis that the number of classes is a dataset property with significant effect on the architecture to performance relationship

Correlation of observed error rates with the cumulative block depth and width parameters for all ImageNet-X datasets.

Dataset     C. Block Depth      C. Block Width

ImageNet       $-0.205$     $-0.511$

ImageNet-100      $-0.022$     $-0.558$

ImageNet-10       $0.249$     $-0.457$

ImageNet-5      $0.51$      $-0.338$

ImageNet-2        $0.425$     $-0.179$

Combining this result with the outcome of the last section, we study the interaction between the number of classes, the cumulated block depth and the cumulative block width. Table <ref> contains the correlations between cumulative block depth/width and the errors on all members of ImageNet-X. With decreasing number of classes, the correlation coefficients increase for cumulative block depth and cumulative block width. Although the effect on cumulative block depth is stronger, there is a significant impact on both parameters. We thus conclude that the number of classes simultaneously impacts both cumulative block depth and cumulative block width.

Test errors of all $500$ sampled architectures on target datasets (y-axis) plotted against the test errors of the same architectures on the ImageNet-X (x-axis). The top $10$ performances on the target dataset are orange, the worst $10$ performances red.

We have observed that the number of classes has a profound effect on the APR associated with ImageNet-X members. It is unlikely that simply varying the number of classes in this dataset is able to replicate the diversity of APRs present in an array of different datasets. However, it is reasonable to assume that a dataset's APR is better represented by the ImageNet-X member closest in terms of class count, instead of just ImageNet. We thus recreate Figure <ref> with the twist of not plotting the target dataset errors against ImageNet, but against the ImageNet-X variant closest in class count (see Figure <ref>). We observe gain in correlation across all datasets, in the cases of MLC2008 or Cifar10 a quite extreme one. The datasets which have a strong negative correlation with ImageNet (Powerline, Natural) have slightly (Natural) or even moderately (Powerline) positive correlation to their ImageNet-X counterparts. A visual inspection shows that the best models on Imagenet-X also yield excellent results on Powerline and Natural, which was not the case for ImageNet. Table <ref> shows the error correlations of all target datasets with ImageNet as well as with their ImageNet-X counterpart. The move from ImageNet to ImageNet-X more than doubles the average correlation (from $0.19$ to $0.507$), indicating that the ImageNet-X family of datasets is capable to represent a much wider variety of APRs than ImageNet alone.

Comparison of error correlations between target datasets and ImageNet as well as the closest  ImageNet-X member.

Dataset     $\rho$ -ImageNet      $\rho$ -ImageNet-X     Difference

concrete       $0.001$     $0.106$     $0.105$

MLC2008        $0.476$      $0.811$     $0.335$

ham10000       $0.517$      $0.608$      $0.091$

powerline      $-0.436$     $0.294$     $0.73$

insects        $0.967$      $0.95$     $-0.017$

natural        $-0.38$     $0.186$     $0.566$

cifar10        $-0.104$     $0.45$     $0.554$

cifar100       $0.476$      $0.595$     $0.119$

Average     $0.19$      $0.507$      $0.317$

§ DISCUSSION AND CONCLUSIONS

ImageNet is not enough. Is it true that models which perform better on ImageNet generally also perform better on other visual classification tasks? This is the question at the heart of our study. Our extensive empirical results show that the answer is no. To get an representative estimate of the merits of a novel architecture, it has to be tested on a variety of datasets which span the whole domain in which this architecture will be applied.
Block depth and block width matter. We have identified block depth and block width as key influence factors on the transferability of an architecture. When designing an architecture for a practical application, where out of the box models perform poorly, these two dimensions should be explored first.

Varying the number of classes helps. It is striking how much more accurately the ImageNet-X family is able to represent the diversity in APRs present in our dataset collection, compared to just ImageNet by itself. In cases where it is not possible to test an architecture on a variety of different datasets, it is thus advisable to instead test it on different subsets (with respect to class count) of the available datasets, similar to how it has become commonplace to test new architectures in multiple complexity regimes [5, 17].
Future directions.
Our study highlights the dangers of optimizing on a single dataset. A future similar study should shed light on how well the breadth of other domains such as object detection or speech classification is represented by their most used datasets.

A labeled dataset will always be a biased description of the visual world, due to having a fixed number of classes and being built with some systematic image collection process. Self-supervised learning of visual representations [39] could serve as remedy for this issue. Self-supervised architectures could be fed with a stream completely unrelated images, collected from an arbitrary number of sources in a randomized way. A comparison of visual features learned in this way could yield a more meaningful measure of the quality of CNN architectures.
As with any experimental analysis of a highly complex process such as training a CNN it is virtually impossible to consider every scenario. We list below three dimensions along which our experiments are limited together with measures we took to minimize the impact of these limitations.
Data scope: We criticize ImageNet for only representing a fraction of the “visual world”. We are aware that our dataset collection does not span the entire “visual world” either but went to great lengths to maximise the scope of our dataset collection by purposefully choosing datasets from different domains, which are visually distinct.
Architecture scope: We sample our architectures from the large AnyNetX network space. It contains the CNN building blocks to span basic designs such as AlexNet or VGG as well as the whole ResNet, ResNeXt and RegNet families. We acknowledge that there are popular CNN components not covered, however, Radosavovic et al. [36] present ablation studies showing that network designs sourced from high performing regions in the AnyNetX space also perform highly when swapping in different originally missing components such as depthwise convolutions [40], swish activation functions [41] or the squeeze-and-excitation [42] operations.
Training scope: When considering data augmentation and optimizer settings there are almost endless possibilities to tune the training process. We opted for a very basic setup with no bells an whistles in general. For certain such aspects of the training, which we assumed might skew the results of our study (such as training duration, dataset prepossessing etc.), we have conducted extensive ablation studies to ensure that this is not the case (see sections <ref> and <ref> in the appendix).

This work has been financially supported by grants 25948.1 PFES-ES “Ada” (CTI), 34301.1 IP-ICT “RealScore” (Innosuisse) and ERC Advanced Grant AlgoRNN nr. 742870. We are grateful to Frank P. Schilling for his valuable inputs.

[1]
Dan C. Ciresan, Ueli Meier, and Jürgen Schmidhuber.
Multi-column deep neural networks for image classification.
In 2012 IEEE Conference on Computer Vision and Pattern
Recognition, pages 3642–3649. IEEE Computer Society, 2012.

[2]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
Imagenet classification with deep convolutional neural networks.
In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C.
Burges, Léon Bottou, and Kilian Q. Weinberger, editors, 26th
Annual Conference on Neural Information Processing Systems, pages
1106–1114, 2012.

[3]
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
In Yoshua Bengio and Yann LeCun, editors, 3rd International
Conference on Learning Representations, 2015.

[4]
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.
Highway networks.
CoRR, abs/1505.00387, 2015.

[5]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In 2016 IEEE Conference on Computer Vision and Pattern
Recognition, pages 770–778. IEEE Computer Society, 2016.

[6]
Matthew D. Zeiler and Rob Fergus.
Visualizing and understanding convolutional networks.
In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne
Tuytelaars, editors, 13th European Conference on Computer Vision,
Proceedings, Part I, pages 818–833. Springer, 2014.

[7]
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.
Aggregated residual transformations for deep neural networks.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition, pages 5987–5995. IEEE Computer Society, 2017.

[8]
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.
Learning transferable architectures for scalable image recognition.
In 2018 IEEE Conference on Computer Vision and Pattern
Recognition, pages 8697–8710. IEEE Computer Society, 2018.

[9]
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun,
Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
Deep learning scaling is predictable, empirically.
CoRR, abs/1712.00409, 2017.

[10]
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit.
A constructive prediction of the generalization error across scales.
In 8th International Conference on Learning Representations.
OpenReview.net, 2020.

[11]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
CoRR, abs/2001.08361, 2020.

[12]
Lukas Tuggener, Mohammadreza Amirian, Fernando Benites, Pius von Däniken,
Prakhar Gupta, Frank-Peter Schilling, and Thilo Stadelmann.
Design patterns for resource-constrained automated deep-learning
AI, 1(4):510–538, 2020.

[13]
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr
On network design spaces for visual recognition.
In International Conference on Computer Vision, pages
1882–1890. IEEE, 2019.

[14]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein,
Alexander C. Berg, and Fei-Fei Li.
Imagenet large scale visual recognition challenge.
CoRR, abs/1409.0575, 2014.

[15]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[16]
Antonio Torralba and Alexei A. Efros.
Unbiased look at dataset bias.
In 2011 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1521–1528. IEEE Computer Society, 2011.

[17]
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision
CoRR, abs/1704.04861, 2017.

[18]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed,
Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.
Going deeper with convolutions.
In 2015 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1–9. IEEE Computer Society, 2015.

[19]
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le.
Regularized evolution for image classifier architecture search.
In The Thirty-Third AAAI Conference on Artificial
Intelligence, pages 4780–4789. AAAI Press, 2019.

[20]
Hanxiao Liu, Karen Simonyan, and Yiming Yang.
DARTS: differentiable architecture search.
In 7th International Conference on Learning Representations.
OpenReview.net, 2019.

[21]
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink,
and Jürgen Schmidhuber.
LSTM: A search space odyssey.
IEEE Trans. Neural Networks Learn. Syst., 28(10):2222–2232,

[22]
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo.
Capacity and trainability in recurrent neural networks.
In 5th International Conference on Learning Representations.
OpenReview.net, 2017.

[23]
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha
Sensitivity and generalization in neural networks: an empirical
In 6th International Conference on Learning Representations.
OpenReview.net, 2018.

[24]
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
Neural Comput., 9(8):1735–1780, 1997.

[25]
Gábor Melis, Chris Dyer, and Phil Blunsom.
On the state of the art of evaluation in neural language models.
In 6th International Conference on Learning Representations,.
OpenReview.net, 2018.

[26]
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio.
Generalization in deep learning.
CoRR, abs/1710.05468, 2017.

[27]
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.
Sharp minima can generalize for deep nets.
In Doina Precup and Yee Whye Teh, editors, 34th International
Conference on Machine Learning, pages 1019–1028. PMLR, 2017.

[28]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning requires rethinking generalization.
In 5th International Conference on Learning Representations.
OpenReview.net, 2017.

[29]
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.
Do imagenet classifiers generalize to imagenet?
In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, 36th
International Conference on Machine Learning, pages 5389–5400. PMLR,

[30]
Jonathan Long, Evan Shelhamer, and Trevor Darrell.
Fully convolutional networks for semantic segmentation.
In 2015 IEEE Conference on Computer Vision and Pattern
Recognition, pages 3431–3440. IEEE Computer Society, 2015.

[31]
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun.
Detnas: Backbone search for object detection.
In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence
d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, 32th
Annual Conference on Neural Information Processing Systems, pages
6638–6648, 2019.

[32]
Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra
Malik, and Silvio Savarese.
Taskonomy: Disentangling task transfer learning.
In Sarit Kraus, editor, International Joint Conference on
Artificial Intelligence 2019, pages 6241–6245. ijcai.org, 2019.

[33]
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.
CNN features off-the-shelf: An astounding baseline for recognition.
In 2014 IEEE Conference on Computer Vision and Pattern
Recognition, pages 512–519. IEEE Computer Society, 2014.

[34]
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric
Tzeng, and Trevor Darrell.
Decaf: A deep convolutional activation feature for generic visual
In 31th International Conference on Machine Learning, pages
647–655. JMLR.org, 2014.

[35]
Simon Kornblith, Jonathon Shlens, and Quoc V. Le.
Do better imagenet models transfer better?
In 2019 IEEE Conference on Computer Vision and Pattern
Recognition, pages 2661–2671. Computer Vision Foundation / IEEE, 2019.

[36]
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and
Piotr Dollár.
Designing network design spaces.
In 2020 IEEE Conference on Computer Vision and Pattern
Recognition, pages 10425–10433, 2020.

[37]
David Freedman, Robert Pisani, and Roger Purves.
Statistics (international student edition).
Pisani, R. Purves, 4th edn. WW Norton & Company, New York,

[38]
Thilo Stadelmann, Mohammadreza Amirian, Ismail Arabaci, Marek Arnold,
Gilbert François Duivesteijn, Ismail Elezi, Melanie Geiger, Stefan
Lörwald, Benjamin Bruno Meier, Katharina Rombach, et al.
Deep learning in the wild.
In IAPR Workshop on Artificial Neural Networks in Pattern
Recognition, pages 17–38. Springer, 2018.

[39]
Longlong Jing and Yingli Tian.
Self-supervised visual feature learning with deep neural networks:
A survey.
CoRR, abs/1902.06162, 2019.

[40]
François Chollet.
Xception: Deep learning with depthwise separable convolutions.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1800–1807. IEEE Computer Society, 2017.

[41]
Prajit Ramachandran, Barret Zoph, and Quoc V. Le.
Searching for activation functions.
In 6th International Conference on Learning Representations
Workshop Track Proceedings. OpenReview.net, 2018.

[42]
Jie Hu, Li Shen, and Gang Sun.
Squeeze-and-excitation networks.
In 2018 IEEE Conference on Computer Vision and Pattern
Recognition, pages 7132–7141. IEEE Computer Society, 2018.

[43]
Ç F Özgenel and A Gönenç Sorguç.
Performance comparison of pretrained convolutional neural networks on
crack detection in buildings.
In ISARC. Proceedings of the International Symposium on
Automation and Robotics in Construction, pages 1–8. IAARC Publications,

[44]
A. S. M. Shihavuddin, Nuno Gracias, Rafael García, Arthur C. R. Gleason,
and Brooke Gintert.
Image-based coral reef classification and thematic mapping.
Remote. Sens., 5(4):1809–1841, 2013.

[45]
Oscar Beijbom, Peter J. Edmunds, David I. Kline, B. Greg Mitchell, and David J.
Automated annotation of coral reef survey images.
In 2012 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1170–1177. IEEE Computer Society, 2012.

[46]
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.
The HAM10000 dataset: A large collection of multi-source
dermatoscopic images of common pigmented skin lesions.
CoRR, abs/1803.10417, 2018.

[47]
Ömer Emre Yetgin, Ömer Nezih Gerek, and Ömer Nezih.
Ground truth of powerline dataset (infrared-ir and visible light-vl).
Mendeley Data, 8, 2017.

[48]
Oskar Liset Pryds Hansen, Jens-Christian Svenning, Kent Olsen, Steen Dupont,
Beulhah H. Garner, Alexandros Iosifidis, Benjamin W. Price, and Toke T.
Image data used for publication "Species-level image classification
with convolutional neural network enable insect identification from habitus
images ", November 2019.

[49]
Puneet Bansal.
Intel image classification.

[50]
Antonio Torralba, Robert Fergus, and William T. Freeman.
80 million tiny images: A large data set for nonparametric object
and scene recognition.
IEEE Trans. Pattern Anal. Mach. Intell., 30(11):1958–1970,

§ CHECKLIST

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?
See section <ref>.

* Did you discuss any potential negative societal impacts of your work?
Our evaluation focuses on CNN design in general and is not specific enough to any (potentially harmful) application of image classification.

* Have you read the ethics review guidelines and ensured that your paper conforms to them?

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
It is not possible for us to add the code in a double-blind way. The final version of this paper will contain a link to the code and instructions to reproduce every experiment we conducted.

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
We show the most important training details in our experimental setup section (see section <ref>). Every last detail will be visible from the actual training code.

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Since we are running experiments on a population of CNN architectures we get more information out of our computational resources by growing the population rather than training each architecture multiple times. However we conducted an ablation study to verify that the level of variability introduced by changing the CNN architectures is far greater then the variability from random initialization effects (see section <ref> in the appendix).

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
We mention the number of GPU hours and used hardware in a footnote of the introduction.

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?
All the datasets are properly cited.

* Did you mention the license of the assets?
We do not specifically mention the licenses of the individual dataset, but we mention that they are all sourced form platforms designed to share scientific data.

* Did you include any new assets either in the supplemental material or as a URL?
We will add an URL to a github repository with our code as well as instructions on how to run it and a download link to all of our trained models. We are not able to do this in a blind fashion so the URL is omitted from this version.

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?
We only use publicly available data published by the original authors with the intent for it to be used in research.

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Most of the datasets we use are of technical nature and therefore not prone problematic samples. The exceptions—Cifar and ImageNet—are extremely widely accepted and used.

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

§ EXTENDED DATASET DESCRIPTION

Example images from each dataset. Images of Cifar10/100 are magnified fourfold, the rest are shown in their original resolution (best viewed by zooming into the digital document).

[43] contains $40$ thousand image snippets produced from $458$ high-resolution images that have been captured from various concrete buildings on a single campus. It contains two classes, positive (which contains cracks in the concrete) and negative (with images that show intact concrete). With $20$ thousand images in both classes the dataset is perfectly balanced.

[44] contains $43$ thousand image snippets taken form the MLC dataset [45], which is a subset of the images collected at the Moorea Coral Reef Long Term Ecological Research site. It contains images from three reef habitats and has nine classes. The class distribution is very skewed with crustose coralline algae (CCA) being the most common by far (see Figure <ref> in the Appendix).

[14] (ILSVRC 2012) is a large scale dataset containing $1.3$ million photographs sourced from flickr and other search engines. It contains $1000$ classes and is well balanced with almost all classes having exactly $1300$ training and $50$ validation samples.

[46] ("Human Against Machine with $10000$ training images") is comprised of dermatoscopic images, collected from different populations and by varied modalities. It is a representative collection of all important categories of pigmented lesions that are categorized into seven classes. It is imbalanced with an extreme dominance of the  melanocytic nevi (nv) class (see Figure <ref> in the Appendix).

[47] contains images taken in different seasons as well as weather conditions from $21$ different regions in Turkey. It has two classes, positive (that contain powerlines) and negative (which do not). The dataset contains $8000$ images and is balanced with $4000$ samples per classes. Both classes contain $2000$ visible-light and $2000$ infrared images.

[48] contains $63$ thousand images of $291$ insect species. The images have been taken of the collection of British carabids from the Natural History Museum London. The dataset is not completely balanced but the majority of classes have $100$ to $400$ examples.

Intel Image Classification
[49] dataset (“natural”) is a natural scene classification dataset containing $25$ thousand images and $6$ classes. It is very well balanced with all classes having between $2.1$ thousand and $2.5$ thousand samples.

Cifar10 and Cifar100
[15] both consist of $60$ thousand images. The images are sourced form the $80$ million tiny images dataset [50] and are therefore of similar nature (photographs of common objects) as the images found in ImageNet, bar the much smaller resolution. Cifar10 has $10$ classes with $6000$ images per class, Cifar100 consists of $600$ images in $100$ classes, making both datasets perfectly balanced.

§ ADDITIONAL ABLATION STUDIES

This Chapter contains additional studies not suited for the main text. Most of these studies are designed to test for possible flaws or vulnerabilities in our experiments and therefore further strengthen the empirical robustness of our results.

§.§ Stability of Empirical Results on Cifar10

The top-1 errors of our sampled architectures on Cifar10 lie roughly between $18$ and $40$, which is fairly poor, not only compared to the state of the art but also compared to performance that can be achieved with fairly simple models. This calls into question if our Cifar10 results are flawed in a way that might have lead us to wrong conclusions. We address this by running additional tests on Cifar10 and evaluate their impact on our main results.
We get a goalpost for what performance would be considered good with our style of neural network and training setup by running the baseline code for Cifar10 published by Radosavovic et al. [36]. Table <ref> shows that these baseline configurations achieve much lower error rates.
We aim to improve the error results on Cifar10 in two ways: First we train our architecture population with standard settings for $200$ epochs instead of $30$, second we replaced the standard network stem with one that is specifically built for Cifar10, featuring less stride and no pooling. Figure <ref> shows scatterplots of the errors from all $500$ architectures on Cifar10 against the errors on ImageNet and ImageNet-10. We can see that both new training methods manage to significantly improve the performance with a minimum top-1 error below $10$ in both cases. More importantly can we observe that both new training methods have, despite lower overall error, a very similar error relationship to ImageNet. The error correlation is even slightly lower than with our original training (replicated in Figure <ref> left row). We can also see that in all three cases the error relationship can be significantly strengthened by replacing ImageNet with ImageNet-10, this shows that tuning for individual performance on a dataset does not significantly impact the error relationships between datasets which further strengthens our core claim.

Top-1 error of reference network implementations [36] for Cifar10.

Model     ResNet-56     ResNet-110     AnyNet-56     AnyNet-110

Error     5.91          5.23           5.68          5.59

The Cifar10 test errors of all 500 architectures plotted against ImageNet (top row) and ImageNet-10 (bottom row), shown for our original Cifar10 training (left column), training with a Cifar10 specific stem in the architecture (middle column), and training for $200$ epochs, which is roughly $6$ times longer (right column). The plots show that the error correlation with ImageNet-10 is much larger in all three cases, confirming that optimizing for individual Cifar10 performance does not alter our core result.

§.§ Verifying Training Duration

Since we have a limited amount of computational resources and needed to train a vast number of networks we opted to train the networks up to the number of epochs where they started to saturate significantly in our pre-studies. As we have seen in section <ref> can the network performance still improve quite a bit if it is trained for much longer. Even though the improved performances on Cifar10 did not yield any results contradicting the findings of our study, we still deemed it necessary to closer inspect what happened in the later stages of training and thus performed a sanity check for Cifar10 as well as the other two datasets that show a negative error correlation with ImageNet—Powerline and Natural.
Figure <ref> shows the Cifar10 test error curves of $20$ randomly selected architectures over $200$ epochs. On the left side we see the same curves zoomed in to epochs $30$ to $200$. We see that the error decreases steadily for all architectures, the ranking among architectures barely changes past epoch $30$. The relative performance between architectures and not absolute error rates are relevant for our evaluations, we can therefore conclude that the errors at epoch $30$ are an accurate enough description of an architecture's power.

Cifar10 test error curves of $20$ randomly sampled architectures trained over $200$ epochs (left). The same error curves but cut to epochs $30$ to $200$.

For Powerline and Natural, we select the five best and five worst architectures respectively and continue training them for a total of five times the regular duration. Figure <ref> shows the resulting error curves. Both datasets exhibit minimal changes in the errors of the top models. On Natural we observe clear improvements on the bottom five models but similar to Cifar10 there are very little changes in terms of relative performance. Powerline exhibits one clear cross-over but for the remainder of the bottom five models the ranking also stays intact. Overall we can conclude that longer training does not have a significant effect on the APR of our datasets.

Test error curves of the five best and five worst models on Powerline and Natural, respectively, when training is continued to epoch 100

§.§ Impact of Training Variability

The random initialization of the model weights has an effect on the performance of a CNN. In an empirical study it would therefore be preferable to train each model multiple times to minimize this variability. We opted to increase the size of our population as high as our computational resources allow, this way we get a large number of measurements to control random effects as well as an error estimate of a large set of architectures. However, we still wanted to determine how much of the total variability is caused by training noise and how much is due to changing the architectures. We estimate this by selecting two of the sampled CNN designs, number $147$ performing slightly above average with an error of $e_{147}=11.9$ and number $122$ performing slightly below average with $e_{122}=14.5$. The quantiles of the error distribution from all $500$ architectures are $q_{0.25}=11.53$, $q_{0.5}=13.02$ and $q_{0.75}=15.46$ with an overall mean of $\mu=13.9$. We then train the architectures $147$ and $122$ each $250$ times. Figure <ref> shows the error distributions of both selected architectures as well as the overall distribution obtained from training each of the $500$ architectures once. There is of course some variability within both architectures but both individual architectures produce very narrow densities and show essentially no overlap. We can therefore conclude that the effect of choosing an architecture is much greater than the variability caused by random training effects.

Error distributions on Cifar10 of two architectures ($122$, $147$) both trained from scratch $250$ times as well as the Cifar10 error distribution of all $500$ architectures. The plot shows that the variability caused by changing architecture is much larger than the one caused by random training effects.

§.§ Relationship of Top-1 with Top-5 Error on ImageNet, Insects and Cifar100

We opted to use top-5 error since it is the most widely reported metric for ImageNet and the top-5 numbers are therefore easy to interpret on that dataset. Many of our datasets have a significantly lower number of classes such that top-5 error makes little sense and we opted to use top-1 for those. This raises the question if comparing top-1 with top-5 errors introduces unwanted perturbations into our analysis. We therefore compare the top-1 and top-5 errors for the three datasets on which we use top-1 error (see Figure <ref>). We see that the two metrics have an almost linear relationship for the ImageNet and Cifar100 datasets. More importantly are the top-1 to top-5 error graphs monotonically ascending for all three datasets, such that the ordering of architectures does not change when swapping between the two metrics. Since we are interested in the relative performances of our sampled architectures changing between top-1 and top-5 error does not impact our analysis.

Top-1 error plotted against top-5 error of all $500$ architectures on ImageNet, Cifar100, and Insects. The plots reveal that on all three datasets the errors have a very close relationship: it is not perfectly linear but is monotonically ascending.

§.§ Overfitting of High-Capacity Architectures

The best architectures on Powerline, Natural and Cifar100 have a very small cumulated depth, so it is only natural to ask if the deeper architectures perform poorly due to overfitting. We address this concern by plotting the training errors of Powerline, Natural, and Cifar100 against the cumulative block depths (see Figure <ref>). The training errors are strongly correlated with the cumulative block depth, just like the test errors. Plots of the cumulated block depth show almost the same structure for training and test errors. We can therefore exclude overfitting as a reason why the shallower networks perform better on Powerline, Natural, and Cifar100.

Training errors of the sampled architectures (x-axis) plotted against the cumulated block depth for the $3$ datasets that have the lowest test errors on shallow architectures. We observe that for all three datasets shallow architectures also have the lowest training errors. Therefore overfitting is not the cause of this behaviour.

§.§ Disentangling the Effects of Class Count and Dataset Size

A core contribution of our paper is that we show how sub-sampled versions of ImageNet matching the number of classes of the target dataset tend to represent the APR of said target dataset far better. A side effect of downsampling ImageNet to a specific number of classes is that the total number of images present in the dataset also shrinks. This raises the question if the increase in error correlation is actually due to the reduced dataset size rather than to the matching class count. We disentangle these effects by introducing another downsampled version of ImageNet, Imagenet-1000-10. It retains all $1000$ classes but only $10$ examples per class resulting in a datastet with the same number of classes as ImageNet but with the total number of images of ImageNet-10. We train our population of architectures on ImageNet-1000-10 and show the error relationship of Cifar10, Natural, and Powerline with ImageNet-1000-10 (as well as with ImageNet and ImageNet-10 as a reminder) in Figure <ref>. The plots show that there are some correlation gains by using ImageNet-1000-10 over ImageNet, but the effect is far lower compared to ImageNet-10. This shows that downsampling size has a minor positive effect but the majority of the gain in APR similarity achieved trough class downsampling actually stems from the reduced the class number.

The errors of all 500 architectures on Cifar10, Natural, and Powerline plotted against the errors on ImageNet (top row), ImageNet-1000-10 (middle row) and ImageNet-10 (bottom row). We observe that class-wise downsampling has the largest positive effect on error correlation.

§.§ Impact of Class Distribution

MLC2008 and HAM1000 have a strong class imbalance. They both have one class which makes up a large amount of the dataset. In order to study the impact of an imbalanced class distribution, we created two new more balanced datasets out of the existing data the following way: we reduced the number of samples in the overrepresented class such that it has the same amount of samples as the second most common class. We call these datasets MLC2008-balanced and HAM10000-balanced. Their new class distributions can be seen in Figure <ref>. We train our architecture population on MLC2008-balanced and HAM10000-balanced leaving the training configuration otherwise unaltered. Figure <ref> shows the errors on the balanced datasets versus the errors on the unbalanced counterparts.

Class distributions of MLC2008, HAM10000, and their balanced versions.

Errors of all $500$ sampled architectures on MLC2008-balanced and HAM1000-balanced (y-axis) plotted against the errors of their unbalanced counterparts (x-axis). The top $10$ performances on the target dataset are plotted in orange, the worst $10$ performances in red. We observe a clear positive correlation for both datasets, hence we conclude that the dataset imbalance has a limited impact on the APRs.

For both HAM10000 and and MLC2008, there is a strong correlation between the errors on the balanced and unbalanced datasets. We can therefore conclude that class imbalance is not a determining factor for the APRs of HAM10000 or MLC2008.

§ IMPACT OF PRETRAINING

The main objective of our study is to identify how well different CNN designs perform on varying datasets and if the best architectures are consistent across the datasets. For this reason we train all of our networks from scratch on each dataset. However, we cannot ignore that pretraining on ImageNet is a huge factor in practice and we therefore study its impact on our evaluations. To this end have we train all of our sampled architectures again on each dataset but this time we initialize their weights with ImageNet pretraining (we omit Concrete, which has very low errors even without pretraining). Figure <ref> shows the errors of each dataset without (blue) and with (green) pretraining plotted against the ImageNet errors. The data shows a distinct trend:
the performance improvement due to pretraining dictates how much stronger the correlation of the pretrained errors with the ImageNet errors is.
For Cifar10 and Cifar100 where the performance gain with pretraining is low to moderate the error correlations do not drastically change. On the other end of the spectrum are Natural and Powerline, where pretraining leads to drastically lower errors. This in turn leads to much higher error correlation with ImageNet(the Powerline correlation can not grow significantly above 0 because the overall errors are so small across all architectures). This shows us that our findings still apply when pretraining is used, but their effects can be masked when pretraining is the most important factor contributing to the overall final performance.

Errors form all 500 architectures trained from scratch (blue) as well as the same architectures pretrained on ImageNet (green), plotted against the respective ImageNet errors. We observe that the error correlation with ImageNet increases relative to the performance gain due to pretraining.

§ RESTRICTING PARAMETERS TO BE MONOTONICALLY INCREASING ALONG STAGES

We have observed that architectures with increasing parameter values across successive stages have tend to perform better on ImageNet. We thus study the impact of restricting the architectures to having strictly increasing parameter values across the stages on the other datasets. We create four subsets of our architecture population by filtering it for ascending bottleneck ratio, block depth, group width and block width each individually.
Figure <ref> shows the $eCDF$ of the whole pouplation as well as the $eCDFs$ of the restricted sub-populations for every dataset.
For most datasets the restrictions lead to a population with a better error distribution. The effect is most impactful for the models with ascending block width on ImageNet and Insects. Again, Powerline, Natural and to some extent Cifar10 are the odd ones where most restrictions lead to a worse overall performance.

The $eCDFs$ of the restricted sub-populations in the context of the $eCDF$ of the whole architecture population, for all datasets.

§ STRUCTURE OF TOP PERFORMING ARCHITECTURES

Figure <ref> shows the configuration of the top performing architecture in blue, as well as the mean and standard deviation of the top $15$ configurations for every dataset. We observe that the top $15$ architectures have very high variance in both bottleneck ratio and group width.

Block width on the other hand shows a clear pattern: almost all high-performing architectures start with a very small block width that increases across the stages. Only Powerline and Natural do not show this pattern. In block depth, we observe a similar pattern with a bit more noise. For block depth, Powerline, Natural, Cifar10 and Cifar100, no such trend of increased parameter values towards the later stages is observed.

Configurations of the top-performing architectures, with the four stages depicted on the x-axis and the parameter values on the y-axis. The best architectures are shown in blue, the mean of the top $15$ architectures is depicted in orange with with a vertical indication of one standard deviation.

§ COMPLETE PARAMETER BY STAGE BY DATASET SCATTERPLOTS

Figures <ref> to <ref> show parameter versus error scatterplots, split across the dimensions dataset, stage and parameter. A visual inspection of these plots reveals that block depth and block width show the most tangible patterns.

Individual parameter by stage versus error scatterplots for the Concrete dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the MLC2008 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the ImageNet dataset. The x-axis gives the error, while the parameter values are given on the ordinate. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the HAM10000 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the Powerline dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the Insects dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the Natural dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the Cifar10 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the Cifar100 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the ImageNet-100 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the ImageNet-10 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the ImageNet-5 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.

Individual parameter by stage versus error scatterplots for the ImageNet-2 dataset. The x-axis gives the error, while the parameter values are given on the ordinate.


capbtabboxtable[][0.4]

# Improving Cell-Free Massive MIMO Detection Performance via Expectation
Propagation

Alva Kosasih1, Vera Miloslavskaya1, Wibowo Hardjawana1, Victor Andrean2, and
Branka Vucetic1  1Centre of Excellence in Telecommunications, University of
Sydney, Sydney, Australia.
<EMAIL_ADDRESS>2Mobilizing Information Technology Lab., National Taiwan University of Science
and Technology, Taipei, Taiwan.

###### Abstract

Cell-free (CF) massive multiple-input multiple-output (M-MIMO) technology
plays a prominent role in the beyond fifth-generation (5G) networks. However,
designing a high performance CF M-MIMO detector is a challenging task due to
the presence of pilot contamination which appears when the number of pilot
sequences is smaller than the number of users. This work proposes a CF M-MIMO
detector referred to as CF expectation propagation (CF-EP) that incorporates
the pilot contamination when calculating the posterior belief. The simulation
results show that the proposed detector achieves significant improvements in
terms of the bit-error rate and sum spectral efficiency performances as
compared to the ones of the state-of-the-art CF detectors.

###### Index Terms:

Cell-free massive MIMO, distributed massive MIMO, expectation propagation,
detection, pilot contamination.

##  I Introduction

Cell-free (CF) massive multiple-input multiple-output (M-MIMO) technology has
been proposed to deliver ubiquitous coverage and high service quality for
beyond fifth-generation (5G) networks [1]. The access points (APs), known as
the base station antennas in standard MIMO literature, are geographically
distributed. The APs are managed by a central processing unit (CPU). Each user
equipment (UE) is served simultaneously by all APs and therefore experiences
no cell boundaries. This CF M-MIMO system provides additional diversity,
increases the coverage probability, and mitigates the inter-cell interference,
leading to a higher spectral efficiency (SE) and energy efficiency [2, 3, 4].
One of the critical issues is to design a detector for CF M-MIMO system which
can overcome various impairments, in particular the pilot contamination, i.e.
interference caused by pilot-sharing UEs [3].

The performance of the classical maximum ratio combining (MRC) detector in a
CF M-MIMO system, referred to as CF-MRC detector, has been investigated in
[2]. The CF-MRC detector estimates the transmitted symbols by multiplying
corresponding received signals with the complex conjugate of the channel
matrix. Although the CF-MRC detector can achieve an extremely low
computational complexity, it fails to overcome the pilot contamination impact,
which leads to a severe performance degradation [3]. To deal with the pilot
contamination in CF M-MIMO systems, a modified minimum mean square error
(MMSE) detector, referred to as CF-MMSE, has been proposed in [5]. It is shown
to significantly outperform the CF-MRC detector by incorporating the
statistical channel estimation error which contains the pilot contamination
term when detecting the symbols. Specifically, the variance of the channel
estimation error is incorporated into the regularization factor in case of the
CF-MMSE detector. To further improve the performance of the CF-MMSE detectors,
interference cancellation techniques have been combined with the CF-MMSE
detectors as proposed in [4, 3, 6]. It is widely known that the MMSE with
successive interference cancellation (SIC) detectors can achieve a high
reliability performance at the cost of a higher computational complexity
compared to the standard MMSE detector [7]. The CF-MMSE-SIC detector,
combining the CF-MMSE scheme with the SIC scheme has been proposed in [3].
Unfortunately, the CF-MMSE-SIC detector provides only a minor performance gain
compared to the CF-MMSE detector.

Recently, Bayesian machine learning techniques, called expectation propagation
(EP) [8] and approximate message passing (AMP) [9], have been employed for
detection to reduce the reliability performance gap between ML and suboptimal
detectors in the collocated M-MIMO systems. In [9], the AMP algorithm has been
shown to achieve a near optimal performance for independent and identically
distributed (i.i.d.) Rayleigh fading channels. However, it performs poorly for
non-i.i.d Rayleigh fading channels [10, 11]. The EP algorithm is an iterative
algorithm used to infer estimates of the transmitted symbols by approximating
their posterior probability density function (pdf) with factorizable Gaussian
posterior belief [8]. For each posterior belief, EP retains only expectations
such as mean and variance. The factorizable belief not only greatly reduces
the complexity of the posterior pdf inference, but also enables the EP to
achieve a near optimal performance [12].

We propose a novel iterative CF M-MIMO detector based on the EP algorithm,
referred to as the CF-EP detector. The main contributions are summarized as
follows:

  * •

To the best of our knowledge, this is the first Bayesian detector for CF
M-MIMO systems which can achieve a high detection reliability in the presence
of the pilot contamination.

  * •

The Gaussian posterior belief calculation of the original EP detector [8] is
modified for systems with the pilot contamination, which is the main
performance degradation factor in CF M-MIMO systems.

The simulation results demonstrate a significant BER and SE performance
improvements compared to the state-of-the-art [2, 5, 4, 3] with a comparable
computational complexity.

Notations: $\mathbf{I}$ denotes an identity matrix. For any matrix
$\mathbf{A}$, the notations $\mathbf{A}^{T}$, $\mathbf{A}^{H}$, ${\rm
tr}(\mathbf{A})$, and $\mathbf{A}^{\dagger}$ stand for transpose, conjugate
transpose, trace, and pseudo-inverse of $\mathbf{A}$, respectively. $\|{\bf
q}\|$ denotes the Frobenius norm of vector ${\bf q}$. $q^{*}$ denotes the
complex conjugate of a complex number $q$. ${{\sf E}}[{\bf x}]$ is the mean of
random vector ${\bf x}$, and ${\mathrm{Var}}[{\bf x}]={{\sf
E}}\big{[}\left({\bf x}-{{\sf E}}[{\bf x}]\right)^{2}\big{]}$ is its variance.
${\mathcal{N}}_{\mathbb{C}}(x_{k}:c_{k},v_{k})$ represents a complex single
variate Gaussian distribution for a random variable $x_{k}$ with mean $c_{k}$
and variance $v_{k}$. Let ${\bf x}=[x_{1},\cdots,x_{K}]^{T}$ and ${\bf
c}=[c_{1},\cdots,c_{K}]^{T}$.

##  II System Model

We consider an uplink CF M-MIMO system which consists of $L$ geographically
distributed single-antenna APs, serving $K$ single antenna UEs, where $K<<L$.
All APs send their pilot and data signals to the CPU which then estimates the
channel state information (CSI) and transmitted symbols. We use a block fading
model, where the channel between the $k$-th UE and the $l$-th AP is expressed
as

$h_{l,k}=\sqrt{\beta_{l,k}}g_{l,k}.$ (1)

Here, $g_{l,k}\sim\mathcal{N}_{\mathbb{C}}\left(0,1\right)$ is the complex
small scale fading coefficient and $\beta_{l,k}$ is the complex large-scale
fading coefficient that describes geometric pathloss and shadowing. We assume
that $\beta_{l,k}$ is known to the CPU. The channel is considered to be
constant in time-frequency blocks of $\tau_{c}$ channel uses. Note that
precise value of $h_{l,k}$ is unavailable to the receiver. Therefore, the
channel is estimated by using the minimum mean-square error criterion, i.e.
minimization of ${\sf E}[\|h_{l,k}-\hat{h}_{l,k}\|^{2}]$, where
$\hat{h}_{l,k}$ denotes the MMSE channel estimate of UE $k$ from AP $l$. This
results in an optimal estimation quality [13].

Figure 1:  Cell-free expectation propagation (CF-EP) detector

###  II-A Uplink Pilot Transmission and MMSE Channel Estimation

The channels between UEs and APs are estimated by using $\tau$ mutually
orthogonal pilot signals
$\boldsymbol{\phi}_{1},\dots,\boldsymbol{\phi}_{\tau}$ with
$\boldsymbol{\phi}_{i}\in\mathbb{C}^{\tau\times
1},\|\boldsymbol{\phi}_{i}\|^{2}=\tau,$ and $i=1,\dots,\tau$. The pilots are
randomly assigned to the UEs. In practice, $\tau<K$ which results in a pilot
contamination [2]. Let $t_{k}$ be the index of the pilot signal assigned to
the $k$-th UE, $1\leq k\leq K$. Obviously, $t_{k}\in\\{1,\dots,\tau\\}$. We
group UEs into $\tau$ sets $\mathcal{S}_{1},\dots,\mathcal{S}_{\tau}$ such
that all UEs in the set use the same pilot, i.e.
$\mathcal{S}_{i}\triangleq\\{k\in\\{1,\dots,K\\}|t_{k}=i\\}$ for $1\leq
i\leq\tau$. At AP $l$, the received pilot ${\bf
z}_{l}\in\mathbb{C}^{\tau\times 1}$ is given as follows,

${\bf z}_{l}=\sqrt{p}\sum_{i=1}^{K}h_{l,i}\boldsymbol{\phi}_{t_{i}}+{\bf
v}_{l},$ (2)

where $p$ is the uplink transmit power of a UE111All UEs are assumed to
transmit with equal power. and ${\bf v}_{l}\in\mathbb{C}^{\tau\times
1}\sim\mathcal{N}_{\mathbb{C}}(\mathbb{0},\sigma^{2}{\bf I})$ is an additive
Gaussian noise with zero mean and variance $\sigma^{2}$ equal to the noise
power.

The MMSE channel estimate of UE $k$ from AP $l$ is given in [3] as

$\hat{h}_{l,k}=\frac{\sqrt{p\tau}\beta_{l,k}}{\sigma^{2}+p\tau\sum_{i\in\mathcal{S}_{{{t_{k}}}}}\beta_{l,i}}\times\frac{{\boldsymbol{\phi}_{t_{k}}^{H}}}{\sqrt{\tau}}{\bf
z}_{l}$ (3)

By substituting ${\bf z}_{l}$ in (2) into (3), we obtain

$\hat{h}_{l,k}=\frac{\sqrt{p\tau}\beta_{l,k}}{\sigma^{2}+p\tau\sum_{i\in\mathcal{S}_{{{t_{k}}}}}\beta_{l,i}}\times\left(\sqrt{p\tau}\sum_{i\in\mathcal{S}_{t_{k}}}h_{l,i}+\tilde{{\bf
v}}_{l}\right),$ (4)

where each element in $\tilde{{\bf v}}_{l}$ follows a Gaussian distribution
with zero mean and variance $\sigma^{2}$. From (4) and (1), we can easily
verify that ${\sf E}\left[\hat{h}_{l,k}\right]=0$ and
${\alpha_{l,k}\triangleq{\sf
E}}\left[\hat{h}_{l,k}(\hat{h}_{l,k})^{*}\right]=\frac{p\tau\beta_{l,k}^{2}}{\sigma^{2}+p\tau\sum_{i\in\mathcal{S}_{{t_{k}}}}\beta_{l,i}}$,
$\hat{h}_{l,k}\sim\mathcal{N}_{\mathbb{C}}\left(0,\alpha_{l,k}\right)$.
Therefore, the channel estimation error of UE $k$ from AP $l$

$\epsilon_{l,k}=h_{l,k}-\hat{h}_{l,k}$ (5)

follows a Gaussian distribution with mean zero and variance
$C_{l,k}\triangleq{\sf
E}[{\epsilon}_{l,k}({\epsilon}_{l,k})^{*}]=\beta_{l,k}-\alpha_{l,k}$. The
channel estimation error from the $k$-th UE to all APs is defined in a vector
form as
$\boldsymbol{\epsilon}_{k}\triangleq[\epsilon_{1,k},\dots,\epsilon_{L,k}]^{T}$,
where $\boldsymbol{\epsilon}_{k}\sim\mathcal{N}_{\mathbb{C}}\left(0,{\bf
C}_{k}\right)$. The covariance matrix ${\bf C}_{k}$ is an $L\times L$ diagonal
matrix whose diagonal elements are $C_{1,k},\dots,C_{L,k}$. The channel
estimation error matrix for $K$ UEs,
$\boldsymbol{\mathcal{E}}\triangleq[\boldsymbol{\epsilon}_{1},\cdots,\boldsymbol{\epsilon}_{K}]$
can be written as

$\boldsymbol{\mathcal{E}}={\bf H}-\hat{{\bf H}}.$ (6)

Since ${\epsilon}_{l,k}\sim\mathcal{N}_{\mathbb{C}}\left(0,C_{l,k}\right)$,
the channel estimation error matrix $\boldsymbol{\mathcal{E}}$ is normally
distributed with zero mean and covariance matrix ${\bf D}\triangleq{\sf
E}\left[\boldsymbol{\mathcal{E}}(\boldsymbol{\mathcal{E}})^{H}\right]=\sum_{k=1}^{K}{\bf
C}_{k}$, i.e.
$\boldsymbol{\mathcal{E}}\sim\mathcal{N}(\mathbf{0},\mathbf{D})$.

###  II-B Uplink Data Transmission

All UEs map their information bit streams to symbols that belong to
constellation of $M$-QAM, ${\boldsymbol{\Omega}}$, where
$\|{\boldsymbol{\Omega}}\|^{2}=M$. The $k$-th user symbol is denoted as
$x_{k}$. The average symbol energy is $E_{x}\triangleq{\sf E}[|x_{k}|^{2}]=1$.
The channel matrix between UEs and APs and its estimate are denoted as ${\bf
H}\in\mathbb{C}^{L\times K}$ and $\hat{{\bf H}}\in\mathbb{C}^{L\times K}$,
respectively. The $(l,k)$-th elements of ${\bf H}$ and $\hat{{\bf H}}$ are
defined by (1) and (3), respectively. The received signal of all APs,
$\mathbb{y}=[y_{1}\ldots y_{L}]^{T}$, available at the CPU, is given as

${\bf y}={\bf H}{\bf x}+{\bf n},$ (7)

where ${\bf n}\sim\mathcal{N}_{\mathbb{C}}(\mathbb{0},\sigma^{2}{\bf I})$ is a
Gaussian noise and ${\bf x}=[x_{1},\cdots,x_{K}]^{T}$ is uniformly distributed
on ${\boldsymbol{\Omega}}^{K}$. By substituting $\hat{{\bf H}}$ in (6) into
(7), we obtain

${\bf y}=\hat{{\bf H}}{\bf x}+\underbrace{\boldsymbol{\mathcal{E}}{\bf x}+{\bf
n}}_{{\bf w}},$ (8)

at the CPU. To detect the symbols we can use the MMSE channel estimate
$\hat{{\bf H}}$, which introduces the channel estimation error
$\boldsymbol{\mathcal{E}}$ as explained in the Section IIA. Considering that
${\sf E}\left[\boldsymbol{\mathcal{E}}{\bf x}+{\bf n}\right]=\mathbb{0}$ and
$\mathrm{Var}\left[\boldsymbol{\mathcal{E}}{\bf x}+{\bf n}\right]={\bf
D}+\sigma^{2}{\bf I}$, we define a new noise vector ${\bf
w}\sim\mathcal{N}_{\mathbb{C}}(\mathbb{0},{\bf D}+\sigma^{2}{\bf I})$.

##  III Cell-Free Expectation Propagation Detector

In this section, we propose a novel CF M-MIMO detector based on the EP concept
[8]. Our detector takes the effect of the pilot contamination into
consideration. We refer it as the cell-free expectation propagation (CF-EP)
detector with the architecture shown in Fig. 1. The CF-EP consists of two
modules: the observation module calculating the posterior beliefs based on the
received signal and the estimation module yielding soft transmitted symbol
estimates. The CF-EP iteratively exchanges the outputs of both modules until
the convergence criterion is satisfied.

###  III-A The Observation Module

Given the received signal ${\bf y}$ in (8), the posterior distribution of the
transmitted symbols is characterized by

$\displaystyle p({\bf x}|{\bf y})=\frac{p({\bf y}|{\bf x})}{p({\bf y})}\times
p({\bf x})\propto\underbrace{\mathcal{N}_{\mathbb{C}}\left({\bf y}:\hat{{\bf
H}}{\bf x},{\bf D}+\sigma^{2}{\bf I}\right)}_{p({\bf y}|{\bf
x})}\underbrace{\prod_{k=1}^{K}p(x_{k})}_{p({\bf x})},$ (9)

where $p(x_{k})=\frac{1}{M}\sum_{x\in{\boldsymbol{\Omega}}}\delta(x_{k}-x)$ is
a priori pdf of $x_{k}$, $\delta$ is the Dirac delta function, and $p({\bf
y})$ is omitted as it does not depend on the distribution of ${\bf x}$. A
direct calculation of (9) results in a prohibitively intensive computation as
we need to consider all possible combinations of the symbols. Therefore, EP is
used to iteratively construct a Gaussian approximation to the true posterior
distribution of the transmitted symbol vector.

Concretely, the EP constructs a Gaussian posterior belief $p^{(t)}({\bf
x}|{\bf y})\approx p({\bf x}|{\bf y})$, where $t$ is the iteration number.
This involves replacing $p({\bf x})$ in (9) with a distribution from the
exponential family, $\chi^{(t)}({\bf
x})\propto\mathcal{N}_{\mathbb{C}}\left({\bf
x}:({\boldsymbol{\lambda}}^{(t)})^{-1}{\boldsymbol{\gamma}}^{(t)},({\boldsymbol{\lambda}}^{(t)})^{-1}\right)$
[14]. Here, ${\boldsymbol{\lambda}}^{(t)}$ is a $K\times K$ diagonal matrix
with diagonal elements $\lambda_{k}^{(t)}>0$ and
${\boldsymbol{\gamma}}^{(t)}=[\gamma_{1}^{(t)},\dots,\gamma_{K}^{(t)}]^{T}$.
Both $\lambda_{k}^{(t)}$ and $\gamma_{k}^{(t)}$ are complex numbers with
$\lambda_{k}^{(0)}=1$ and $\gamma_{k}^{(0)}=0$. To calculate the Gaussian
posterior belief, we first treat ${\bf x}$ as a random vector and approximate
$p({\bf y}|{\bf x})$ as $\mathcal{N}_{\mathbb{C}}\left({\bf x}:{\hat{{\bf
H}}}^{\dagger}{\bf y},\left(\hat{{\bf H}}^{H}({\bf D}+\sigma^{2}{\bf
I})^{-1}\hat{{\bf H}}\right)^{-1}\right)$. Note that the matrix ${\bf D}$
corresponds to the channel estimation error variance which is caused by the
pilot contamination, defined in Section IIA. The Gaussian posterior belief is
then expressed as

$\displaystyle p^{(t)}({\bf x}|{\bf y})\propto$ $\displaystyle p({\bf y}|{\bf
x})\cdot\chi^{(t)}({\bf x})$ $\displaystyle\propto$
$\displaystyle\mathcal{N}_{\mathbb{C}}\left({\bf x}:{\hat{{\bf
H}}}^{\dagger}{\bf y},\left(\hat{{\bf H}}^{H}({\bf D}+\sigma^{2}{\bf
I})^{-1}\hat{{\bf H}}\right)^{-1}\right)$
$\displaystyle\cdot\mathcal{N}_{\mathbb{C}}\left({\bf
x}:({\boldsymbol{\lambda}}^{(t)})^{-1}{\boldsymbol{\gamma}}^{(t)},({\boldsymbol{\lambda}}^{(t)})^{-1}\right)$
$\displaystyle\propto$ $\displaystyle\mathcal{N}_{\mathbb{C}}\left({\bf
x}:{\boldsymbol{\mu}}^{(t)},{\boldsymbol{\Sigma}}^{(t)}\right).$ (10)

We can compute the product of two Gaussians in (III-A) by using the Gaussian
product property222The product of two Gaussians results in another Gaussian,
$\mathcal{N}_{\mathbb{C}}({\bf x}:{\bf a},{\bf
A})\cdot\mathcal{N}_{\mathbb{C}}({\bf x}:{\bf b},{\bf
B})\propto\mathcal{N}_{\mathbb{C}}({\bf x}:({\bf A}^{-1}+{\bf
B}^{-1})^{-1}({\bf A}^{-1}{\bf a}+{\bf B}^{-1}{\bf b}),({\bf A}^{-1}+{\bf
B}^{-1})^{-1}$., given in Appendix A.1 of [15]. By using this property, we
obtain the variance and mean of $p^{(t)}({\bf x}|{\bf y})$ as

$\displaystyle{\boldsymbol{\Sigma}}^{(t)}={\left(\hat{{\bf H}}^{H}({\bf
D}+\sigma^{2}{\bf I})^{-1}\hat{{\bf
H}}+{\boldsymbol{\lambda}}^{(t)}\right)}^{-1}$ (11a)
$\displaystyle{\boldsymbol{\mu}}^{(t)}={\boldsymbol{\Sigma}}^{(t)}{\left(\hat{{\bf
H}}^{H}({\bf D}+\sigma^{2}{\bf I})^{-1}{\bf
y}+{\boldsymbol{\gamma}}^{(t)}\right)}.$ (11b)

######  Remark 1 (CF-EP Gaussian posterior belief)

The Gaussian posterior belief $p^{(t)}({\bf x}|{\bf y})$ in the CF-EP detector
is different from that in the existing EP detector, i.e. equation (27) in [8],
as the channel estimation error containing the pilot contamination term is
incorporated in the proposed CF-EP detector, specifically in (III-A).

We then compute $p^{(t)}({\bf y}|{\bf x})$ based on $p^{(t)}({\bf x}|{\bf
y})$,

$\displaystyle p^{(t)}({\bf y}|{\bf x})\triangleq\frac{p^{(t)}({\bf x}|{\bf
y})}{\chi^{(t)}({\bf x})}$
$\displaystyle\propto\frac{\mathcal{N}_{\mathbb{C}}\left({\bf
x}:{\boldsymbol{\mu}}^{(t)},{\boldsymbol{\Sigma}}^{(t)}\right)}{\mathcal{N}_{\mathbb{C}}\left({\bf
x}:({\boldsymbol{\lambda}}^{(t)})^{-1}{\boldsymbol{\gamma}}^{(t)},({\boldsymbol{\lambda}}^{(t)})^{-1}\right)}\propto\mathcal{N}_{\mathbb{C}}\left({\bf
x}:{\bf x}^{(t)}_{{\rm obs}},{\bf V}^{(t)}_{{\rm obs}}\right),$ (12)

where ${\bf x}^{(t)}_{\rm obs}=[x^{(t)}_{{\rm obs},1},\dots,x^{(t)}_{{\rm
obs},K}]$ and ${\bf V}^{(t)}_{\rm obs}$ is a $K\times K$ diagonal matrix with
$v^{(t)}_{{\rm obs},k}$ as the $k$-th diagonal element, which can be expressed
as

$\displaystyle v_{{\rm
obs},k}^{(t)}=\frac{\Sigma_{k}^{(t)}}{1-\Sigma_{k}^{(t)}\lambda_{k}^{(t)}}$
(13a) $\displaystyle x_{{\rm obs},k}^{(t)}=v_{{\rm
obs},k}^{(t)}{\left(\frac{\mu_{k}^{(t)}}{\Sigma_{k}^{(t)}}-\gamma_{k}^{(t)}\right)}.$
(13b)

Here, $\mu_{k}^{(t)}$ is the $k$-th element of vector
${\boldsymbol{\mu}}^{(t)}$ and $\Sigma_{k}^{(t)}$ is the $k$-th diagonal
element of matrix ${\boldsymbol{\Sigma}}^{(t)}$. The pair $\left({\bf x}_{{\rm
obs}}^{(t)},{\bf V}_{{\rm obs}}^{(t)}\right)$ from (11) is then forwarded to
the estimation module.

###  III-B The Estimation Module

Based on $p^{(t)}({\bf y}|{\bf x})$ provided by the observation module, the
estimation module computes a new posterior belief

$\hat{p}^{(t)}({\bf x}|{\bf y})\propto p^{(t)}({\bf y}|{\bf x})p({\bf x}).$
(14)

This is different from the Gaussian posterior belief $p^{(t)}({\bf x}|{\bf
y})$ in (III-A), obtained based on $\chi^{(t)}({\bf x})$ from exponential
families. In fact, the EP algorithm is used to ensure a similarity between the
mean-variance pairs of $p^{(t)}({\bf x}|{\bf y})$ and $\hat{p}^{(t)}({\bf
x}|{\bf y})$. This similarity is specified in [16] by the moment matching (MM)
condition, minimizing the Kullback-Leibler divergence between posterior
beliefs. The MM condition is expressed as follows

$\displaystyle\mathbb{E}_{p^{(t)}({\bf x}|{\bf y})}[{\bf
x}]\sim\mathbb{E}_{\hat{p}^{(t)}({\bf x}|{\bf y})}[{\bf x}],$ (15)

where $\mathbb{E}_{\hat{p}^{(t)}({\bf x}|{\bf y})}[{\bf x}]$ denotes the first
two moments, i.e. the mean and variance of ${\bf x}$ with respect to
$\hat{p}^{(t)}({\bf x}|{\bf y})$. Furthermore, the mean and variance of
$\hat{p}^{(t)}({\bf x}|{\bf y})[{\bf x}]$ are given in [8] as

${\bf V}^{(t)}={\sf E}\left[\left|{\bf x}-\hat{{\bf
x}}^{(t)}\right|^{2}\right]$ (16a) $\hat{{\bf x}}^{(t)}=c^{(t)}\sum_{{\bf
x}\in{\boldsymbol{\Omega}}^{K}}{\bf x}p^{(t)}({\bf y}|{\bf x}).$ (16b)

Here, $c^{(t)}$ is the normalization constant to ensure that $\sum_{{\bf
x}\in{\boldsymbol{\Omega}}^{K}}\hat{p}^{(t)}({\bf x}|{\bf y})=1$. The
iteration of the CF-EP detector will be terminated when

$\|\mathbb{E}_{p^{(t)}({\bf x}|{\bf y})}[{\bf
x}]-\mathbb{E}_{\hat{p}^{(t)}({\bf x}|{\bf y})}[{\bf x}]\|\leq 10^{-4},$ (17)

i.e. when (15) is satisfied, or once the algorithm once the maximum number of
iterations $T_{\rm max}$ is reached. Hard estimates of the transmitted symbols
are then made from $\hat{{\bf x}}^{(T)}$, by comparing its Euclidean distance
from the transmitted $M$-QAM symbol sets $\Omega$ where $T$ is the last
iteration number.

Figure 2: The average number of iterations of the CF-EP detector

Until (15) is satisfied, the estimation module will re-evaluate the Gaussian
posterior belief, ${p}^{(t)}({\bf x}|{\bf y})$, by first calculating
$\chi^{(t+1)}({\bf x})$ based on the $\hat{p}^{(t)}({\bf x}|{\bf y})$,
expressed as

$\displaystyle\chi^{(t+1)}({\bf x})$
$\displaystyle\triangleq\frac{\hat{p}^{(t)}({\bf x}|{\bf y})}{p^{(t)}({\bf
y}|{\bf x})}$ $\displaystyle\propto\frac{\mathcal{N}_{\mathbb{C}}\left({\bf
x}:\hat{{\bf x}}^{(t)},{\bf
V}^{(t)}\right)}{\mathcal{N}_{\mathbb{C}}\left({\bf x}:{\bf x}^{(t)}_{{\rm
obs}},{\bf V}^{(t)}_{{\rm obs}}\right)}$
$\displaystyle=\mathcal{N}_{\mathbb{C}}\left({\bf
x}:({\boldsymbol{\lambda}}^{(t+1)})^{-1}{\boldsymbol{\gamma}}^{(t+1)},({\boldsymbol{\lambda}}^{(t+1)})^{-1}\right),$
(18)

where

$\displaystyle{\boldsymbol{\lambda}}^{(t+1)}=({\bf V}^{(t)})^{-1}-({\bf
V}^{(t)}_{{\rm obs}})^{-1}$ (19a)
$\displaystyle{\boldsymbol{\gamma}}^{(t+1)}=({\bf V}^{(t)})^{-1}\hat{{\bf
x}}^{(t)}-({\bf V}^{(t)}_{{\rm obs}})^{-1}{\bf x}^{(t)}_{{\rm obs}}.$ (19b)

Note that ${\boldsymbol{\lambda}}$ and ${\boldsymbol{\gamma}}$ are referred to
as the updating parameters as they are used to update the Gaussian posterior
beliefs. We smoothen the update of
$({\boldsymbol{\lambda}},{\boldsymbol{\gamma}})$ by using a convex combination
with the former values,

$\displaystyle{\boldsymbol{\lambda}}^{(t+1)}$
$\displaystyle=(1-\eta){\boldsymbol{\lambda}}^{(t+1)}+\eta{\boldsymbol{\lambda}}^{(t)}$
(20a) $\displaystyle{\boldsymbol{\gamma}}^{(t+1)}$
$\displaystyle=(1-\eta){\boldsymbol{\gamma}}^{(t+1)}+\eta{\boldsymbol{\gamma}}^{(t)},$
(20b)

where $\eta\in[0,1]$ is a weighting coefficient.

The estimation module sends the updating parameters
$({\boldsymbol{\gamma}}^{(t+1)},{\boldsymbol{\lambda}}^{(t+1)})$ to the
observation module, as illustrated in Fig. 1. The complete pseudo-code is
shown in Alg. 1. One can also compute the SE of the CF-EP detector by using
signal-to-interference-noise ratio, given in[17] as

${\rm SE}^{\rm CF-EP}_{k}=\left(1-\frac{\tau_{p}}{\tau_{c}}\right){\sf
E}\left[{\sf log}_{2}\left(1+{{\rm SINR}_{k}^{\rm CF-EP}}\right)\right],$ (21)

where ${{\rm SINR}_{k}^{\rm CF-EP}}=\frac{K}{{\sf tr}\left({\bf V}_{\rm
obs}^{(T)}\right)}$.

Algorithm 1 CF-EP detector

1:Input:  ${\boldsymbol{\gamma}}^{(0)}={\bf
0},{\boldsymbol{\lambda}}^{(0)}=\textbf{I},\eta=0.7,t=0,T_{\rm max}=10$

2:Output:  Hard symbol estimates from $\hat{{\bf x}}^{(t)}$

3:for $t=1,\dots,T_{\rm max}$ do

4: The Observation Module:

5: Compute ${\boldsymbol{\Sigma}}^{(t)}$ and ${\boldsymbol{\mu}}^{(t)}$ in
(11)

6: Compute $v_{{\rm obs},k}^{(t)}$ and $x_{{\rm obs},k}^{(t)}$ in (13),
$k=1,\dots,K$

7: The Estimation Module:

8: Compute ${\bf V}^{(t)}$ and $\hat{{\bf x}}^{(t)}$ in (16)

9: Compute ${\boldsymbol{\lambda}}^{(t+1)}$ in (19a) and smoothen it using
(20a)

10: Compute ${\boldsymbol{\gamma}}^{(t+1)}$ in (19b) and smoothen it using
(20b)

11: if (17) is satisfied then

12: break

13: end if

14:end for

15:$T:=t$

16:Calculate hard symbol estimates from $\hat{{\bf x}}^{(T)}$

##  IV Numerical Results

We follow the simulation setup used in [3]. Specifically, we consider $L=100$
single-antenna APs, randomly deployed in a $1\times 1$ km area, serving $K=60$
randomly located UEs, in an urban environment complying with 3GPP Urband
Microcell model [18]. The large scale fading coefficient is given as

$\beta_{l,k}[dB]=-30.5-36.7{\sf log}_{10}\left(\frac{d_{l,k}}{1{\rm
m}}\right)+F_{l,k},$ (22)

where $d_{l,k}$ is the distance between AP $l$ and UE $k$ and
$F_{l,k}\sim\mathcal{N}_{\mathbb{C}}(0,4^{2})$ is the shadow fading. The
shadowing terms from an AP to different UEs are correlated as ${\sf
E}[F_{l,k}F_{l,i}]=4^{2}2^{-\delta_{k,i}/9{\rm m}}$, where $\delta_{k,i}$ is
the distance between UE $k$ and UE $i$. The pilots are randomly indexed and
assigned to all UEs. We employ $4$-QAM modulation scheme and set the signal-
to-noise ratio (SNR) $25$ dB. We evaluate the performance depending on the
pilot-to-user ratio defined as $\tau/K$. Finally, we compare the performance
of our proposed detector with the CF-MRC [2], CF-MMSE [5], and CF-MMSE-SIC [3]
detectors.

###  IV-A Convergence and Complexity Analyses

We first analyse the convergence behaviour of the proposed CF-EP detector by
plotting the average number of iterations needed to satisfy the MM condition
in (15). It can be seen from Fig. 2 that the CF-EP detector needs less than
$6$ iterations to converge for various pilot-to-user ratios, $\tau/K$.

As tabulated in Table 1, the computational complexity of the CF-EP detector is
slightly higher than the MMSE detector depending on its number of iterations.
The CF-EP requires on average $2-6$ iterations to converge. Therefore, the
complexity order of the CF-EP detector remains the same as the CF-MMSE
detector. Note that the detector will be implemented at the CPU, which is
assumed to be equipped with a high performance computer. Therefore, a minor
increase in the computational complexity will not be a critical issue.

Detector | Complexity
---|---
CF-MRC | $\mathcal{O}(NK)$
CF-MMSE | $\mathcal{O}(N^{2}K)$
CF-MMSE-SIC | $\mathcal{O}(\sum_{k=0}^{K-1}(N-k)^{2}(K-k))$
CF-EP | $\mathcal{O}(N^{2}KT)$
Table I: Computational complexity comparison

###  IV-B Performance Analysis

(a) The BER performance (b) The sum SE performance

Figure 3: The BER and sum SE performances of CF-MRC [2], CF-MMSE [5], CF-MMSE-
SIC [3], and CF-EP detectors with $L=100,K=60$, $M=4$, SNR = $25$ dB

To analyse the proposed detector’s performance, we plot the BER and sum SE
over the random users locations versus the pilot-to-user ratio. Fig. 3a
demonstrates that the CF-EP detector outperforms the CF-MMSE-SIC and the CF-
MMSE detectors in terms of BER, for all pilot-to-user-ratio $\tau/K$. More
specifically, the CF-EP detector can achieve BER $10^{-4}$ with around $25\%$
lower pilot-to-user ratio than the CF-MMSE and CF-MMSE-SIC detectors. In Fig.
3b, the CF-EP detector demonstrates $26\%$ and $11\%$ sum SE improvements
compared to the CF-MMSE and CF-MMSE-SIC detectors, respectively. From these
facts, we conclude that the developed EP based detector, dealing with the
pilot contamination impact, achieves a significant performance gain compared
to the existing CF M-MIMO detectors.

##  V Conclusion

We propose a novel EP based detector for CF M-MIMO systems, namely the CF-EP
detector. Our simulation results show that the BER and sum SE performances of
the CF-EP detector are better than those of the state-of-the-art CF M-MIMO
detectors with comparable computational complexity.

## Acknowledgment

This research was supported by the research training program stipend from The
University of Sydney. The work of Branka Vucetic was supported in part by the
Australian Research Council Laureate Fellowship grant number FL160100032.

## References

  * [1] J. Zhang, E. Björnson, M. Matthaiou, D. W. K. Ng, H. Yang and D. J. Love, “Prospective multiple antenna technologies for beyond 5G,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1637–1660, Aug. 2020.
  * [2] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO versus small cells,” _IEEE Trans. Wireless Commun_ , vol. 16, no. 3, pp. 1834–1850, Mar. 2017.
  * [3] E. Björnson and L. Sanguinetti, “Making cell-free massive MIMO competitive with MMSE processing and centralized implementation,” _IEEE Trans. Wireless Commun_ , vol. 19, no. 1, pp. 77–90, Jan. 2020.
  * [4] Y. Feng, M. Wang, D. Wang, and X. You, “Low complexity iterative detection for a large-scale distributed MIMO prototyping system,” in _Proc. IEEE Int. Conf. on Commun._ , China, May. 2019, pp. 1–6.
  * [5] E. Nayebi, A. Ashikhmin, T. L. Marzetta and B. D. Rao, “Performance of cell-free massive MIMO systems with MMSE and LSFD receivers,” in _Proc. 50th Asilomar Conf. on Signals, Syst. and Comput._ , USA, Nov. 2016, pp. 203–207.
  * [6] R. Mosayebi, M. M. Mojahedian and A. Lozano, “Linear interference cancellation for the cell-free C-RAN uplink,” _IEEE Trans. Wireless Commun_ , Nov. 2020.
  * [7] T. Liu and Y. Y. Liu, “Modified fast recursive algorithm for efficient MMSE-SIC detection of the V-BLAST system,” _IEEE Trans. Wireless Commun_ , vol. 7, no. 10, pp. 3713–3717, Oct. 2008.
  * [8] J. Céspedes, P. M. Olmos, M. Sánchez-Fernández, and F. Pérez-Cruz, “Expectation propagation detection for high-order high-dimensional MIMO systems,” _IEEE Trans. Commun._ , vol. 62, no. 8, pp. 2840–2849, Aug. 2014.
  * [9] M. Bayati and A. Montanari, “The dynamics of message passing on dense graphs with applications to compressed sensing,” _IEEE Trans. Inf. Theory_ , vol. 57, no. 2, p. 764–785, Feb. 2011.
  * [10] S. Rangan, P. Schniter and A. K. Fletcher, “Vector approximate message passing,” in _IEEE Int. Symp. on Inform. Theory (ISIT)_ , Germany, June 2017, pp. 1588–1592.
  * [11] K. Takeuchi and C. Wen, “Rigorous dynamics of expectation-propagation signal detection via the conjugate gradient method,” in _IEEE 18th Int. Workshop on Sig. Process. Adv. in Wireless Commun. (SPAWC)_ , Japan, June 2017, pp. 1–5.
  * [12] A. Kosasih, W. Hardjawana, B. Vucetic, and C. K. Wen, “A linear Bayesian learning receiver scheme for massive MIMO systems,” in _Proc.IEEE Wireless Commun. and Netw. Conf._ , Korea, May. 2020, pp. 1–6.
  * [13] E. Björnson, J. Hoydis, and L. Sanguinetti, “Massive MIMO networks: Spectral, energy, and hardware efficiency,” _Found. Trends Mach. Learn._ , vol. 11, no. 3-4, pp. 154–655, 2017.
  * [14] J. Zhang, Y. He, Y. W. Li, C. K. Wen, and S. Jin, “Meta learning-based MIMO detectors: Design, simulation, and experimental test,” _IEEE Trans. Wireless Commun._ , Oct. 2020.
  * [15] C. E. Rasmussen and C. K. I. Williams, _Gaussian processes for machine learning_. MIT Press, 2006.
  * [16] T. P. Minka, “Expectation propagation for approximate bayesian inference,” in _Proc. of the 17th conf. on Uncertainty in artificial intell._ , 2001, pp. 362–369.
  * [17] E. Biglieri, J. Proakis, and S. Shamai, “Fading channels: Information theoretic and communications aspects,” _IEEE Trans. Inf. Theory_ , vol. 44, no. 6, pp. 2619–2691, Oct. 1998.
  * [18] 3rd Generation Partnership Project (3GPP), “Further advancements for E-UTRA physical layer aspects (Release 9),” 3GPP TS 36.814, Tech. Rep., Mar. 2017.