(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# GP-NAS-ensemble: a model for NAS Performance Prediction

Kunlong Chen1 Liu Yang2 Yitian Chen3 Kunjin Chen4 Yidan Xu1 Lujun Li5
1Meituan 2Tencent 3Bigo Technology 4Alibaba Group 5Chinese Academy of Sciences
1{chenkunlong<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

It is of great significance to estimate the performance of a given model
architecture without training in the application of Neural Architecture Search
(NAS) as it may take a lot of time to evaluate the performance of an
architecture. In this paper, a novel NAS framework called GP-NAS-ensemble is
proposed to predict the performance of a neural network architecture with a
small training dataset. We make several improvements on the GP-NAS model to
make it share the advantage of ensemble learning methods. Our method ranks
second in the CVPR2022 second lightweight NAS challenge performance prediction
track.

##  1 Introduction

With the development of Neural Architecture Search (NAS) techniques, it
becomes popular to design deep neural network architectures automatically [1].
It is a critical part to design a performance predictor to analyze the
relationship between model architecture and its accuracy evaluated on certain
tasks for NAS algorithms, as training a large net can be fairly expensive
[50].

GP-NAS-ensemble model is proposed as a novel predictor in this paper. Based on
GP-NAS model [23], we make several improvements to make it more accurate and
robust. The validity of our method is verified in the Performance Prediction
Track of CVPR2022 Second lightweight NAS challenge.

###  1.1 Literature Review

General NAS. In a wide range of computer vision tasks[46, 12, 3, 29, 48, 14,
19, 44, 22, 20, 21, 18, 8], manually constructed neural networks have had
great success. Artificial designs are usually thought to be suboptimal. Both
academia and industry have recently been more interested in neural
architecture search (NAS). Early efforts made use of reinforcement learning
[37, 51, 52, 53] and evolutionary algorithms[33, 34, 36, 45, 26] and
discovered several high-cost and high-performance designs. Later work aims to
lower the cost of searching while increasing performance, which can be divided
into three categories: one-shot NAS [13, 49, 9, 6], gradient-based
approaches[25, 5, 40], and predictor NAS, which differ in the network
architecture modeling process. One-Shot NAS [6] firstly trains an over-
parameterized supernet, then searches a discrete search space that includes
numerous candidate models. The sample strategy is important during the
training stage, since it determines how to train an effective supernet for
performance estimation. Gradient-based methods incorporate the architecture
parameters for each operator and use backpropagation to jointly optimize them
and the weights of the network.

Predictor NAS. Predictor NAS approaches attempt to accurately and effectively
forecast the performance of a particular neural network. These procedures
learns the accuracy predictor by sampling pairs of architectures and their
accuracies. Some works [4, 16, 7, 15] extend this line of thought by training
a predictor to extrapolate the NAS learning curves. The regression problem
[42] or ranking [30, 47] can be used to describe the goal of fitting the
predictor [41, 28, 11, 27]. Moreover, feature representation and sampling
methods are crucial for search performance. [28] uses acyclic graphs in a
continuous space of potential embeddings along with performance predictors.
[39] improves the two-stage NAS with Pareto-aware sampling strategies. [35]
uses Bayesian regression as a proxy model to select candidates, and [43]
replaces a strong predictor with a set of weaker predictors.

Multi-task NAS. Multi-task learning refers to different tasks sharing part of
the network backbone or weights. These tasks are often able to learn from each
other to achieve better performance and training efficiency. Recent neural
network architecture search methods and benchmarks [10, 38, 15] for multi-task
and cross-task have attracted a lot of attention from the community. Despite
being underappreciated in comparison to single-task NAS, there are still
several excellent algorithms. [31] uses continuous learning to find a single
cell structure that can generalize well to unknown tasks via multi-task
architecture search based on the weight sharing technique. [24] used gradient-
based NAS to find the best cell structure for a variety of autonomous driving
tasks. [17] proposed to construct graphs from datasets in a meta-learning
approach to make the methods generalize well across numerous tasks.

##  2 Proposed Method

In this paper, we introduce the GP-NAS-ensemble model [23], which ranked 2nd
in the CVPR 2022 NAS competition: performance estimation track. Based on the
GP-NAS model, we aim at establishing a model with better performance using
ensemble learning technique.

###  2.1 GP-NAS

GP-NAS is a powerful method to predict the performance of a neural network
given its architecture, especially when the size of training data is small
[23]. To be more specific, the GP-NAS model uses the Gaussian process
regression model to predict the accuracy of a neural network model under the
assumption that the joint distribution between the training observations
$\mathbf{y}$ and the test function values
$f(\boldsymbol{X}_{*})=\boldsymbol{f}_{*}$ is [32]

$p\left(\boldsymbol{y}_{,}\boldsymbol{f}_{*}\mid\boldsymbol{X},\boldsymbol{X}_{*}\right)=\mathcal{N}\left(\left[\begin{array}[]{c}m(\boldsymbol{X})\\\
m\left(\boldsymbol{X}_{*}\right)\end{array}\right],\left[\begin{array}[]{cc}\mathbf{K}+\sigma_{n}^{2}\mathbf{I}&k\left(\boldsymbol{X}^{\prime},\boldsymbol{X}_{*}\right)\\\
k\left(\boldsymbol{X}_{*},\boldsymbol{X}\right)&k\left(\boldsymbol{X}_{*},\boldsymbol{X}_{*}\right).\end{array}\right]\right)$
(1)

We then have

$\begin{split}p\left(\boldsymbol{f}_{*}\mid\boldsymbol{X},\boldsymbol{y},\boldsymbol{X}_{*}\right)&=\mathcal{N}\left(\mathbb{E}\left[\boldsymbol{f}_{*}\mid\boldsymbol{X},\boldsymbol{y},\boldsymbol{X}_{*}\right],\mathbb{V}\left[\boldsymbol{f}_{*}\mid\boldsymbol{X},\boldsymbol{y},\boldsymbol{X}_{*}\right]\right),\\\
\mathbb{E}\left[\boldsymbol{f}_{*}\mid\boldsymbol{X},\boldsymbol{y},\boldsymbol{X}_{*}\right]&=m_{\text{post
}}\left(\boldsymbol{X}_{*}\right)=\underbrace{m\left(\boldsymbol{X}_{*}\right)}_{\text{prior
mean
}}+\underbrace{k\left(\boldsymbol{X}_{*},\boldsymbol{X}\right)\left(\boldsymbol{K}+\sigma_{n}^{2}\boldsymbol{I}\right)^{-1}}_{\text{“Kalman
gain” }}\underbrace{(\boldsymbol{y}-m(\boldsymbol{X}))}_{\text{error
}},\end{split}$ (2)

where $\mathbf{K}$ is the Gram matrix, which contains the kernel functions
evaluating on all pairs of data points, $\sigma_{n}^{2}$ is the noise variance
of observations.

Simply put, the Gaussian process model uses the prior mean as an initial guess
for the target value. The guess is then corrected by a mechanism similar to
Kalman filter. It is therefore important to note that, the cores of the GP-NAS
method are two parts: (1) the estimation of the prior mean; (2) the specific
formula of the kernel function. In the original implementation of the
algorithm, the linear regression method is used to estimate the prior mean
with a variant of the radial basis function (RBF) kernel:

$\begin{split}m(\mathbf{x})&=\mathbf{w}\cdot\mathbf{x},\\\
k_{rbf}^{s}(\mathbf{x}_{1},\mathbf{x}_{2})&=\exp[-\sqrt{||(\mathbf{x}_{1}-\mathbf{x}_{2})||}/l],\end{split}$
(3)

where $\mathbf{w}$ is the coefficient vector of the linear regression model
and $l$ is the length scale of the kernel.

###  2.2 Feature Engineering

In the original dataset, an ordinal encoder is used to represent each
architecture. For example, the search space of features such as depth of
network, number of heads of each layer, and MLP ratio of each layer are
$\\{10,11,12\\}$. They are encoded as $\\{1,2,3\\}$, respectively, in the
feature space.

One-hot encoding is a sparse way of representing data in a binary string in
which only a single bit can be 1, while all others are 0 [2]. This method is
considered as a popular way to deal with catogorical features in the machine
learning community. However, the disadvantage of one-hot encoding is that it
does not contain the information of the ‘similarity” of two data points. For
instance, the distance between “3” and “1” should be larger than the distance
between “2” and “1”. In order to solve this problem, we also use the two-hot
encoding method to represent the similarity in the feature space. An example
of applying one-hot encoding and two-hot encoding methods is shown in Table 1.

original feature | one-hot encoding | two-hot encoding
---|---|---
0 | $[0,0,0,0]$ | $[0,0,0,0]$
1 | $[1,0,0,0]$ | $[1,1,0,0]$
2 | $[0,1,0,0]$ | $[0,1,1,0]$
3 | $[0,0,1,0]$ | $[0,0,1,1]$
Table 1: The feature engineering methods used in our framework. Figure 1: An illustration of the proposed GP-NAS-ensemble model. Task | kernel length | weighted kernel ratio | Label transformation method | set of base learner
---|---|---|---|---
task 0 | 22 | (0.18, 0.82) | Normal distribution | GP-NAS, KNN
task 1 | 28 | (0.62, 0.38) | Left-skewed Normal distribution | GP-NAS, SVR
task 2 | 24 | (0.02, 0.98) | Left-skewed Normal distribution | GP-NAS, SVR
task 3 | 25 | (0.6, 0.4) | Normal distribution | GP-NAS, SVR
task 4 | 22 | (0.7, 0.3) | Left-skewed Normal distribution | GP-NAS, SVR
task 5 | 22 | (0.3, 0.7) | Normal distribution | GP-NAS, SVR
task 6 | 22 | - | Normal distribution | GP-NAS
task 7 | 22 | (0.3, 0.7) | Normal distribution | GP-NAS, SVR
Table 2: The configurations for each task. Modifications | score (public)
---|---
GP-NAS | $~{}0.668$
\+ feature engineering | $~{}0.787$
\+ label transformation | $0.796$
\+ ensemble learning | $0.798$
\+ weighted ensemble kernel | $0.800$
Table 3: The model performance on the public leaderboard.

###  2.3 Label transformation

The dataset does not provide us with the prediction accuracy of each
architecture on test dataset. Instead, we only have access to the relative
rank of each data point. We believe that it is more natural to predict the
accuracy of each model architecture instead of predicting ranks through
several trials. Since we do not have enough domain knowledge on this field, we
need to guess the probability distribution of model performances and then
assign each model with a score by sampling from the proposed distribution.
There are two ways to guess the distribution of the accuracy: (1) they follow
a normal distribution, (2) they follow a left-skewed bell-shaped distribution.
We decide the distribution by conducting trials on the public leaderboard.

###  2.4 Weighted ensemble kernel function

The idea behind the weighted kernel function is that in each task, we should
focus on different parts of the feature set. The weighted kernel function is

$k_{w}(\mathbf{x}_{1},\mathbf{x}_{2})=\exp[-\sqrt{(\mathbf{x}_{1}-\mathbf{x}_{2})^{T}I_{w}(\mathbf{x}_{1}-\mathbf{x}_{2})}/l],$
(4)

where $I_{w}$ is a diagonal matrix. In order to estimate $I_{w}$ for each
task, we use the Bayesian optimization method to maximize the Kendall rank
correlation coefficient on the training dataset. Considering the fact that
only a small amount of data is available, it may be prone to over-fitting by
applying Bayesian optimization method directly. We know that if $k_{1},k_{2}$
are valid kernels, $k_{1}+k_{2}$ is still a valid kernel [32]. We thus
proposed a new kernel $k_{w}^{e}$, which is the weighted sum of $k_{w}$ and
$k_{rbf}^{s}$:

$k_{w}^{e}(\mathbf{x}_{1},\mathbf{x}_{2})=\beta_{1}k_{rbf}^{s}(\mathbf{x}_{1},\mathbf{x}_{2})+\beta_{2}k_{w}(\mathbf{x}_{1},\mathbf{x}_{2}),$
(5)

where the values of $\beta_{1}$ and $\beta_{2}$ are selected by their
performances on the public leaderboard.

###  2.5 GP-NAS-ensemble model

In this section, we briefly introduce the structure of our proposed GP-NAS-
ensemble model. It adopts the modifications described in previous sections.
The basic scheme of the GP-NAS-ensemble model is shown in Figure 1, which
contains the following steps:

  1. 1.

Weighted ensemble kernel computing. We use the Bayesian optimization method to
estimate the most suitable weighted ensemble kernel function for each task.

  2. 2.

Base model establishment. Two GP-NAS models are built as base models. The
difference lies in that we feed the data with one-hot encoding to the first
model and we feed the data with two-hot encoding to the second one. In
addition, several classical supervised learning methods are also considered as
parts of our base-model set since we need to enhance the diversity between
base models.

  3. 3.

Base model training. We train each base model separately on the training
dataset.

  4. 4.

Ensemble model establishment. A GP-NAS-ensemble model can be built on top of
the base models mentioned above. To be more specific, we predict the prior
mean of a given test data by averaging the output of each base model. The
estimation of the posterior mean is the same as the basic GP-NAS model.

##  3 Experiments

The configurations of our final submission except for task 6 are shown in
Table 2. Specifically, we observe that the above-mentioned weighted ensemble
kernel doesn’t not increase the score of task 6. Therefore, we use nine-hot
encoding method for the feature engineering step of task 6. The configurations
are tuned based on scores on the public leaderboard.

In Table 3, we show the results of an ablation study which compares the
proposed model with the original GP-NAS model. The GP-NAS model can achieve
about $0.668$ on public leaderboard. If we transform input features with one-
hot encoding or two-hot encoding, the score is increased to about $0.787$.
After label transformation with normal distribution, we obtain a score about
$0.796$. When all modifications are applied, we get a final score close to
$0.800$.

##  4 Conclusion

Unlike most other competitors of this competition who use deep-learning method
or other classical supervised machine learning to achieve a high score, we
fully explored the potential of GP-NAS model with only small modifications on
the model architecture and the feature engineering pipeline. The score of the
proposed method on the public leaderboard increased from $0.668$ to about
$0.800$.

## References

  * [1] https://aistudio.baidu.com/aistudio/competition/detail/150/0/introduction.
  * [2] https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f.
  * [3] Pattathal V. Arun, Ittai Herrmann, Krishna M. Budhiraju, and Arnon Karnieli.  Convolutional network architectures for super-resolution/sub-pixel mapping of drone-derived images.  Pattern Recognition, 88:431–446, 2019.
  * [4] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik.  Accelerating neural architecture search using performance prediction.  In ICLR, 2018.
  * [5] Jianlong Chang, Yiwen Guo, MENG Gaofeng, Zhouchen Lin, Shiming XIANG, Chunhong Pan, et al.  Data: Differentiable architecture approximation with distribution guided sampling.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [6] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li.  FairNAS: Rethinking evaluation fairness of weight sharing neural architecture search.  arXiv preprint arXiv:1907.01845, 2019.
  * [7] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter.  Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves.  In IJCAI, 2015.
  * [8] Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan.  Prior-guided one-shot neural architecture search.  arXiv preprint arXiv:2206.13329, 2022.
  * [9] Xuanyi Dong and Yi Yang.  One-shot neural architecture search via self-evaluated template network.  In ICCV, 2019.
  * [10] Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li.  TransNAS-Bench-101: Improving transferability and generalizability of cross-task neural architecture search.  In CVPR, 2021.
  * [11] Lukasz Dudziak, Thomas C. P. Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane.  BRP-NAS: prediction-based NAS using gcns.  In Advances in Neural Information Processing Systems 33, 2020.
  * [12] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, and Tsuhan Chen.  Recent advances in convolutional neural networks.  Pattern Recognition, 77:354–377, 2018.
  * [13] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun.  Single path one-shot neural architecture search with uniform sampling.  In ECCV, 2020.
  * [14] Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu.  Improving one-shot nas with shrinking-and-expanding supernet.  Pattern Recognition, 2021.
  * [15] Minbin Huang, Zhijian Huang, Changlin Li, Xin Chen, Hang Xu, Zhenguo Li, and Xiaodan Liang.  Arch-Graph: Acyclic architecture relation predictor for task-transferable neural architecture search.  In CVPR, 2022.
  * [16] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter.  Learning curve prediction with bayesian neural networks.  In ICLR, 2017.
  * [17] Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang.  Rapid neural architecture search by learning to generate graphs from datasets.  In ICLR, 2021.
  * [18] Lujun Li.  Self-regulated feature learning via teacher-free feature distillation.  In ECCV, 2022.
  * [19] Lujun Li and Zhe Jin.  Shadow knowledge distillation: Bridging offline and online knowledge transfer.  In NeuIPS, 2022.
  * [20] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin.  Boosting online feature transfer via separable feature fusion.  In IJCNN, 2022.
  * [21] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin.  Teacher-free distillation via regularizing intermediate representation.  In IJCNN, 2022.
  * [22] Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He.  Explicit connection distillation.  2020\.
  * [23] Zhihang Li, Teng Xi, Jiankang Deng, Gang Zhang, Shengzhao Wen, and Ran He.  GP-NAS: Gaussian process based neural architecture search.  In CVPR, 2020.
  * [24] Hao Liu, Dong Li, Jinzhang Peng, Qingjie Zhao, Lu Tian, and Yi Shan.  MTNAS: search multi-task networks for autonomous driving.  In ACCV, 2020.
  * [25] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  In ICLR, 2019.
  * [26] Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh D. Dhebar, Kalyanmoy Deb, Erik D. Goodman, and Wolfgang Banzhaf.  NSGA-NET: A multi-objective genetic algorithm for neural architecture search.  CoRR, abs/1810.03522, 2018.
  * [27] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture search with GBDT.  CoRR, abs/2007.04785, 2020.
  * [28] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization.  In Advances in Neural Information Processing Systems 31, 2018.
  * [29] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In ECCV, 2018.
  * [30] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang.  A generic graph-based neural architecture encoding scheme for predictor-based NAS.  In ECCV, 2020.
  * [31] Ramakanth Pasunuru and Mohit Bansal.  Continual and multi-task architecture search.  In ACL, 2019.
  * [32] Carl Edward Rasmussen.  Gaussian processes in machine learning.  In Summer school on machine learning, pages 63–71. Springer, 2003\.
  * [33] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le.  Regularized evolution for image classifier architecture search.  In AAAI, 2019.
  * [34] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In ICML, 2017.
  * [35] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang.  Bridging the gap between sample-based and one-shot neural architecture search with BONAS.  In Advances in Neural Information Processing Systems 33, 2020.
  * [36] Masanori Suganuma, Mete Ozay, and Takayuki Okatani.  Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search.  In ICML, 2018.
  * [37] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In CVPR, 2019.
  * [38] Renbo Tu, Mikhail Khodak, Nicholas Roberts, and Ameet Talwalkar.  NAS-bench-360: Benchmarking diverse tasks for neural architecture search.  CoRR, abs/2110.05668, 2021.
  * [39] Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra.  AttentiveNAS: Improving neural architecture search via attentive sampling.  In CVPR, 2021.
  * [40] Naiyan Wang, Shiming XIANG, Chunhong Pan, et al.  You only search once: Single shot neural architecture search via direct sparse optimization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [41] Chen Wei, Chuang Niu, Yiping Tang, and Jimin Liang.  NPENAS: neural predictor guided evolution for neural architecture search.  CoRR, abs/2003.12857, 2020.
  * [42] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans.  Neural predictor for neural architecture search.  In ECCV, 2020.
  * [43] Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen, and Lu Yuan.  Stronger NAS with weaker predictors.  arXiv preprint arXiv:2102.10490, 2021.
  * [44] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao.  Norm: Knowledge distillation via n-to-one representation matching.  2022\.
  * [45] Lingxi Xie and Alan L. Yuille.  Genetic CNN.  In ICCV, 2017.
  * [46] Ting-Bing Xu, Peipei Yang, Xu-Yao Zhang, and Cheng-Lin Liu.  Lightweightnet: Toward fast and lightweight convolutional neural networks via architecture distillation.  Pattern Recognition, 88:272–284, 2019.
  * [47] Yixing Xu, Yunhe Wang, Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu.  Renas: Relativistic evaluation of neural architecture search.  In CVPR, 2021.
  * [48] Mohamed Yousef, Khaled F Hussain, and Usama S Mohammed.  Accurate, data-efficient, unconstrained text recognition with convolutional neural networks.  Pattern Recognition, 108:107482, 2020.
  * [49] Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Chuan Zhou, Zongyuan Ge, and Steven W Su.  One-shot neural architecture search: Maximising diversity to overcome catastrophic forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [50] Ruyi Zhang, Ziwei Yang, Zhi Yang, Xubo Yang, Lei Wang, and Zheyang Li.  Cascade bagging for accuracy prediction with few training samples.  arXiv preprint arXiv:2108.05613, 2021.
  * [51] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu.  Practical block-wise neural network architecture generation.  In CVPR, 2018.
  * [52] Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning.  In ICLR, 2017.
  * [53] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition.  In CVPR, 2018.