Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture. In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability. Experiment results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0 . Neural architecture search (NAS) emerged as an important branch of automatic machine learning (AutoML), and has been attracting increasing attentions from both academia and industry. The key methodology of NAS is to build a large space of network architectures, develop an efficient algorithm to explore the space, and discover the optimal structure under a combination of training data and constraints (e.g., network size and latency). Different from early approaches that often incur large computation overheads (Zoph & Le, 2017; Real et al., 2019) , recent oneshot approaches (Pham et al., 2018; have reduced the search costs by orders of magnitudes, which advances its applications to many real-world problems. In particular, DARTS converts the operation selection into weighting a fixed set of operations. This makes the entire framework differentiable to architecture hyper-parameters and thus the network search can be efficiently accomplished in an end-to-end fashion. Despite its sophisticated design, DARTS is still subject to a large yet redundant space of network architectures and thus suffers from heavy memory and computation overheads. This prevents the search process from using larger batch sizes for either speedup or higher stability. Prior work proposed to reduce the search space, which leads to an approximation that may sacrifice the optimality of the discovered architecture. In this paper, we present a simple yet effective approach named Partially-Connected DARTS (PC-DARTS) to reduce the burdens of memory and computation. The core idea is intuitive: instead of sending all channels into the block of operation selection, we randomly sample a subset of them in each step, while bypassing the rest directly in a shortcut. We assume the computation on this subset is a surrogate approximating that on all the channels. Besides the tremendous reduction in memory and computation costs, channel sampling brings another benefit -operation search is regularized and less likely to fall into local optima. However, PC-DARTS incurs a side effect, where the selection of channel connectivity would become unstable as different subsets of channels are sampled across iterations. Thus, we introduce edge normalization to stabilize the search for network connectivity by explicitly learning an extra set of edge-selection hyper-parameters. By sharing these hyper-parameters throughout the training process, the sought network architecture is insensitive to the sampled channels across iterations and thus is more stable. Benefiting from the partial connection strategy, we are able to greatly increase the batch size. Specifically, as only 1/K of channels are randomly sampled for an operation selection, it reduces the memory burden by almost K times. This allows us to use a K times larger batch size during search, which not only accelerates the network search but also stabilizes the process particularly for largescale datasets. Experiments on benchmark datasets demonstrate the effectiveness of PC-DARTS. Specifically, we achieve an error rate of 2.57% in less than 0.1 GPU-days (around 1.5 hours) on a single Tesla V100 GPU, surpassing the result of 2.76% reported by DARTS that required 1.0 GPUday. Furthermore, PC-DARTS allows a direct search on ImageNet (while DARTS failed due to low stability), and sets the state-of-the-art record with a top-1 error of 24.2% (under the mobile setting) in only 3.8 GPU-days (11.5 hours on eight Tesla V100 GPUs). First of all, there are two major contributions of our approach, namely, channel sampling and edge normalization. Channel sampling, as the key technique in this work, has not been studied in NAS for reducing computational overhead (other regularization methods like Dropout (Srivastava et al., 2014) and DropPath (Larsson et al., 2017 ) cannot achieve the same efficiency, in both time and memory, as channel sampling). It accelerates and regularizes search and, with the help of edge normalization, improves search stability. Note that both search speed and stability are very important for a search algorithm. Combining channel sampling and edge normalization, we obtain the best accuracy on ImageNet (based on the DARTS search space), and the direct search cost on ImageNet (3.8 GPU-days) is the lowest known. Moreover, these two components are easily transplanted to other search algorithms to improve search accuracy and speed, e.g., edge normalization boosts the accuracy and speed of the original DARTS methods. Other researchers also tried to alleviate the large memory consumption of DARTS. Among prior efforts, ProxylessNAS (Cai et al., 2019) binarized the multinomial distribution α o i,j and samples two paths at each time, which significantly reduced memory cost and enabled direct search on ImageNet. PARSEC (Casale et al., 2019) also proposed a sampling-based optimization method to learn a probability distribution. Our solution, by preserving all operations for architecture search, achieves a higher accuracy in particular on challenging datasets like ImageNet (+0.7% over ProxylessNAS and +1.8% over PARSEC). Another practical method towards memory efficiency is Progressive-DARTS , which eliminated a subset of operators in order to provide sufficient memory for deeper architecture search. In comparison, our approach preserves all operators and instead performs sub-sampling on the channel dimension. This strategy works better in particular on large-scale datasets like ImageNet. In this paper, we proposed a simple and effective approach named partially-connected differentiable architecture search (PC-DARTS). The core idea is to randomly sample a proportion of channels for operation search, so that the framework is more memory efficient and, consequently, a larger batch size can be used for higher stability. Additional contribution to search stability is made by edge normalization, a light-weighted module that requires merely no extra computation. Our approach can accomplish a complete search within 0.1 GPU-days on CIFAR10, or 3.8 GPU-days on ImageNet, and report state-of-the-art classification accuracy in particular on ImageNet. This research delivers two important messages that are important for future research. First, differentiable architecture search seems to suffer even more significant instability compared to conventional neural network training, and so it can largely benefit from both (i) regularization and (ii) a larger batch size. This work shows an efficient way to incorporate these two factors in a single pipeline, yet we believe there exist other (possibly more essential) solutions for this purpose. Second, going one step further, our work reveals the redundancy of super-network optimization in NAS, and experiments reveal a gap between improving super-network optimization and finding a better architecture, and regularization plays an efficient role in shrinking the gap. We believe these insights can inspire researchers in this field, and we will also follow this path towards designing stabilized yet efficient algorithms for differentiable architecture search.
Allowing partial channel connection in super-networks to regularize and accelerate differentiable architecture search An extension of the neural architecture search method DARTS that addresses its shortcoming of immense memory cost by using a random subset of channels and a method to normalize edges. This paper proposes to improve DARTS in terms of training efficiency, from large memory and computing overheads, and proposes a partially-connected DARTS with partial channel connection and edge normalization.