A search framework for finding effective architectural building blocks for deep convolutional neural networks is disclosed. The search framework described herein utilizes a building block which incorporates branch and skip connections. At least some operations of the architecture of the building block are undefined and treated as hyperparameters which can be automatically selected and optimized for a particular task. The search framework uses random search over the reduced search space to generate a building block and repeats the building block multiple times to create a deep convolutional neural network.
:NASNet considers learning an architectural building block rather than learning the full architecture. The learned building block is then repeated many times to create the deep architecture. An rnn is used to generate the descriptions of the building block in a manner similar to @cite_24 . For cnn models, two types of building blocks are learned: and , where these blocks return same size feature maps and reduced size feature maps respectively. The controller rnn is trained using proximal policy optimization to generate better architectural building blocks over time. This approach also generates models that are transferable with the best building blocks for one learning task also showing good performance over other learning tasks as well. However the rnn itself still lacks transferability and needs to be retrained for each new problem. Also, because the blocks are learned from scratch, this approach requires many more trials to achieve state-of-the-art performance.
@cite_24
2553303224
Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.