The process of designing neural architectures requires expert knowledge and extensive trial and error.
 While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.
 We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.
 The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.   Using two different candidate generation techniques, random search with a ranking function and reinforcement learning, 
we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.
 The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed. Developing novel neural network architectures is at the core of many recent AI advances BID28 BID14 BID35 . The process of architecture search and engineering is slow, costly, and laborious. Human experts, guided by intuition, explore an extensive space of potential architectures where even minor modifications can produce unexpected results. Ideally, an automated architecture search algorithm would find the optimal model architecture for a given task. Many explorations into the automation of machine learning have been made, including the optimization of hyperparameters BID3 BID24 and various methods of producing novel model architectures BID27 BID1 Zoph and Le, 2017) . For architecture search, ensuring these automated methods are able to produce results similar to humans usually requires traversing an impractically large search space, assuming high quality architectures exist in the search space at all. The choice of underlying operators composing an architecture is further typically constrained to a standard set across architectures even though recent work has found promising results in the use of non-standard operators BID31 .We propose a meta-learning strategy for flexible automated architecture search of recurrent neural networks (RNNs) which explicitly includes novel operators in the search. It consists of three stages, outlined in Figure 1 , for which we instantiate two versions.1. A candidate architecture generation function produces potential RNN architectures using a highly flexible DSL. The DSL enforces no constraints on the size or complexity of the generated tree and can be incrementally constructed using either a random policy or with an RL agent. 2. A ranking function processes each candidate architecture's DSL via a recursive neural network, predicting the architecture's performance. By unrolling the RNN representation, the ranking function can also model the interactions of a candidate architecture's hidden state over time. Figure 1: A generator produces candidate architectures by iteratively sampling the next node (either randomly or using an RL agent trained with REINFORCE). Full architectures are processed by a ranking function and the most promising candidates are evaluated. The results from running the model against a baseline experiment are then used to improve the generator and the ranking function.3. An evaluator, which takes the most promising candidate architectures, compiles their DSLs to executable code and trains each model on a specified task. The results of these evaluations form architecture-performance pairs that are then used to train the ranking function and RL generator. In this section, we describe a domain specific language (DSL) used to define recurrent neural network architectures. This DSL sets out the search space that our candidate generator can traverse during architecture search. In comparison to Zoph and Le (2017) , which only produced a binary tree with matrix multiplications at the leaves, our DSL allows a broader modeling search space to be explored. When defining the search space, we want to allow for standard RNN architectures such as the Gated Recurrent Unit (GRU) BID7 or Long Short Term Memory (LSTM) BID15 to be defined in both a human and machine readable manner. The core operators for the DSL are 4 unary operators, 2 binary operators, and a single ternary operator: DISPLAYFORM0 MM represents a single linear layer with bias, i.e. MM (x) := W x + b. Similarly, we define: DISPLAYFORM1 The operator Mult represents element-wise multiplication: Mult(x, y) = x • y. The Gate3 operator performs a weighted summation between two inputs, defined by DISPLAYFORM2 These operators are applied to source nodes from the set [x t , x t−1 , h t−1 , c t−1 ], where x t and x t−1 are the input vectors for the current and previous timestep, h t−1 is the output of the RNN for the previous timestep, and c t−1 is optional long term memory. The Gate3 operator is required as some architectures, such as the GRU, re-use the output of a single Sigmoid for the purposes of gating. While allowing all possible node re-use is out of scope for this DSL, the Gate3 ternary operator allows for this frequent use case. Using this DSL, standard RNN cell architectures such as the tanh RNN can be defined: tanh(Add (MM (x t ), MM (h t−1 ))). To illustrate a more complex example that includes Gate3 , the GRU is defined in full in Appendix A. With the operators defined above it is not possible to refer to and re-use an arbitrary node. The best performing RNN architectures however generally use not only a hidden state h t but also an additional hidden state c t for long term memory. The value of c t is extracted from an internal node computed while producing h t .The DSL above can be extended to support the use of c t by numbering the nodes and then specifying which node to extract c t from (i.e. c t = Node 5 ). We append the node number to the end of the DSL definition after a delimiter. As an example, the nodes in bold are used to produce c t , with the number appended at the end indicating the node's number. Nodes are numbered top to bottom (h t will be largest), left to right. DISPLAYFORM0 While the domain specific language is not entirely generic, it is flexible enough to capture most standard RNN architectures. This includes but is not limited to the GRU, LSTM, Minimal Gate Unit (MGU) BID34 , Quasi-Recurrent Neural Network (QRNN) BID4 , Neural Architecture Search Cell (NASCell) (Zoph and Le, 2017) , and simple RNNs. While many standard and non-standard RNN architectures can be defined using the core DSL, the promise of automated architecture search is in designing radically novel architectures. Such architectures should be formed not just by removing human bias from the search process but by including operators that have not been sufficiently explored. For our expanded DSL, we include: DISPLAYFORM0 These extensions add inverses of currently used operators (Sub(a, b) = a − b instead of addition, Div (a, b) = a b instead of multiplication), trigonometric curves (Sin and Cos are sine and cosine activations respectively, PosEnc introduces a variable that is the result of applying positional encoding BID31 according to the current timestep), and optimizations (LayerNorm applies layer normalization BID0 to the input while SeLU is the activation function defined in BID18 ). For a given architecture definition, we can compile the DSL to code by traversing the tree from the source nodes towards the final node h t . We produce two sets of source code -one for initialization required by a node, such as defining a set of weights for matrix multiplication, and one for the forward call during runtime. For details regarding speed optimizations, refer to Appendix A2. The candidate architecture generator is responsible for producing candidate architectures that are then later filtered and evaluated. Architectures are grown beginning at the output h t and ordered to prevent multiple representations for equivalent architectures:Growing architectures from h t up Beginning from the output node h t , operators are selected to be added to the computation graph, depicted in Figure 2 . Whenever an operator has one or more children to be filled, the children are filled in order from left to right. If we wish to place a limit on the height (distance from h t ) of the tree, we can force the next child to be one of the source nodes when it would otherwise exceed the maximum height. Architectures in the DSL are constructed incrementally a node at a time starting from the output h t . The simplest agent is a random one which selects the next node from the set of operators without internalizing any knowledge about the architecture or optima in the search space. Allowing an intelligent agent to construct architectures would be preferable as the agent can learn to focus on promising directions in the space of possible architectures. For an agent to make intelligent decisions regarding which node to select next, it must have a representation of the current state of the architecture and a working memory to direct its actions. We propose achieving this with two components:1. a tree encoder that represents the current state of the (partial) architecture. 2. an RNN which is fed the current tree state and samples the next node. The tree encoder is an LSTM applied recursively to a node token and all its children with weights shared, but the state reset between nodes. The RNN is applied on top of the encoded partial architecture and predicts action scores for each operation. We sample with a multinomial and encourage exploration with an epsilon-greedy strategy. Both components of the model are trained jointly using the REINFORCE algorithm BID32 .As a partial architecture may contain two or more empty nodes, such as h t = Gate3(∅, ∅, σ(∅)), we introduce a target token, T , which indicates which node is to next be selected. Thus, in h t = Gate(T, ∅, σ(∅)), the tree encoder understands that the first argument is the slot to be filled. Even with an intelligent generator, understanding the likely performance of an architecture is difficult, especially the interaction of hidden states such as h t−1 and c t−1 between timesteps. We propose to approximate the full training of a candidate architecture by training a ranking network through regression on architecture-performance pairs. This ranking function can be specifically constructed to allow a richer representation of the transitions between c t−1 and c t .As the ranking function uses architecture-performance samples as training data, human experts can also inject previous best known architectures into the training dataset. This is not possible for on-policy reinforcement learning and when done using off-policy reinforcement learning additional care and complexity are required for it to be effective .Given an architecture-performance pair, the ranking function constructs a recursive neural network that reflects the nodes in a candidate RNN architecture one-to-one. Sources nodes are represented by a learned vector and operators are represented by a learned function. The final vector output then passes through a linear activation and attempts to minimize the difference between the predicted and real performance. The source nodes (x t , x t−1 , h t−1 , and c t−1 ) are represented by learned vector representations. For the operators in the tree, we use TreeLSTM nodes BID29 . All Unrolling the graph for accurately representing h t−1 and c t−1 : A strong assumption made above is that the vector representation of the source nodes can accurately represent the contents of the source nodes across a variety of architectures. This may hold true for x t and x t−1 but is not true for h t−1 or c t−1 . The value of h t and c t are defined by the operations within the given architecture itself. To remedy this assumption, we can unroll the architecture for a single timestep, replacing h t−1 and c t−1 with their relevant graph and subgraph. This would allow the representation of h t−1 to understand which source nodes it had access to and which operations were applied to produce h t−1 .While unrolling is useful for improving the representation of h t−1 , it is essential for allowing an accurate representation of c t−1 . This is as many small variations of c t−1 are possible -such as selecting a subgraph before or after an activation -that may result in substantially different architecture performance. We evaluated our architecture generation on two experiments: language modeling (LM) and machine translation (MT). Due to the computational requirements of the experiments, we limited each experiment to one combination of generator components. For language modeling, we explore the core DSL using randomly constructed architectures (random search) directed by a learned ranking function. For machine translation, we use the extended DSL and construct candidate architectures incrementally using the RL generator without a ranking function. For evaluating architectures found during architecture search, we use the WikiText-2 dataset BID21 . When evaluating a proposed novel RNN cell c, we construct a two layer c-RNN with a 200 unit hidden size. Aggressive gradient clipping is performed to ensure that architectures such as the ReLU RNN would be able to train without exploding gradients. The weights of the ranking network were trained by regression on architecture-perplexity pairs using the Adam optimizer and mean squared error (MSE). Further hyperparameters and training details are listed in Appendix B1.Explicit restrictions on generated architectures During the candidate generation phase, we filter the generated architectures based upon specific restrictions. These include structural restrictions and restrictions aimed at effectively reducing the search space by removing likely invalid architectures. For Gate3 operations, we force the input to the forget gate to be the result of a sigmoid activation. We also require the cell to use the current timestep x t and the previous timestep's output h t−1 to satisfy the requirements of an RNN. Candidate architectures were limited to 21 nodes, the same number of nodes as used in a GRU, and the maximum allowed distance (height) from h t was 8 steps. We also prevent the stacking of two identical operations. While this may be an aggressive filter it successfully removes many problematic architectures. These problematic architectures include when two sigmoid activations, two ReLU activations, or two matrix multiplications are used in succession -the first of which is unlikely to be useful, the second of which is a null operator on the second activation, and the third of which can be mathematically rewritten as a single matrix multiplication. If a given candidate architecture definition contained c t−1 , the architecture was queried for valid subgraphs from which c t could be generated. The subgraphs must contain c t−1 such that c t is recurrent and must contain three or more nodes to prevent trivial recurrent connections. A new candidate architecture is then generated for each valid c t subgraph. Random architecture search directed by a learned ranking function Up to 50,000 candidate architecture DSL definitions are produced by a random architecture generator at the beginning of each search step. This full set of candidate architectures are then simulated by the ranking network and an estimated perplexity assigned to each. Given the relative simplicity and small training dataset, the ranking function was retrained on the previous full training results before being used to estimate the next batch of candidate architectures. Up to 32 architectures were then selected for full training. 28 of these were selected from the candidate architectures with the best perplexity while the last 4 were selected via weighted sampling without replacement, prioritizing architectures with better estimated perplexities.c t architectures were introduced part way through the architecture search after 750 valid h t architectures had been evaluated with h t architectures being used to bootstrap the c t architecture vector representations. FIG0 provides a visualization of the architecture search over time, showing valid h t and c t architectures. Analyzing the BC3 cell After evaluating the top 10 cells using a larger model on WikiText-2, the top performing cell BC3 (named after the identifying hash, bc3dc7a. . .) was an unexpected layering of two Gate3 operators, DISPLAYFORM0 where • is an element-wise multiplication and all weight matrices W, U, V, X ∈ R H×H .Equations 1 to 3 produce the first Gate3 while equations 4 and 5 produce the second Gate3 . The output of the first Gate3 becomes the value for c t after passing through a tanh activation. While only the core DSL was used, BC3 still breaks with many human intuitions regarding RNN architectures. While the formulation of the gates f and o are standard in many RNN architectures, the rest of the architecture is less intuitive. The Gate3 that produces c t (equation 3) is mixing between a matrix multiplication of the current input x t and a complex interaction between c t−1 and x t (equation 2). In BC3, c t−1 passes through multiple matrix multiplications, a gate, and a tanh activation before becoming c t . This is non-conventional as most RNN architectures allow c t−1 to become c t directly, usually through a gating operation. The architecture also does not feature a masking output gate like the LSTM, with outputs more similar to that of the GRU that does poorly on language modeling. That this architecture would be able to learn without severe instability or succumbing to exploding gradients is not intuitively obvious. B4. Our model uses equal or fewer parameters compared to the models it is compared against. While BC3 did not outperform the highly tuned AWD-LSTM BID20 or skip connection LSTM BID19 , it did outperform the Recurrent Highway Network BID35 and NASCell (Zoph and Le, 2017) on the Penn Treebank, where NASCell is an RNN found using reinforcement learning architecture search specifically optimized over the Penn Treebank. For our experiments involving the extended DSL and our RL based generator, we use machine translation as our domain. The candidate architectures produced by the RL agent were directly used without the assistance of a ranking function. This leads to a different kind of generator: whereas the ranking function learns global knowledge about the whole architecture, the RL agent is trimmed towards local knowledge about which operator is ideal to be next. Training details Before evaluating the constructed architectures, we pre-train our generator to internalize intuitive priors. These priors include enforcing well formed RNNs (i.e. ensuring x t , h t−1 , and one or more matrix multiplications and activations are used) and moderate depth restrictions (between 3 and 11 nodes deep). The full list of priors and model details are in Appendix C1.For the model evaluation, we ran up to 28 architectures in parallel, optimizing one batch after receiving results from at least four architectures. As failing architectures (such as those with exploding gradients) return early, we needed to ensure the batch contained a mix of both positive and negative results. To ensure the generator yielded mostly functioning architectures whilst understanding the negative impact of invalid architectures, we chose to require at least three good architectures with a maximum of one failing architecture per batch. For candidate architectures with multiple placement options for the memory gate c t , we evaluated all possible locations and waited until we had received the results for all variations. The best c t architecture result was then used as the reward for the architecture. Baseline Machine Translation Experiment Details To ensure our baseline experiment was fast enough to evaluate many candidate architectures, we used the Multi30k English to German BID8 ) machine translation dataset. The training set consists of 30,000 sentence pairs that briefly describe Flickr captions. Our experiments are based on OpenNMT codebase with an attentional unidirectional encoder-decoder LSTM architecture, where we specifically replace the LSTM encoder with architectures designed using the extend DSL.For the hyper-parameters in our baseline experiment, we use a hidden and word encoding size of 300, 2 layers for the encoder and decoder RNNs, batch size of 64, back-propagation through time of 35 timesteps, dropout of 0.2, input feeding, and stochastic gradient descent. The learning rate starts at 1 and decays by 50% when validation perplexity fails to improve. Training stops when the learning rate drops below 0.03. FIG1 shows the relative frequency of each operator in the architectures that were used to optimize the generator each batch. For all the architectures in a batch, we sum up the absolute number that each operator occurs and divide by the total number of operators in all architectures of the batch. By doing this for all batches (x-axis), we can see which operators the generator prefers over time. Intriguingly, the generator seems to rely almost exclusively on the core DSL (MM , Gate3 , Tanh, Sigmoid , x t , h t−1 ) when generating early batches. The low usage of the extended DSL operators may also be due to these operators frequently resulting in unstable architectures, thus being ignored in early batches. Part way through training however the generator begins successfully using a wide variety of the extended DSL (Sub, Div , Sin, Cos, . . .). We hypothesize that the generator first learns to build robust architectures and is only then capable of inserting more varied operators without compromising the RNN's overall stability. Since the reward function it is fitting is complex and unknown to the generator, it requires substantial training time before the generator can understand how robust architectures are structured. However, the generator seems to view the extended DSL as beneficial given it continues using these operators. Overall, the generator found 806 architectures that out-performed the LSTM based on raw test BLEU score, out of a total of 3450 evaluated architectures (23%). The best architecture (determined by the validation BLEU score) achieved a test BLEU score of 36.53 respectively, compared to the standard LSTM's 34.05. Multiple cells also rediscovered a variant of residual networks (Add (Transformation(x t ), x t )) BID14 or highway networks (Gate3 (Transformation(x t ), x t , Sigmoid (. . .))) BID25 . Every operation in the core and extended DSL made their way into an architecture that outperformed the LSTM and many of the architectures found by the generator would likely not be considered valid by the standards of current architectures. Figure 5 highlights how often the full range of operators occur in architectures that out-performed the LSTM. These results suggest that the space of successful RNN architectures might hold many unexplored combinations with human bias possibly preventing their discovery. In Table 3 we take the top five architectures found during automated architecture search on the Multi30k dataset and test them over the IWSLT 2016 (English to German) dataset BID6 . The training set consists of 209,772 sentence pairs from transcribed TED presentations Operator frequency of architectures that out-perform LSTM on multi30k DISPLAYFORM0 Figure 5: Operator frequency of architectures that out-perform LSTM on Multi30k (colored like FIG1 ). For every architecture with a BLEU score higher than LSTM, we count if an operator occurs in its architecture. While variables x t and h t−1 are inherent to every architecture, the generator also picked up on the Gate3 − Sigmoid combination for every one of its top architectures. Intriguingly, even operators that are less commonly used in the field such as sine curves and positional encoding occur in a large number of architectures and thus seem to contribute to successful architectures. Table 3 : Model loss and BLEU on the Multi30k and IWSLT'16 MT datasets. All architectures were generated on the Multi30k dataset other than the LSTM and BC3 from the LM architecture search. We did not perform any hyperparameter optimizations on either dataset to avoid unfair comparisons, though the initial OpenNMT hyperparameters likely favored the baseline LSTM model.that cover a wide variety of topics with more conversational language than in the Multi30k dataset. This dataset is larger, both in number of sentences and vocabulary, and was not seen during the architecture search. While all six architectures achieved higher validation and test BLEU on Multi30k than the LSTM baseline, it appears the architectures did not transfer cleanly to the larger IWSLT dataset. This suggests that architecture search should be either run on larger datasets to begin with (a computationally expensive proposition) or evaluated over multiple datasets if the aim is to produce general architectures. We also found that the correlation between loss and BLEU is far from optimal: architectures performing exceptionally well on the loss sometimes scored poorly on BLEU. It is also unclear how these metrics generalize to perceived human quality of the model BID30 and thus using a qualitatively and quantitatively more accurate metric is likely to benefit the generator. For hyper parameters of the IWSLT model, refer to Appendix C3. Architecture engineering has a long history, with many traditional explorations involving a large amount of computing resources and an extensive exploration of hyperparamters BID17 BID11 BID5 . The approach most similar to our work is Zoph and Le (2017) which introduces a policy gradient approach to search for convolutional and recurrent neural architectures. Their approach to generating recurrent neural networks was slot filling, where element-wise operations were selected for the nodes of a binary tree of specific size. The node to produce c t was selected once all slots had been filled. This slot filling approach is not highly flexible in regards to the architectures it allows. As opposed to our DSL, it is not possible to have matrix multiplications on internal nodes, inputs can only be used at the bottom of the tree, and there is no complex representation of the hidden states h t−1 or c t−1 as our unrolling ranking function provides. Many other similar techniques utilizing reinforcement learning approaches have emerged such as designing CNN architectures with Q-learning BID1 .Neuroevolution techniques such as NeuroEvolution of Augmenting Topologies (NEAT) BID26 and HyperNEAT BID27 ) evolve the weight parameters and structures of neural networks. These techniques have been extended to producing the non-shared weights for an LSTM from a small neural network BID12 and evolving the structure of a network BID9 BID2 ). We introduced a flexible domain specific language for defining recurrent neural network architectures that can represent most human designed architectures. It is this flexibility that allowed our generators to come up with novel combinations in two tasks. These architectures used both core operators that are already used in current architectures as well as operators that are largely unstudied such as division or sine curves. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed. We also introduce a component-based concept for architecture search from which we instantiated two approaches: a ranking function driven search which allows for richer representations of complex RNN architectures that involve long term memory (c t ) nodes, and a Reinforcement Learning agent that internalizes knowledge about the search space to propose increasingly better architectures. As computing resources continue to grow, we see automated architecture generation as a promising avenue for future research. APPENDIX A: DOMAIN SPECIFIC LANGUAGE DISPLAYFORM0 To improve the running speed of the RNN cell architectures, we can collect all matrix multiplications performed on a single source node (xt, xt−1, ht−1, or ct−1) and batch them into a single matrix multiplication. As an example, this optimization would simplify the LSTM's 8 small matrix multiplications (4 for xt, 4 for ht−1) into 2 large matrix multiplications. This allows for higher GPU utilization and lower CUDA kernel launch overhead. There exist many possible DSL specifications that result in an equivalent RNN cell. When two matrix multiplications are applied to the same source node, such as Add (MM (xt), MM (xt)), a matrix multiplication reaching an equivalent result can be achieved by constructing a specific matrix and calculating MM (xt). Additional equivalences can be found when an operator is commutative, such as Add (xt, ht−1) being equivalent to the reordered Add (ht−1, xt). We can define a canonical ordering of an architecture by sorting the arguments of any commutative nodes. In our work, nodes are sorted according to their DSL represented as a string, though any consistent ordering is allowable. For our core DSL, the only non-commutative operation is Gate3 , where the first two arguments can be sorted, but the input to the gate must remain in its original position. For our extended DSL, the Sub and Div operators are order sensitive and disallow any reordering. The models are trained using stochastic gradient descent (SGD) with an initial learning rate of 20. Training continues for 40 epochs with the learning rate being divided by 4 if the validation perplexity has not improved since the last epoch. Dropout is applied to the word embeddings and outputs of layers as in BID33 at a rate of 0.2. Weights for the word vectors and the sof tmax were also tied BID16 BID23 . Aggressive gradient clipping (0.075) was performed to ensure that architectures such as the ReLU RNN would be able to train without exploding gradients. The embeddings were initialized randomly between [−0.04, 0.04].During training, any candidate architectures that experienced exploding gradients or had perplexity over 500 after five epochs were regarded as failed architectures. Failed architectures were immediately terminated. While not desirable, failed architectures still serve as useful training examples for the ranking function. For the ranking function, we use a hidden size of 128 for the TreeLSTM nodes and a batch size of 16. We use L2 regularization of 1 × 10 −4 and dropout on the final dense layer output of 0.2. As we are more interested in reducing the perplexity error for better architectures, we sample architectures more frequently if their perplexity is lower. For unrolling of the architectures, a proper unroll would replace xt with xt−1 and xt−1 with xt−2. We found the ranking network performed better without these substitutions however and thus only substituted ht−1 to ht−2 and ct−1 to ct−2. The baseline experiments that are used during the architecture search are important in dictating what models are eventually generated. As an example, BC3 may not have been discovered if we had used all the standard regularization techniques in the baseline language modeling experiment. Analyzing how variational dropout BID10 would work when applied to BC3 frames the importance of hyperparameter selection for the baseline experiment. On LSTM cells, variational dropout BID10 ) is only performed upon ht−1, not ct−1, as otherwise the long term hidden state ct would be destroyed. For BC3, equation 6 shows that the final gating operation mixes ct and ht−1. If variational dropout is applied to ht−1 in this equation, BC3's hidden state ht will have permanently lost information. Applying variational dropout only to the ht−1 values in the two gates f and o ensures no information is lost. This observation provides good justification for not performing variational dropout in the baseline experiment given that this architecture (and any architecture which uses ht−1 in a direct manner like this) would be disadvantaged otherwise. For the Penn Treebank BC3 language modeling results, the majority of hyper parameters were left equal to that of the baseline AWD-LSTM. The model was trained for 200 epochs using NT-ASGD with a learning rate of 15, a batch size of 20 and BPTT of 70. The variational dropout for the input, RNN hidden layers, and output were set to 0.4, 0.25, and 0.4 respectively. Embedding dropout of 0.1 was used. The word vectors had dimensionality of 400 and the hidden layers had dimensionality of 1080. The BC3 used 3 layers with weight drop of 0.5 applied to the recurrent weight matrices. Activation regularization and temporal activation regularization of 2 and 2 were used. Gradient clipping was set to 0.25. Finetuning was run for an additional 13 epochs. For the WikiText-2 BC3 language modeling results, the parameters were kept equal to that of the Penn Treebank experiment. The model was run for a total of 100 epochs with 7 epochs of finetuning. For the Penn Treebank GRU language modeling results, the hyper parameters were equal to that of the BC3 PTB experiment but with a hidden size of 1350, weight drop of 0.3, learning rate of 20, and gradient clipping of 0.15, and temporal activation regularization of 1. The model was run for 280 epochs with 6 epochs of finetuning. For the WikiText-2 GRU language modeling results, the hyper parameters were kept equal to those of the Penn Treebank experiment. The model was run for 125 epochs with 50 epochs of finetuning where the weight drop was reduced to 0.15. To represent an architecture with the encoder, we traverse through the architecture recursively, starting from the root node. For each node, the operation is tokenized and embedded into a vector. An LSTM is then applied to this vector as well as the result vectors of all of the current node's children. Note that we use the same LSTM for every node but reset its hidden states between nodes so that it always starts from scratch for every child node. Based on the encoder's vector representation of an architecture, the action scores are determined as follows: action scores = sof tmax(linear(LST M (ReLU (linear(architecture encoding)))))We then choose the specific action with a multinomial applied to the action scores. We encourage exploration by randomly choosing the next action according to an epsilon-greedy strategy with = 0.05.The reward that expresses how well an architecture performed is computed based on the validation loss. We re-scale it according to a soft exponential so that the last few increases (distinguishing a good architecture from a great one) are rewarded more. The specific reward function we use is R(loss) = 0.2 × (140 − loss) + 4 0.3815×(140−loss)−50) which follows earlier efforts to keep the reward between zero and 140.For pre-training the generator, our list of priors are: Given the differences in generated architectures, and the usage of components likely to impact the long term hidden state of the RNN models, we began to explore the progression of the hidden state over time. Each of the activations differs substantially from those of the other architectures even though they are parsing the same input. As the input features are likely to not only be captured in different ways but also stored and processed differently, this suggests that ensembles of these highly heterogeneous architectures may be effective. Figure 6: Visualization of the hidden state over time for a variety of different generated architectures. Figure 7: This figure shows the progress of the generator over time, highlighting the switches between exploitation (increasing running average up to plateau) and exploitation (trying out new strategies and thus decreasing running average at first). Only valid architectures are shown. Higher reward is better with the x-axis showing progress in time. We ran the architecture search for 5 days on one CPU head node and several worker nodes with a total of 28 GPUs on AWS. 24 GPUs were of the type Tesla K80 and 4 GPUs were of the type Tesla M40. The best architecture (Table 3 , bottom row) was found after 40 hours. However, as evident in Figure 6 , the generator created well-performing architectures more and more consistently with more training.
We define a flexible DSL for RNN architecture generation that allows RNNs of varying size and complexity and propose a ranking function that represents RNNs as recursive neural networks, simulating their performance to decide on the most promising architectures. Introduces a new method to generate RNNs architectures using a domain-specific language for two types of generators (random and RL-based) together with a ranking function and evaluator. This paper casts the search of good RNN Cell architectures as a black-box optimization problem where examples are represented as an operator tree and scored based on learnt functions or generated by a RL agent. This paper investigates meta-learning strategy for automated architecture search in the context of RNN by using a DSL that specifies RNN recurrent operations.