selects the attention cell for evaluation by optimizing the Gaussian process upper confidence conditioned on those already evaluated attention cells.

![Illustration of the supergraph used by the differentiable method. []{data-label="fig-supergraph"}](SuperGraph.pdf){width="0.85\columnwidth"}

Differentiable Architecture Search {#sec-differentiable}
----------------------------------

Inspired by recent progress on differentiable architecture search [@liu2018darts], we develop a differentiable formulation of our proposed search space. The formulation makes it possible to jointly learn the attention cell design and network weights with back-propagation, without explicitly sampling and evaluating different cells.

### Differentiable Formulation of Search Space

We propose to represent the attention cell search space as a supergraph, where all the possible attention cells are different subgraphs of this supergraph. The supergraph representation allows us to parameterize the design of an attention cell with a set of continuous and differentiable connection weights between the nodes in the supergraph.

To be more specific, we define the supergraph to have $m$ levels, where each level has $n$ nodes. Each node is an attention operation of a pre-defined type (map-based or dot-product attention) and a pre-defined attention dimension. Fig. \[fig-supergraph\] shows an example supergraph with 2 levels, where each level has 4 nodes. The input feature map to the entire attention cell is passed to 