to non-local blocks [@wang2018non], we add a residual connection between the input and output of the attention cell. So the final output of the attention cell is the sum of $f_0$ and $f_\text{comb}$. The combination procedure is the same for all attention cells.

Search Algorithm
================

Gaussian Process Bandit (GPB)
-----------------------------

Given $K$, i.e., the number of attention operations inside the attention cell, the attention cell design can be parameterized by a fixed number of hyper-parameters, including the attention dimension, the type and the activation function of each attention operation, and the input to each attention operation.

We employ GPB [@Srinivas2009GaussianPO; @snoek2012practical], a popular hyper-parameter optimization algorithm, to optimize all the hyper-parameters for the attention cell design jointly. Intuitively, GPB can predict the performance of an attention cell at a modest computational cost without actually training the entire network, based on those already evaluated attention cells. Such prediction helps GPB to select promising attention cells to evaluate in the following step and makes it possible to discover high-performing attention cells within a limited number of search steps.

Concretely, in GPB, the performance of an attention cell is modeled as a sample from a Gaussian process. At each search step, GPB 