transformation layer, and each $\mathcal{D}_{ti}$ is a fully connected classification layer.

$G_{ti}$ is a DAG, whose single source node represents the input layer for that task, and whose single sink node represents the output layer, e.g., a classification layer. All other nodes either point to a module $\mathcal{M}_k$ to be applied at that location, or a parameterless adapter layer that ensures adjacent modules are technically compatible. In the experiments in this paper, all adapters are $2\times2$ max-pooling layers. Whenever a node of $G_{ti}$ has multiple incoming edges, their contents are combined in a learned soft merge (Eq. \[eq:soft\_merge\]). The algorithm begins by initializing the shared modules $\{\mathcal{M}_k\}_{k=1}^{K}$ with random weights. Then, each champion $(\mathcal{E}_{t1}, G_{t1}, \mathcal{D}_{t1})$ is initialized, with $\mathcal{E}_{t1}$ and $\mathcal{D}_{t1}$ initialized with random weights, and $G_{t1}$ according to some graph initialization policy. For example, the initialization of $G_{t1}$ can be minimal or random. In the experiments in this paper, $G_{t1}$ is initialized to reflect the classical deep multitask learning approach, i.e., $$\mathcal{E}_{t1} \rightarrow \mathcal{M}_1 \rightarrow \mathcal{M}_2 \rightarrow \ldots \rightarrow \mathcal{M}_K \rightarrow \mathcal{D}_{t1} \, ,$$ with adapters added as needed.

At the start of each meta-iteration, a challenger $(\mathcal{E}_{t2}, G_{t2}, \mathcal{D}_{t2})$ is generated by mutating the $t$th champion as 