$s_m$ are scalars trained simultaneously with internal layer weights via backpropagation, and the constraint that all $s_m$ sum to 1 is enforced via a softmax function. Figure \[fig:soft\_order\] shows an example soft ordering network.

More formally, given shared layers $W_1, W_2, \ldots, W_D$, the soft ordering model $\bm{y}_t = f(\bm{x}_t)$ for the $t$-th task $\{(\bm{y}_{ti}, \bm{x}_{ti})\}_{i=1}^N$ is given by $\bm{y}_t = \mathcal{D}_t(\bm{y}^D_t)$, where $\bm{y}^0_t = \mathcal{E}_t(\bm{x}_t)$ and $$\begin{aligned}
    \bm{y}^{d}_t &= \text{softmerge}\big(W_1(\bm{y}^{d-1}_t), \ldots , W_D(\bm{y}^{d-1}_t)\big) \ \ \ \forall \ d \in 1..D\, , \label{eq:soft_order}\end{aligned}$$ where $\mathcal{E}_t$ is a task-specific encoder mapping the task input to the input of the shared layers, $\mathcal{D}_t$ is a task-specific decoder mapping the output of the shared layers to an output layer, e.g., classification. Although soft ordering allows flexible sharing across depths, layers are still only applied in a fixed grid-like topology, which biases and restricts the type of sharing that can be learned. This paper generalizes soft ordering layers to more general modules, and introduces evolutionary approaches to both design these modules and to discover how to assemble these modules into appropriate topologies for multitask learning. To our knowledge, this is the first paper that takes an evolutionary approach to deep MTL.

![Assembling networks 