follows (the insertion of adapters is omitted for clarity):

1.  The challenger starts as a copy of the champion, including learned weights, i.e., $(\mathcal{E}_{t2}, G_{t2}, \mathcal{D}_{t2}) \coloneqq (\mathcal{E}_{t1}, G_{t1}, \mathcal{D}_{t1})$.

2.  A pair of nodes $(u, v)$ is randomly selected from $G_{t2}$ such that $v$ is an ancestor of $u$.

3.  A module $\mathcal{M}_k$ is randomly selected from $\{\mathcal{M}_k\}_{k=1}^{K}$.

4.  A new node $w$ is added to $G_{t2}$ with $\mathcal{M}_k$ as its function.

5.  New edges $(u, w)$ and $(w, v)$ are added to $G_{t2}$.

6.  The scalar weight of $(w, v)$ is set such that its value after the softmax is some $\alpha \in (0, 1)$. To initially preserve champion behavior, $\alpha$ is set to be small. I.e., if $s_1, \ldots, s_m$ are the scales of the existing inbound edges to $v$, $s_{m+1}$ is the initial scale of the new edge, and $s_{\max} = \max(s_1, \ldots, s_m)$ then $$s_{m+1} = \ln\big(\frac{\alpha}{1-\alpha}\sum_{j=1..m} e^{s_j - s_{\max}}\big) + s_{\max} \, .$$

After challengers are generated, all champions and challengers are trained jointly for $M$ iterations with a gradient-based optimizer. Note that the scales of $G_{t1}$ and $G_{t2}$ diverge during training, as do the weights of $\mathcal{D}_{t1}$ and $\mathcal{D}_{t2}$. After training, all champions 