As described in Jiang et al. (2021a, b), PLR contains the following hyper-
parameters: replay probability $p$, maximum replay size $N_{\textrm{max}}$,
minimum replay size $N_{\textrm{min}}$ (we set $p=0$ if
$|\mathcal{P}|<N_{\textrm{min}}$), $N_{\textrm{train}}$ the total number of
trials for which to run a training task before re-sampling, and $s$ the
staleness coefficient. See Table D.5 for the hyper-parameters used. We
conducted a grid search over the replay probability $p\in\\{0.1,0.2,0.5\\}$,
size of the replay pool $N_{\textrm{max}}\in\\{1000,10000,50000\\}$ and
staleness coefficient $s\in\\{0.1,0.2\\}$, and in all cases set
$N_{\textrm{min}}$ to be $90\%$ of $N_{\textrm{max}}$.

Table D.5: PLR hyperparameters. Parameter | Value
---|---
$p$ | 0.2
$N_{\textrm{max}}$ | 1000
$N_{\textrm{min}}$ | 900
$N_{\textrm{train}}$ | 30
$s$ | 0.2

#### PLR fitness metric.

It remains for us to define the per-step fitness score $f_{i,k}$. For this, we
use the simplest regret-like metric, the 1-step TD-error (Jiang et al., 2021b;
Schaul et al., 2015). Concretely, we estimate the _TD-error fitness_ based on
the immediate value-predictions of the model:
$|r_{t}+\gamma\hat{v}^{t+1}_{0}-\hat{v}^{t}_{0}|$. In some settings this may
be undesirable, for example, TD-errors typically increase as the agent
achieves higher rewards. Therefore, we also propose to compute fitness metrics
based on the Muesli dynamics model. Rather than simply using the accuracy of
the model prediction, we look at the impact of the prediction on the value
function and action predictions. We define the _value-model fitness_ as
$|\hat{v}^{t+1}_{0}-\hat{v}^{t}_{1}|$, the difference between the value
estimate at the predicted next state and the true next state. We also define a
value-agnostic metric, the _action-model fitness_ as follows:
$JS(\hat{\pi}^{t+1}_{0},\hat{\pi}^{t}_{1})$, i.e. the difference between the
action predictions at the predicted next state and the actual next state,
where difference is measured with the Jensen-Shannon divergence (Xu et al.,
2017; Farquhar et al., 2021; Filos et al., 2022; Pislar et al., 2022).

Figure D.1:  PLR fitness metric comparison for zero-shot generalisation
($k=1$) and few-shot adaptation ($k=13$). We compare the TD-error fitness used
in our main agents against two approaches using the Muesli dynamics model. We
see that action-model fitness matches TD-error fitness in few-shot
performance, with weaker zero-shot performance.

In Figure D.1 we show training curves for TD-error fitness, value-model
fitness, and action-model fitness. Table D.6 shows the experimental setup for
these experiments. We see that both TD-error and action-model fitness metrics
outperform the value-model fitness. We chose TD-error for our PLR training
runs because it has better asymptotic performance in both the zero-shot and
the few-shot setting, and because it was shown to perform well in previous
work (Jiang et al., 2021a).

Table D.6: Experimental setup for comparing different PLR fitness functions. Model parameters | Memory | Task pool | Curriculum | Teacher | Steps | Fitness function
---|---|---|---|---|---|---
23M TXL / 76M total | 1800 | 25B | PLR | None | 25B | TD error
Value model
Action model

#### Curriculum efficiency.

(a)

(b)

Figure D.2:  (a) Sample efficiency in steps for different choices of
curricula. Both No-op and PLR significantly improves sample efficiency over
uniform sampling of tasks. Few-shot denotes $k=13$ score and zero-shot denotes
$k=1$ score. (b) Sample efficiency in FLOPs for different choices of
curricula. No-op has an initial advantage over PLR, but PLR outperforms No-op
later in training.

Next, we compare training sample efficiency for baseline uniform sampling and
the different curriculum methods, in units of both learner steps and FLOPS.
Figure D.2 shows the median last-trial scores for few-shot ($k=13$) and zero-
shot evaluation tasks as a function of learner steps. We see that both No-op
filtering and PLR curricula strongly improve training sample efficiency over
uniform sampling, with PLR being more efficient early in the training process.
When we plot the same few-shot median performance as a function of FLOPS, we
see that No-op has a slight early advantage, but PLR outperforms No-op later
in training. The initial advantage for No-op may be because No-op expends more
FLOPS (10 evaluations per task vs. 1 in PLR) for task evaluation, which finds
higher quality training tasks at the start of training.

#### Emergent curricula.

In Figure D.3 we show task metrics analysing the tasks selected by PLR and No-
op filtering, expanding upon the results shown in Figure 11.

Figure D.3:  Emergent curricula for No-op filtering and PLR. Plots show the
full set of task metrics for the dynamic training set, averaged over all tasks
in the set, with standard error shaded. In all plots, a higher metric value
corresponds to greater task difficulty. Horizontal lines show the same metric
values averaged over the test (dashed) and hand-authored (dotted) evaluation
task sets.

###  D.6 Distillation teacher for scaling experiments

In all of our scaling experiments (Sections 3.4 and 3.5), we distill the
policy from an identical teacher snapshot to ensure our experiments are
comparable. Training details for the teacher are detailed in Table D.7. This
teacher is used to kickstart our agents for their first 4B frames of training.

Table D.7: Distillation teacher for scaling experiments. Model parameters | Memory | Task pool | Curriculum | Teacher | Steps
---|---|---|---|---|---
23M TXL / 76M total | 1800 | 200M | No-op | None | 23B

###  D.7 Scaling the network

Table D.8 shows the experimental setup for the model size scaling experiments
in Section 3.4. More details about the number of parameters for the various
model sizes can be found in Table D.9.

Table D.8: Experimental setup for model size scaling. Model parameters | Memory | Task pool | Curriculum | Teacher | Steps
---|---|---|---|---|---
6M TXL / 41M total | 1800 | 25B | No-op | Table D.7 | 75B
23M TXL / 76M total
42M TXL / 112M total
57M TXL / 141M total
75M TXL / 175M total
169M TXL / 353M total
265M TXL / 533M total

Transformer-XL memory is a cached memory of previous attention layer inputs,
concatenated to the keys and values during each forward pass. Inputs to
intermediate layer are activations from the previous layer, which in
themselves contain information about the past. Caching $M$ activations this
way theoretically allows for an effective memory horizon of $M$ x $L$, where
$L$ is the number of attention layers in the network. Therefore, to avoid
implicitly scaling effective Transformer-XL memory length, in our model size
scaling experiments, we fix the number of layers in the Transformer, and scale
parameters only by altering the Transformer embedding size
($d_{\textrm{model}}$), with the feed-forward size fixed at
$4d_{\textrm{model}}$, as is standard in Transformer architectures (Vaswani et
al., 2017).

Table D.9: Transformer hyperparameters for different model sizes. Model parameters | Embedding size | Blocks | Key size | Value size | Heads | FFW size
---|---|---|---|---|---|---
6M TXL / 41M total | 288 | 6 | 48 | 48 | 6 | 1152
23M TXL / 76M total | 576 | 6 | 48 | 48 | 12 | 2304
42M TXL / 112M total | 768 | 6 | 32 | 32 | 24 | 3072
57M TXL / 141M total | 896 | 6 | 32 | 32 | 28 | 3584
75M TXL / 175M total | 1024 | 6 | 32 | 32 | 32 | 4096
169M TXL / 353M total | 1536 | 6 | 48 | 48 | 32 | 6144
265M TXL / 533M total | 1920 | 6 | 48 | 48 | 40 | 7680

###  D.8 Scaling the memory length

Table D.10 shows the details of the experimental setup for the memory length
scaling experiments in Section 3.4. We show the effective memory timesteps for
each experiment, computed as the number of cached network activations times
the number of transformer blocks (6).

Table D.10: Experimental setup for scaling the memory length. Model Parameters | Memory | Training task pool | Curriculum | Teacher | Training steps
---|---|---|---|---|---
23M TXL / 76M total | 600 | 200M | No-op | Table D.7 | 25B
1800
3000
4200

###  D.9 Scaling the size of the task pool

Table D.11 shows the details of the experimental setup for scaling the size of
the task pool in Section 3.5.

Table D.11: Experimental setup for scaling the task pool size. Model parameters | Memory | Training task pool | Curriculum | Teacher | Training steps
---|---|---|---|---|---
23M TXL / 76M total | 1800 | 200M | No-op | Table D.7 | 25B
25B
75M TXL / 175M total | 200M
25B

###  D.10 Scaling the complexity of the task pool

Table D.12 shows the details of the experimental setup for scaling the
complexity of the task pool in Appendix E.3. In this experiment, the
distillation teachers are different for the two agents we compare. Therefore
we cannot disentangle the effects of distillation and task complexity.
Nevertheless, the results remain indicative of the importance of task
complexity. The teacher for the task distribution across multiple world
topologies is trained as in Table D.7. The teacher for the task distribution
in a single room comes from a long lineage (> 6 generations) of distillation
teachers starting with agents trained on XLand 1.0 (OEL Team et al., 2021).

Table D.12: Experimental setup for scaling the complexity of the task distribution. Model parameters | Memory | Task pool | Curriculum | Steps
---|---|---|---|---
6M TXL / 41M total | 1800 | 4k worlds $\times$ 50k games | No-op | 23B
23M TXL / 76M total
42M TXL / 112M total
57M TXL / 141M total
75M TXL / 175M total
6M TXL / 41M total | 1800 | 1 world $\times$ 5k inits $\times$ 50k games | No-op | 23B
23M TXL / 76M total
42M TXL / 112M total
57M TXL / 141M total

###  D.11 Distillation enables scaling agents

Table D.13 shows the experimental setup for the distillation experiments in
Section 3.6.

Table D.13: Experimental setup for distillation experiments. Model Parameters | Memory | Training task pool | Curriculum | Teacher | Steps
---|---|---|---|---|---
TXL 23M TXL / 76M total | 1800 | See Sec D.3 | PLR (D.5) | Table D.2 | 22B
None
TXL 265M TXL / 533M total | Table D.2
None

###  D.12 Training on more trials with skip memory

Table D.14 shows the details of the experimental setup for the memory scaling
experiments in Section 3.7. This is the same setup as in Table D.14, except
for the number of training steps and the variation of memory architecture and
training trials discussed in the main text.

Table D.14: Experimental setup for experiments training on more trials with skip memory. Model Parameters | Memory | Training task pool | Curriculum | Teacher | Steps
---|---|---|---|---|---
23M TXL / 76M total | 1800 | 200M | No-op | Table D.7 | 50B
$1800\times 4=7200$

##  Appendix E Additional Experiments

###  E.1 Multi-agent adaptation

Figure E.1: We report the distribution of normalised task scores over the
multi-agent task test set when evaluated with various numbers of trials. All
tasks are evaluated in cooperative self-play. On the $y$-axis is the total
last-trial reward relative to that of an agent fine-tuned on the test tasks
(approximating “infinite trials” performance). Curves moving further towards
the top right corner indicate better performance. When given more trials, the
agent achieves higher scores in the last trial, showing test-time adaptation
across most of the task distribution (shaded regions).  Figure E.2: Average
performance and representative behaviour of AdA on the probe task Irreversible
Production for Two when evaluated in self-play with various numbers of trials.
AdA’s performance increases when given more trials, showing test-time
adaptation. The top-down view images show representative last-trial
trajectories when given different numbers of total trials. A corresponding
video for the case $k=8$ shows the behaviour across all trials within one
episode.

Figure E.1 demonstrates adaptation across a wide range of percentiles on a
test-task set of multi-agent tasks. Figure E.2 demonstrates last-trial
performance of AdA in one particular probe task. To generate these plots, AdA
was trained as described in Section 3.1 (Multi-agent).

###  E.2 Conditioning on number of shots doesn’t affect agents’ performance

Figure E.3 shows the score obtained by AdA for each percentile, in trial 1 of
episodes with only 1 trial ($k=1$) and in trial 1 of episodes with 8 trials
($k=8$) in our held-out test set. The overlap of these lines indicates that
AdA does not use the trial conditioning information it observes to adjust its
behaviour in any way that affects its score. If the agent were to follow a
more exploratory policy when it has more trials, we might expect the scores of
trial 1 with $k=8$ to be lower than the score of trial 1 with $k=1$.

This may be the optimal policy for our XLand 2.0 tasks, or it may reveal a
limitation of our training procedure. One can imagine a scenario in which,
knowing that there are $8$ trials in total, a Bayes-optimal policy chooses to
display a less rewarding and more exploratory behaviour in trial $1$, compared
to how it would behave if told that there was only a single trial in which to
collect reward. For instance, an agent may be able to guarantee a
deterministic reward later, having discovered some key information, at the
cost of foregoing an stochastic reward early on. We did not directly
incentivise this behaviour in our training process. In fact, we may have
discouraged it, since AdA learns from all rewards in an episode (not just in
the last trial), and with a discount factor smaller than 1, which could lead
to myopic behaviour.

Figure E.3: A comparison of the first-trial score in episodes with 1 trial and
episodes with 8 trials. The lines are almost perfectly overlapping, which
indicates that our agent does not leverage number-of-trials conditioning
information to adjust its policy to a more exploratory one in early trials
when more trials are available.

###  E.3 Scaling complexity of the task pool

One final axis along which it is possible to scale our method is the overall
complexity of the task distribution. We compare our main task distribution, as
described in Section 2.1 and Appendix A.2 to a subset which maintains the use
of the same goals, production rules, and objects, but eliminates any
navigational complexity by having a single world topology: an empty room.
Recall that we count the number of tasks as the product of number of worlds
and the number of games. To disentangle the effects of scaling complexity
versus scaling the sheer number of tasks, we add 5,000 unique object
initialisation points for the empty room. These serve as the proxy “4,000
worlds” and are, by design, much less diverse and complex than the 4,000
worlds in the main training pool.777Note that the distribution over world
topologies we use here is smaller than the distribution used in the model
scaling experiments in Section 3.4, and results are therefore not comparable
across these sections. For more details of the experimental setup, see
Appendix D.10 and Table D.12.

In Figure E.4, we show that low environment complexity can be a bottleneck to
scaling, by comparing the effectiveness of model scaling between agents
trained on the two distributions, and each evaluated on their respective test
sets. On both the median (Figure 4(a)) and 20th (Figure 4(b)) percentiles in
the empty room, we see that past a certain point (42M Transformer parameters),
scaling model size begins to reduce performance. By contrast, in the
distribution with many world topologies (Figures 4(c) and 4(d)), increased
model size continues to improve performance far beyond this, showing
improvements through at least 75M Transformer parameters. Open-ended settings
with unbounded environment complexity, such as multi-agent systems, may
therefore be particularly important for scaling up adaptive agents.

(a) Task distribution with only empty room inhibits scaling (median).

(b) Task distribution with only empty room inhibits scaling (20th percentile).

(c) Task distribution with many world topologies facilitates scaling (median).

(d) Task distribution with many world topologies facilitates scaling (20th
percentile).

Figure E.4: The benefit of scaling model size is bottlenecked if the
distribution is not complex enough, even if the total number of tasks is
accounted for.

###  E.4 Computational cost

In the scaling experiments (Sections 3.4 and 3.5), we compare agents after
they have been trained for an equivalent number of steps. While this controls
for sample efficiency of models, here we provide an analysis of the
computational cost in FLOPs for a given experiment, and reproduce some of our
scaling results, controlling for for compute cost. We see that bigger is not
always better from this perspective. For each model size and memory length we
use JAX (Bradbury et al., 2018) cost analysis to estimate the number of FLOPs
per frame of the learner step and actor step (Table E.1).

(a)

(b)

Figure E.5:  Scaling Transformer model size controlling for the total number
of FLOPs for the learner and actors, including auto-curriculum evaluation
actors.

(a)

(b)

Figure E.6:  Scaling Transformer-XL memory controlling for the total number of FLOPs for the learner and actors, including auto-curriculum evaluation actors.  Table E.1: FLOPs per frame for different model sizes and memory lengths. Model parameters | Memory | Learner FLOPs per frame | Actor FLOPs per frame
---|---|---|---
6M TXL / 41M total | 1800 | 1,138,005,632 | 736,987,392
23M TXL / 76M total | 1,380,255,078 | 2,580,445,440
42M TXL / 112M total | 1,623,461,663 | 4,489,239,552
57M TXL / 141M total | 1,821,422,230 | 6,063,742,976
75M TXL / 175M total | 2,048,253,663 | 7,879,863,808
169M TXL / 353M total | 3,243,189,525 | 17,560,326,144
265M TXL / 533M total | 4,443,008,234 | 27,358,181,376
23M TXL / 76M total | 600 | 1,278,491,404 | 963,739,136
3000 | 1,481,009,258 | 4,181,042,688
4200 | 1,582,270,213 | 5,789,702,144

We multiply the values in Table E.1 by the number of learner/actor steps for
each experiment, then, for a given comparison, we take the largest such value
common to all experiments (usually associated with the smallest model) as the
total FLOPs, and make the comparison of each model at this number of FLOPs.
The results for the FLOPs-matched model scaling experiments are shown in
Figure E.5. We see a reduction in performance as the model size grows beyond a
“sweet spot” around 57M Transformer parameters (141M total parameters).
Results for the FLOPs-matched memory scaling experiments in Figure E.6 show
that there is still benefit to increasing context lengths given a fixed
computational budget. Details of the compute used for these experiments can be
found in Tables E.2 (model size) and E.3 (memory).

(a)

(b)

Figure E.7:  Scaling Transformer-XL model size controlling for the number of
learner FLOPs.

(a)

(b)

Figure E.8:  Scaling Transformer-XL memory length controlling for the number
of learner FLOPs.

We note that Table E.1 indicates poor scaling of actor step FLOPs with model
size, and suspect this could be due to poor optimisation of a single-step
query of the Transformer on TPU, compared to the operation batching achieved
with a rollout length of 80 on the learner. In order to account for this and
for potential discrepancies in number of actor steps due to differences in
curriculum evaluation based on model quality, we also provide plots which only
account for the learner step FLOPs for each model: Figures E.7 (model size),
and E.8 (memory length). These might be more informative, and show that
performance still increases as a function of model size and memory length,
albeit not as steeply as when controlling for sample efficiency directly.
Details of the compute used for these experiments is shown in Tables E.4
(model size) and E.5 (memory length).

Table E.2: Corresponding learner steps given total FLOPs for each model size. Model parameters | Memory | Total FLOPs | Learner steps
---|---|---|---
6M TXL / 41M total | 1800 | $2.0\times 10^{20}$ | 97B
23M TXL / 76M total | 44B
42M TXL / 112M total | 27B
57M TXL / 141M total | 23B
75M TXL / 175M total | 18B
169M TXL / 353M total | 9B
265M TXL / 533M total | 5B
Table E.3: Corresponding learner steps given total FLOPs for each memory length. Model parameters | Memory | Total FLOPs | Learner steps
---|---|---|---
23M TXL / 76M total | 600 | $7.9\times 10^{19}$ | 31B
1800 | 17B
3000 | 11B
4200 | 8B
Table E.4: Corresponding learner steps given learner-step-only FLOPs for each model size. Model parameters | Memory | Learner step FLOPs | Learner steps
---|---|---|---
6M TXL / 41M total | 1800 | $1.0\times 10^{20}$ | 92B
23M TXL / 76M total | 76B
42M TXL / 112M total | 65B
57M TXL / 141M total | 58B
75M TXL / 175M total | 51B
169M TXL / 353M total | 32B
265M TXL / 533M total | 23B
Table E.5: Corresponding learner steps given learner-step-only FLOPs for each memory length. Model parameters | Memory | Learner step FLOPs | Learner steps
---|---|---|---
23M TXL / 76M total | 600 | $3.9\times 10^{19}$ | 39B
1800 | 36B
3000 | 29B
4200 | 25B

(a)

(b)

Figure E.9:  Normalised few-shot score over three generations using the 23M
parameter Transformer-XL. The first and second generations correspond to the
agents shown in Figure 16. The third generation is distilled from the second
after it has been trained for 25 billion steps. The $x$-axis counts the
combined amount of experience, starting from the experience collected by the
original teacher. The third generation shows additional gains over the second,
but to a lesser degree than the gap between the first and second generations.

###  E.5 Repeated distillation

In Section 3.6, we show that distilling an agent into an identical student can
lead to large increases in the agent’s performance. Here, we investigate the
potential benefits of applying the procedure repeatedly. To this end, we
continue the experiment shown in Figure 16 and add a third generation, using a
snapshot taken from the previous student after 25 billion frames, equivalent
to 50 billion frames of total experience when taking the teacher’s experience
into account. Figure E.9 shows that applying this procedure repeatedly can
indeed lead to additional benefits; however, we observe diminishing returns
with successive generations.

##  Appendix F Human-Timescale Adaptation

In this section we provide more details regarding our claim of human-timescale
adaptation.

###  F.1 Probe tasks

Tables LABEL:probe-task-desc-single-agent and LABEL:probe-task-desc-multi-
agent describe in detail the single-agent and multi-agent probe tasks
respectively. Unless noted otherwise, all probe tasks are set in complex
worlds with many objects in them, and use multiple production rules, which are
fully hidden from the players.

Table F.1: Single-agent probe tasks |
---|---
Name |  Description
Wrong pair disappears |  The player’s goal is to hold a black cube, which does not exist among the initial objects. There are two (hidden) rules. The player needs to identify the correct world state which triggers the rule that creates the cube and not the one which destroys the necessary inputs. All this is embedded in a challenging world layout with one-way drops and limited visibility.
Wrong pair disappears, partial hiding |  ‘Wrong pair disappears’, but instead of hiding all rules completely we only hide the input objects for both rules (outputs and conditions are fully visible).
Irreversible production |  Similar to ‘Wrong pair disappears’, but this task has multiple dead ends (rules which create unsolvable states).
Irreversible production, all rules visible |  ‘Irreversible production’, but with all rules fully visible to the player.
Push, don’t lift |  The vast majority of training and evaluation tasks require lifting objects. Here two hidden rules destroy any object when lifted. In order to create the goal state, some “lateral thinking” is necessary: the player needs to identify that pushing the cubes with their body is possible.
Push, don’t lift, with distractors |  Similar to ‘Push, don’t lift’, but here a large number of distractor objects in the world make this a much more challenging exploration task for any player ignoring the objects mentioned in the goal.
Spacer tool |  Two objects need to be brought close together, but lifting them or touching them (with the avatar’s body) destroy them. The solution is to use another object in the world as a tool to push them together.
Transform to transport |  Again, two objects need to be brought close to each other, but lifting and touching them destroys them. The solution here is to exploit a set of rules that can turn one of the objects into something that can be carried safely, and then turn it back into the original object once it is in the right place.
Small workstation |  A very hard object handling task. 8 objects near the player’s spawn position need to be combined in different ways and 5 rules need to be triggered (some multiple times) to create the goal object. This task is set on top of a tall plateau and it is very easy to fail by touching an object too hard and throwing it off the edge of the plateau.
Small workstation, all rules visible |  The same as ‘Small workstation’, but all rules are visible to the player.
Crafting pyramid |  Eight objects need to be recursively combined first into four, then two and then ultimately one final object. This requires triggering a chain of 7 rules. This seems easy for humans. But the lack of intermediate reward makes this a hard hierarchical credit assignment task for agents.
Crafting tree, all rules visible |  Similar to the crafting trees in video games like Minecraft, this multi-step task requires triggering different rules in a chain to create the goal object. All objects exist in the world multiple times, making many different solutions viable.
Crafting tree, hidden shortcut |  Identical to ‘Crafting tree, all rules visible’, but one additional (hidden) rule exists. This allows the player to take a shortcut that lets them finish the task faster than executing only the visible rules.
Antimatter |  In a world full of yellow and black spheres, the goal is for no pair of black and yellow spheres to “see each other” (no direct line of sight). Beyond moving the objects and blocking line of sight with the avatar there exists a production rule which destroys any yellow sphere and black sphere pair which touch (similar in spirit to matter and antimatter particles). This is all embedded in a world requiring advanced navigation skills.
Antimatter with creation |  Similar to ‘Antimatter’, only here a third object (purple pyramids) exists which duplicates any sphere touching it. In addition, this task is set on two plateaus, making it easier to break line of sight and reducing the navigation challenge.
Pyramid in a haystack |  To create the necessary yellow pyramid, the player needs to find and hold the purple cube. There are several distractor objects and distractor rules in this world, requiring the player to deal not just with a hard exploration challenge but also a very noisy environment.
Protect the egg |  The player is tasked to hold the single yellow sphere in the world. A large number of other spheres exist in the world. These destroy the yellow sphere on collision. As these touch each other or get near the player they get duplicated. This can lead to a constantly growing number of objects, filling up the world.
3 spheres jumbled |  3 spheres exist in the world. Holding one of them creates the goal object. Only one sphere can be reached within the 10-second time limit, meaning that the optimal policy on the first trial is to choose uniformly at random.
Two doors |  The goal object is hidden behind one of the two large objects (the “doors”) positioned at opposite ends of the world. Only one of them can be reached in time, so the player needs to decide between exploring one of them per trial.
Same signature: match colours |  All ‘Same signature’ tasks have the exact same world layout, goal and number of fully-hidden rules. This means they look exactly the same to a player starting out. This one only requires two objects of matching colour to be brought close together to create the goal object, using only one production rule out of three.
Same signature: three steps |  This ‘Same signature’ task variant requires the player to trigger all three (hidden) production rules to create the goal object.
Same signature: two dead ends |  In this ‘Same signature’ task variant, two of the tree rules are dead ends, leading to an unsolvable world state. Only one rule is helpful (and in fact required) to solve the task.
Same signature: destroy to protect |  To solve this ‘Same signature’ task variant the player first needs to destroy a black sphere (by getting near it) before creating the goal object (a yellow cube). Otherwise when the black sphere “sees” the yellow cube, both get destroyed. This is a very hard credit assignment challenge even for human players (it is hard to notice what is going on).
Don’t act |  This task is pre-solved: the player is getting reward from the very beginning. If they lift or touch one of the two objects in the world, the object gets destroyed and reward is now impossible.
Don’t peek |  This task is ‘pre-solved’: the player is getting reward from the very beginning. However, the player will destroy any object they look at, at which point reward is impossible. So the optimal strategy is to not look at anything but the sky.
Navigation: find the cube |  This memory task is not using production rules. It uses a large world with the goal object (a cube) hidden after a very winding path.
Navigation: find the cube with teaser |  This memory task is not using production rules. It is set in a large world with the goal object (a cube) hidden after a very winding path. The object is visible from the spawn point but out of sight after starting to traverse the terrain.
Navigation: hold up high |  This memory task is not using production rules. The goal object (a pyramid) is hidden on top of a plateau and can only be seen when nearly there.
Object permanence: yellow cube |  This memory task is not using production rules. The goal object (a yellow cube) is visible from the spawn point. After moving for a bit, a decision between two paths has to be made, with the correct path being to the right. At this point the cube is no longer visible.
Object permanence: black cube |  This memory task is not using production rules. This has same world layout as the task above, only here the player is asked to find the black cube, which requires going to the left.
Table F.2: Multi-agent probe tasks |
---|---
Name |  Description
Pass over the wall |  The players are separated by an opaque wall. They cannot see each other, only their half of the world. The solution requires “passing” the accessible objects on either player’s side to other player to combine them into the goal object. Then the players must make sure the correct player holds this object.
Pass over the wall repeatedly |  Similar to ‘Pass over the wall’ but here 2 out of 3 initial objects are ‘frozen’ (cannot be moved). This prescribes a very specific solution strategy that requires the players to pass 3 objects over the wall in a specific order.
Coordinated production |  This task requires each player to be near a sphere to turn this sphere into a pyramid, and then for both pyramids to be touching each other to create the goal object. The spheres are slightly hidden in a complex world, requiring some exploration.
Coordinated production with deadends |  Like ‘Coordinated production’ but here each player will destroy one of the initial objects if they get near it. These dead-end rules make this a much harder exploration problem.
Coordinated exchange |  This requires each player to create a new object by holding an existing object, then to hold the object created by the other player to turn it into another intermediate object and finally for both objects to be combined into the goal object. While the world is fully accessible to both players, this can only be solved if both players actively participate.
Overcooked: coordination ring |  Inspired by the video game Overcooked (Carroll et al., 2019; Strouse et al., 2021), in this task both players need to “serve tomato soup to a hungry patron”. This is implemented as a repeatable four-step production rule chain which requires both players to traverse their shared space (a circular layout) carefully in order to not block the other player.
Overcooked: coordination ring, all rules visible |  While in ‘Overcooked: Coordination ring’ all production rules are hidden from both players, here they are fully visible (to match the dynamics of the original Overcooked game).
Overcooked: cramped room |  Similar to ‘Overcooked: coordination ring’ but with a different layout for the shared space and a different number of initial objects, using different shapes and colours.
Overcooked: cramped room, all rules visible |  While in ‘Overcooked: cramped room’ all production rules are hidden from both players, here they are fully visible (to match the dynamics of the original Overcooked game).
Overcooked: forced coordination |  Similar to the other ‘Overcooked’ task variants, but here both players are restricted to only a certain part of the world and so are forced to coordinate to solve this task. No player can solve this alone since they cannot reach all initial objects.
Overcooked: forced coordination, all rules visible |  While in ‘Overcooked: forced coordination’ all production rules are hidden from both players, here they are fully visible (to match the dynamics of the original Overcooked game).
Kickball |  This task is set in large world with two frozen pyramids on opposite sides of the world. Both players want to bring all of the plentiful purple spheres to the yellow pyramid. But lifting them destroys the pyramids so they need to “kick” them by bouncing them off the avatar. Think “soccer practice”.
Lemon eater |  The first player destroys all yellow spheres (of which there are many) when bumping into them. This is also the goal for both players. So they need to cooperate to bring all yellow spheres to the first player as quickly as possible.
Careful lemon eater |  Like ‘Lemon eater’ but any collision between two spheres turns them from yellow to purple. Purple spheres are “not edible” and so the players need to be careful to not create those, otherwise they will lose out on reward.
Lemon eater and maker |  Like ‘Lemon eater’ but we start out with only purple spheres. Only the second player can turn purple into yellow spheres by lifting them, effectively having to “create food” for the first player.
Antimatter for two |  Identical in nature to the single-player ‘Antimatter’ task but set in a different world layout and with two players who share the same goal.
Antimatter with creation for two |  Identical in nature to the single-player ‘Antimatter with creation’ task but set in a different world layout and with two players who share the same goal.
Antimatter with copies for two |  Identical to ‘Antimatter for two’ but here any two spheres of the same colour colliding leads to the creation of another sphere of that colour. This can set in motion runaway growth in the number of spheres, making it very hard to solve the task.
Irreversible production for two |  Identical in nature to the single-player ‘Irreversible production’ task but set in a different world layout and with two players who share the same goal.
Irreversible production for two, all rules visible |  Like ‘Irreversible production for two’ but with all production rules visible to both players.
Wrong pair disappears for two |  Identical in nature to the single-player ‘Wrong pair disappears’ task but set in a different world layout and with two players who share the same goal.
Wrong pair disappears for two, partial hiding |  Like ‘Wrong pair disappears for two’ but with only the input objects of the productions rules being hidden, the outputs and condition being visible.
Crafting pyramid for two |  Identical in nature to the single-player ‘Crafting pyramid’ task but set in a different world layout and with two players who share the same goal.
Information asymmetry |  A simple world in which the players are asked to execute a two-step production rule in the presence of multiple dead ends. While the first player knows all the rules, they are completely hidden from the second player. This task is intended to measure a specific flavor of third-person imitation.
Information asymmetry with repetition |  While ‘Information asymmetry’ only allows for up to four completions, this task variant is (given unlimited time) infinitely repeatable, providing more opportunities for imitation.
Combine outputs |  Similar in nature to ‘Coordinated production’ but set in a larger world and (through the use of frozen objects) requiring both players to repeatedly navigate quite far to create input objects for shared creation.
Two machines |  Many objects of different colours and shapes litter this world. The frozen yellow pyramid on one end of the world transforms these objects into an intermediary object. The players then need to bring the intermediary object to the frozen black pyramid on the other end of the world to “cash it in” for instantaneous reward, at which point the intermediary object is destroyed. Therefore to get more reward, the players must repeat this process.
Two machines with information asymmetry |  Like ‘Two machines’, but all rules are visible to one player, and all rules are hidden from the other player. This creates an information asymmetry and thus an opportunity for third-person imitation.

###  F.2 Comparing human and agent scores on every probe task

Figure F.1: Comparison of AdA against 19 human players on each of the 30 held-
out single-agent hand-authored tasks. The performance of a baseline agent
trained to optimise zero-shot performance is shown as a black dashed line and
indicates that AdA does not sacrifice its zero-shot generalisation to achieve
adaptation. The reward of an agent fine-tuned on the hand-authored tasks is
also shown as a red dashed line to provide some indication of the maximum
reward achievable in a trial of each task.

In Figure F.1 we show the raw last-trial total reward for humans and our agent
as a function of number of trials across every one of the $30$ evaluation
probe tasks.

###  F.3 Quantifying stochasticity

(a)

(b)

Figure F.2:  (a) Mean single-repetition score and 95% confidence intervals
over 50 samples. (b) Mean of the aggregated 50-repetition score and 95%
confidence intervals over 50 samples.

#### Task variation.

Figure 2(a) shows that there is fairly high variance in the score obtained by
a single agent over 50 repetitions of a single task. This is due to random
spawn rotations of the avatar following every environment reset and
stochasticity in the agent’s policy. To address this we run 50 repetitions of
every (task, $k$) combination and average the score over these repetitions.
This reduces the standard deviation to that shown in Figure 2(b). Here, the
standard deviation of the mean task score reduced from a maximum of 0.43 with
1 repetition, to a maximum of 0.06 with 50 repetitions.

Figure F.3: 95% bootstrap confidence intervals around the median and 20th
percentile over our test set of 5 agents trained with different initialisation
seeds. The maximum observed standard deviation was 0.04 around the median and
0.02 around the 20th percentile.

#### Agent initialisation variation.

After accounting for task variation, Figure F.3 shows the variance due to
agent initialisation seed during training. We plot the score as a function of
$k$ on our test set for the 76M total parameter version of AdA with 5
different initialisation seeds. The maximum standard deviation observed for
any number of trials in the median was 0.04 and for the 20th percentile was
0.02. This low initialisation seed variance led us to run our ablations with
one initialisation seed to save on compute. We note that the results shown in
our ablation section have significantly larger than one standard deviation
differences.

###  F.4 Prompting through first-person demonstrations

Figure F.4 shows the performance of AdA prompted with a fine-tuned agent
compared to an unprompted baseline on each of the 30 single agent hand-
authored probe tasks. The figure reveals a set of tasks on which AdA is able
to leverage information in the prompt, resulting in perfect or near perfect
scores. There are also tasks where AdA does not seem to be able to do this. In
all but one case, prompting does not hurt performance.

Figure F.4: A comparison of a prompted agent and unprompted baseline for our
full set of 30 hand-authored single-agent tasks. The dashed red lines indicate
the score obtained by the teacher providing the first-person demonstration to
the prompted agent in the first trial.

Analysing the tasks in Figure F.4 suggests that prompting is useful for short
navigational tasks such as Navigation: hold up high in which the agent follows
a short and simple path to reach the goal object. Prompting does not, however,
improve performance for longer and more complex navigation tasks like
Navigation: find the cube, likely due to the full demonstration being too long
to fit in the agent’s memory context.

We observe a similar pattern in tasks involving production rules. For tasks
with up to 2 production rules in the solution path, such as Same signature:
match colors, we observe the unprompted agent exploring different objects to
determine the correct rule to trigger. When prompted with a demonstration it
subsequently triggers the correct rule immediately and achieves a perfect
score. An exception to this is Same signature: destroy to protect where one of
the production rules involves destroying an object, which the agent does not
appear to remember from the demonstration. For tasks using 3 or more
production rules like Same signature: three steps (3 production rules in the
solution path) and Small workstation (5 production rules), the agent tends to
only remember a subset of the rules to trigger and continues engaging in
exploratory behaviour following the demonstration. The performance on these
tasks tend to match the unprompted baseline.

Another factor appearing to influence the effectiveness of prompting is the
topology and configuration of objects in the world, as seen in the Small
workstation tasks. While the teacher demonstrations for these tasks present a
clean trajectory, the agent subsequently knocks into and displaces distractor
objects, leading to environment states not observed during the demonstration
(and thus not recallable from memory). On the other hand, a favourable
configuration of objects appears to make the demonstration easier to learn
from, as observed in Irreversible production. Here the objects required to
trigger the last production rule are positioned close together. The agent
shows optimal behaviour here despite the task requiring 3 production rules on
the solution path. The agent also appears unable to infer from prompting
certain more subtle requirements like relative positioning of objects or tool
use. This is observed in tasks like Antimatter in which prompted AdA is able
to trigger the destruction rule but we did not observe it immediately hiding
objects from each other.

Prompting may also help eliminate biases that the agent may have acquired
during training. This is reflected in the lower score obtained by the
unprompted agent for small $k$ in Object permanence: yellow cube compared to
Object permanence: black cube. These tasks are identical except for the goal
being to navigate to a yellow cube on the right, or a black cube on the left
respectively. This suggests that the agent may have acquired a bias during
training to either prefer black cubes or to navigate towards objects on its
left. The significantly higher prompted scores on Object permanence: yellow
cube for small $k$ suggest that a prompt may help the agent overcome these
biases.

Figure F.5: Performance of AdA on 6 hand-authored tasks when prompted with an
expert first-person human demonstration, compared with an unprompted baseline.
Figure F.6: Top-down views depicting the behaviour of AdA with and without a
human expert prompt on the task Object permanence: yellow cube. On its own,
AdA appears to have a bias of navigating to the black cube which is a dead end
in this task. When prompted with a human (or fine-tuned) expert trajectory,
AdA is able to overcome this bias and navigate to the yellow cube in the
second trial.

#### Prompting with human demonstrations.

We prompted AdA with expert human demonstrations in a small selection of 6
hand-authored tasks, depicted in Figure F.5. These tasks were chosen to be a
mixture of tasks where AdA excelled with fine-tuned teacher prompting, where
it failed with fine-tuned teacher prompting and where even the fine-tuned
teacher failed.

The results show the same pattern as those obtained when prompting with a
fine-tuned teacher. In both Navigation: hold up high and Object permanence:
yellow cube, prompted AdA achieved close to optimal performance, exceeding
both the baseline and the human demonstration. Figure F.6 depicts the latter
behaviour in detail. AdA continues to fail to learn from a demonstration in
Navigation: find the cube with teaser and in both Spacer tool and Transform to
transport, which our fine-tuned teacher also failed at. A successful human
demonstration did not unlock any capabilities AdA was previously not capable
of demonstrating, suggesting that these tasks are perhaps too far out-of-
distribution with respect to AdA’s training tasks. The fact that prompting
with off-policy human demonstrations is partially successful is worthy of
note, and opens an exciting area of future research.