solves 23 tasks. This indicates that although learning in the loop outperforms domain-specific heuristics, the two approaches can be combined to achieve better performance than either approach alone. Finally, we can see that premise selection (purple) was slightly helpful, leading to 2 more solved tasks within 10 seconds compared to the baseline, but with much less improvement than [<span style="font-variant:small-caps;">Bustle</span>]{}. This is evidence that using learning from partial executions within the synthesis loop is crucial for [<span style="font-variant:small-caps;">Bustle</span>]{}’s success.

![ Plots of human-written benchmarks solved over time. (Left) The number of benchmarks solved as a function of intermediate expressions considered. This metric makes [<span style="font-variant:small-caps;">Bustle</span>]{} look somewhat better than it is, because it ignores slowdowns in wall-clock time, but it is still important to look at. First, it gives us an upper bound on how good we can do in wall-clock terms given by putting more engineering effort into speeding up the model. Second, it gives us a sense of how well the model is generalizing to the test set that is invariant to engineering considerations. (Right) Benchmarks solved per unit of elapsed wall-clock time. [<span style="font-variant:small-caps;">Bustle</span>]{} still outperforms baselines here, but not by quite as much, since inference takes 