to reduce the search space. Second, we consider the number of benchmarks solved as a function of wall-clock time, which takes into account the computational cost of calling the model.

We compare [<span style="font-variant:small-caps;">Bustle</span>]{} to three other methods. First we compare to the baseline bottom-up synthesizer without machine learning. Without learning, the synthesizer explores expressions in order of size, which is reasonable in this domain, because the correct programs are fairly small.

Second, we compare to the baseline synthesizer augmented with simple but effective human-written heuristics, to quantify whether the improvement is due to learning or due to having good features. Specifically, we reweight the intermediate string values encountered during the search based on whether they are a substring of the desired output and the edit distance compared to the output, normalized to a fraction.

Finally, we explore whether learning within the synthesis loop provides benefits compared to simply using learning once at the beginning of the search to choose which operations to prioritize, as in @DEEPCODER. Specifically, we train a model similar to the model trained for [<span style="font-variant:small-caps;">Bustle</span>]{}, but larger (since inference time is essentially inconsequential in this context) on the same data set used to train the 