a_i.type = op.argtypes_i\}$ ${\mathcal{V}}\gets $ $w' \gets w$ $s_{vo} \gets \Call{PropertySignature}{{\mathcal{V}}, \mathcal{O}, \mathit{Props}}$ \[alg:bustle\_line\_2\] $w' \gets \Call{ReweightWithModel}{M, s_{io}, s_{vo}, w}$ \[alg:bustle\_line\_3\] $E[w'] \gets E[w'] \cup \{{\mathcal{V}}\}$

Benchmarks {#section:benchmarks}
----------

We evaluate [<span style="font-variant:small-caps;">Bustle</span>]{} using a suite of 56 human-written benchmarks. These benchmarks were inspired by questions on help forums for spreadsheet programs, and designed to be difficult enough to stress our system; in fact, our best system solves less than half of them. The search space explored by the synthesizers to solve these benchmarks is quite large: on average, [<span style="font-variant:small-caps;">Bustle</span>]{} (orange line in Figure \[fig:progress\_plots\]) searches over 6 million expressions per benchmark attempt, and 1.5 million expressions per successful attempt. Most benchmarks have 2 or 3 input-output pairs, and no benchmark solved by [<span style="font-variant:small-caps;">Bustle</span>]{} has more than 4, though for some unsolved benchmarks we did feel that we needed more than 4 pairs to fully specify the semantics of the desired program. In each case, we gave what we felt was the number of pairs a user of such a system would find reasonable (though of course this is a subjective judgment). Three representative benchmarks are shown in Listing \[listing:benchmarks\]. See the Supplementary Material for a full list.


    