Practical semantic parsers are expected to understand user utterances and map them to
executable programs, even when these are ambiguous.  We introduce
a new benchmark, ,
which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries.
In each case, the ambiguity persists even when the
database context is provided. This is achieved through a novel
approach that involves controlled generation of databases from
scratch.  We benchmark various LLMs on , revealing
that even the most advanced models struggle to identify
and interpret ambiguity in questions.

§ INTRODUCTION

Semantic parsing translates natural language utterances to logical
forms or executable programs in some machine-readable language
(e.g., SQL). It has emerged as an important component in many
real-world applications
as it allows users to seek information and control computer systems
naturally and flexibly in natural language. Practical semantic parsers
are expected to understand user utterances and map them to executable
even when these are ambiguous
Figure <ref> where a user request allows multiple
interpretations, each corresponding to a different logical
Ambiguity is a pervasive challenge in natural language
applications <cit.>, and semantic parsing is no exception.
<cit.> show that more than half of failure cases
for a text-to-SQL semantic parser are due to ambiguity which can occur
in different forms and at different levels.

Although the problem of mapping natural language utterances to formal
representations has been studied extensively, the issue of ambiguity
has received less attention.
<cit.> evaluate the ability of large language
models to parse ambiguous sentences to first-order logic, focusing on five well-known linguistic ambiguities.
In the context
of text-to-SQL parsing, other work <cit.> introduces vagueness into the
questions of popular benchmarks like Spider
by modifying their databases, e.g., through synonyms.
Although targeting a real-world application, database augmentation is
limited to a single type of ambiguity[Vagueness and ambiguity
are often considered distinct properties <cit.>;
however, for simplicity, we will refer to vagueness as a type of
ambiguity.]  and often operates in an artificial setting.  For
example, consider the database shown in
Figure <ref>b. We could add a “Scriptwriters” table
with the same content as the existing “Screenwriters” one. Our new
database would allow vague questions, but would not be very realistic
or well-designed.

In this paper we introduce , a novel benchmark  which we hope will both inform and inspire the development of parsers capable of
recognizing and interpreting ambiguous queries.   covers 16 distinct domains, it contains
846 multi-table databases, ambiguous questions,  their unambiguous interpretations provided by humans,
and complex SQL queries (4,242 in total).
It  includes three types of ambiguity, i.e., scope
ambiguity, attachment ambiguity, and vagueness,
showcasing a diverse range of SQL queries.
Figure <ref> shows examples of  ambiguous questions in blue blocks and their unambiguous interpretations in green, both of which can be phrased as requests.
to mimic real-world semantic parsing scenarios
with realistic and diverse databases, we create them automatically in three steps:
(a) we specify a domain of interest (e.g.,
Banking); (b) we
generate key concepts and relations such that they obey constraints
imposed by a chosen ambiguity type (e.g., for scope ambiguity,
the database must have a many-to-many relationship with a common element;
Figure <ref>a where multiple gyms offer the same
class, namely yoga); and (c) we generate SQL statements to construct
tables with the desired structure.
use a large language model for generation (OpenChat; )
and  view database creation as a semantic parsing
problem. Since we can automatically filter predicted SQL statements
based on execution results, minimal manual effort is required to
validate the generated databases and their content.[The code and data are publicly available at: <ambrosia-benchmark.github.io>]

We benchmark multiple advanced large language models of different sizes on , including the most recent
Llama 3 <cit.> and GPT-4o. Our experiments reveal that models struggle to recognize ambiguity and provide all possible SQL queries for all interpretations. They often capture only one interpretation and are biased towards a specific type of ambiguity.
The best model, Llama 3-70B, achieves only 32% recall on ambiguous questions compared to 65% on unambiguous ones.
offers a diverse range of questions from various domains, each introducing distinct types of ambiguity along with their interpretations. This diversity offers invaluable insights into the challenges of real-world semantic parsing.

Types of ambiguous questions (highlighted in blue), their interpretations (highlighted in green), and
corresponding SQL queries. Database elements that could lead to ambiguity are highlighted in orange.

§ RELATED WORK

The ambiguity inherent in natural language has been studied through
the lens of various tasks,
where models have been broadly found
lacking in their ability to resolve ambiguities.
The bulk of previous work has focused on question answering, with
emphasis on asking clarification questions to understand user intent
<cit.>, open-domain question answering where
a query can plausibly have multiple valid answers
<cit.>, disambiguating database search results
in the context of task-oriented dialogue systems
<cit.>, and
leveraging relevance feedback to rerank the answers returned from a QA
engine based on knowledge graphs <cit.>.

Within the broader area of semantic parsing, some work
<cit.> has concentrated on
clarifying vague questions for code generation. There is also interest
in creating datasets with ambiguous utterances and corresponding
logical form representations.  <cit.>
collect a dataset of $\lambda$-calculus translations that includes
examples of scope ambiguity, <cit.> explore
different types of ambiguities that arise in the task of mapping
search queries into SCFGs, and <cit.> create a
benchmark for mapping ambiguous sentences into first-order logic.
We also collect ambiguous utterances, however, our benchmark is
designed for parsing questions into SQL database queries.  Unlike
<cit.> and <cit.> who
create synthetic examples from templates, we ask human annotators to
write natural questions for a real-world application.

Related work in text-to-SQL parsing has primarily focused on modifying
existing datasets to obtain vague questions.
<cit.> detect questions containing ambiguous
tokens that could map to multiple columns.  Their dataset builds on
WikiSQL <cit.> and Squall
<cit.> which are limited to
single-table databases.  <cit.>
modify Spider <cit.> with ChatGPT to create
databases that exclusively support vague questions.
Despite relying on Spider, their approach often yields unrealistic
databases, e.g., they introduce ambiguity in table names by copying and renaming existing tables, which leads to information being
duplicated.  Our dataset, however, introduces various types of
ambiguity in questions, features diverse SQL queries within a single
type, and supports multi-table databases.

§.§ Design Considerations

Executable Logical Forms In designing our benchmark, we   concentrated on  text-to-SQL semantic parsing for several
reasons. It represents a real-world use case where ambiguity arises naturally, e.g., in  questions posed by
users and the structure of  databases.  Unlike some
other logical forms, SQL queries can be easily executed and inspected
for correctness. The task is familiar to LLMs, they have
demonstrated strong performance on standard benchmarks like Spider
<cit.> and BIRD-Bench <cit.>; it  is reasonable to expect them to be able to parse SQL queries even
in zero-shot settings since they likely have learned SQL syntax during
training. This allows us to focus on ambiguity per se, rather than the
model's ability to generate well-formed SQL.

Databases that Support Ambiguity Another important
consideration is ambiguity in the context of a database; it is
not enough to just have ambiguous questions, they must also retain
their ambiguity in relation to the database context.  For instance,
the question “What activities does each gym offer?” in
Figure <ref>a is ambiguous precisely because there are
fitness classes in the database common to multiple gyms (see the “Gyms_Classes”
table). Most databases  in academic text-to-SQL benchmarks
(e.g., Spider)
do not support ambiguous
As discussed earlier, modifying
these databases, e.g., by adding tables or columns with synonymous
names, makes them unrealistic with duplicate
information and does not cover different types of ambiguity.
In Section <ref> we describe a controllable,
multi-step approach that uses LLMs to generate databases supporting
question ambiguity.

Different Ambiguity Types
Finally, we wish to include different types of ambiguity  (see Figure <ref>).

Scope ambiguity arises when it is unclear
which elements a quantifier, such as “each”, “every”, or “all”, refers to.
There are two possible interpretations
for the ambiguous question in Figure <ref>a: in the collective
interpretation, the quantifier is interpreted widely (i.e., “each gym”
refers to all gyms in the database) and in the
distributive interpretation the quantifier is interpreted narrowly
(i.e., “each gym” is considered separately).

Attachment ambiguity occurs when it is unclear how a modifier
or phrase is attached to the rest of the sentence. There are two
possible interpretations for the question in
Figure <ref>b: in the high attachment reading, the
prepositional phrase  “on a work-for-hire” is attached to the verb “show” (i.e., both
screenwriters and editors are on work-for-hire contracts), whereas in the low
attachment reading it is attached to “editors” (i.e., only editors
have work-for-hire contracts, and screenwriters are on any contract). Within this
category, we also consider attachment ambiguities for relative clauses
(e.g., “writers and editors who have work-for-hire contracts”) and adjectives
(e.g., “work-for-hire editors and screenwriters”) as their underlying
database structure and SQL queries are similar to prepositional

Vagueness occurs when context creates uncertainty about which
set of entities is being referred to. Similarly to ambiguous
questions, there can be several interpretations. In the example in
Figure <ref>c, the question has three
interpretations depending on whether the answer refers to a general
entity (e.g., the bank) or a more specific subtype (e.g., the branch),
or both.

We create  with these considerations in mind, following
three steps: we select a domain and generate a database that supports one of the above ambiguity types; next, human annotators write ambiguous questions along with their
unambiguous interpretations;
finally, we automatically generate SQL queries based on templates for scope and attachment ambiguity, and ask annotators to write SQL queries for vague questions. Figure <ref> shows a sketch of the annotation process for scope ambiguity.

Annotation process for scope ambiguity  in the “Health” domain.

§.§ Database Generation

In this section we describe the database generation process.
We break this task into the following subtasks: selecting domains,  generating key concepts and relations for each type of  ambiguity,   and generating  and  SQL statements to build the database.

Domains, Concepts, and Relations

After analyzing existing semantic parsing datasets and publicly
available databases, we compiled a list of real-world domains for
database use. We refined this list to 16 domains (e.g., Entertainment,
Banking, Hospitality; see the full list in Appendix <ref>) as we found that some were either too narrow or
too broad for LLM-based database generation.

To allow different interpretations in the context of a database, the
latter must comply with constraints specific to a particular type of
ambiguity. In the scope ambiguity example in
Figure <ref>a, the database has information
pertaining to different gyms and classes: a gym can offer multiple
classes and a class can be offered by multiple gyms
but one class, i.e., yoga, is common among them. Due to
this structure (a many-to-many relationship with a common element),
the question “What activities does each gym offer?” has two different
interpretations (collective vs. distributive reading).

Similar to the conceptual data modeling stage used in database
modeling, we first identify  key concepts and relations that later
become sources of ambiguity in questions and databases. For each type
of ambiguity, we define a template that captures the general structure
of the data and the potential for ambiguity. Using this template and
ten in-context learning examples, we generate different structures
(key concepts and relationships) within a given
domain. Figure <ref> shows the prompt, templates,
in-context examples, and predictions which we obtain with a large
language model (see Appendix <ref> for the full prompts).  Specifically, we employ OpenChat
<cit.>, one of the strongest open-sourced 7B LLMs with high performance in both code and text generation.

The prompt,
templates, in-context examples (only one out of ten is shown for brevity, see Appendix <ref> for the full versions), and predictions of key concepts and relations for each ambiguity type.
key concepts and relations later become sources of ambiguity in
questions and databases (shown at the bottom for illustrative

We manually inspect and filter LLM predictions with hallucinations or
expressions otherwise unsuitable as database elements (e.g., proper
nouns are unlikely to serve as table or column names). We found the
vague category to be the most difficult to generate, requiring the
most filtering.  Examples include entities unsuitable for
further use as table or column names (e.g., “CGI” and “Practical
Effects” can  be valid answers for “What special effects were used
in a movie?” but do not suit table or column names in a realistic
database) or mutually exclusive (e.g., “Major Studio” and “Indie
generally do not produce
the same movie but are both types of
production). Out of 1,883 predictions generated across all
three categories, 881 (47%) were retained.

Database Generation via SQL statements
Based on the domain, key concepts, and relations from the
previous step, we generate and execute E statements to
define database tables and their structure, as well as
statements to add values to tables. For this, we use the OpenChat model in a zero-shot setting.
We determine possible database configurations for each type of
ambiguity, since the concepts and relations generated in the previous
step can be mapped to database elements in different ways. For
instance, “Screenwriters” and “Editors” are two concepts predicted for
the attachment example in Figure <ref>c, and can be
used as table names (shown in the example) or column names within a
single table.
In total, we
consider four configurations for databases that support attachment
ambiguity, two for databases that support vagueness, and one for scope
ambiguity, each of these corresponds to a separate instruction to the
LLM (see details in Appendix <ref>).

We first generate  statements (4–6 tables per database on
average), and then proceed to generate  statements
(3–5 rows per table on average). At each step, we do not accept
predictions that are non-executable or different from the selected
configuration.  We manually validate the generated databases and
filter those we consider unnatural.  For example, in
Figure <ref>c, a database with “Contract Type” as a
table would not be realistic, even though possible to generate.
contrast, “Projects”, an alternative common property for screenwriters and editors, could
be mapped to a column or table name.
The total number of databases
we obtain after filtering is 846.

§.§ Question and SQL Annotation

Ambiguous questions and their interpretations were written by human
SQL queries for scope and attachment ambiguity were generated
automatically, while those for vague questions were written by annotators. Annotators were recruited via the Prolific crowdsourcing  platform based on various screening tasks; they  were all native English speakers with prior knowledge of SQL or experience in database management.
We manually reviewed all submissions and ensured all annotators followed our instructions.
Details on crowdsourcing are provided in Appendix <ref>.

Scope and Attachment Ambiguity
To create examples of scope and attachment ambiguity, we first
automatically generate SQL queries using pre-defined templates for
each database configuration (one for scope and four for attachment
ambiguity). SQL queries are executed to ensure they yield
non-empty, distinct results (and thus correspond to an ambiguous
intent). We  then automatically generate templates for questions (e.g., “What
of  do we have for every ?” in Figure <ref>a). Prolific annotators (20 in total) are
shown a database as context and
asked to write ambiguous questions and
their interpretations in natural language based on these templates.
In total, we obtained 501 questions with scope ambiguity, 362 questions with attachment ambiguity and two interpretations in natural language per question. Each ambiguous question corresponds to two golden SQL queries, and each interpretation is itself an unambiguous question and corresponds to one of these SQL queries.

A major difference between vague questions and those with scope and
attachment ambiguities is that the former can map to very different
SQL queries. To promote the diversity of SQL queries in our dataset,
we ask experts (10 in total) to write SQL queries from scratch for this category
rather than relying on predefined templates.  To make the task easier,
we simplify the databases by dropping and renaming tables or
columns. For example, the “Bank” and “Branch” tables shown in
Figure <ref>d are merged into a “Banking Institution”
table (so that annotators inspect one general entity instead of two
specific ones). Annotators write SQL queries and corresponding
questions for the simpler tables (e.g., “Banking Institution”) which
we restore to the original databases once the annotation is
complete. A question is ambiguous as an entity or its reference can be
interpreted in different ways (e.g., a banking institution might be a
bank, a branch, or both). From the SQL query written by the
annotators, we create SQL queries corresponding to different
interpretations by replacing table or column names referring to the
general entity with the specific original names. Again, we execute the
queries to validate that they produce non-empty, distinct results.

Finally, a different batch of annotators  (the same
ones who  wrote scope and attachment questions and
interpretations) were shown vague questions and corresponding key concepts (e.g., a vague question “Who issued CD Special?” and key concepts “Bank” and “Branch” shown in Figure <ref>d) and asked  to write interpretations in natural language.  In total, we obtained
414 vague questions and 1,239 interpretations (two or three per
question). Each  question has  two or three
golden SQL queries, and each interpretation
is itself an unambiguous question and corresponds to one of these SQL

Comparison between  and other text-to-SQL datasets with ambiguous questions, NoisySP <cit.> and AmbiQT <cit.>.
#  ambig and #  unambig refer to the number of ambiguous questions and their unambiguous interpretations.

2cScope     2cAttachment     2cVague     2*# DB     2*# Tab/DB

# ambig     # unambig     # ambig     # unambig     # ambig     # unambig
NoisySP     0     0     0     0     8,673      0     8,086     1.0

AmbiQT     0     0     0     0     23,295     0     200     5.1
501     1,002     362     724     414     1,239     846     5.0

§.§ Dataset Analysis

Table <ref> shows  dataset statistics for  compared to two other text-to-SQL datasets with ambiguous questions <cit.>.  A unique aspect of  is that it includes three different types of ambiguity and provides  interpretations in natural language for  ambiguous questions.
The number of tables per database in  is comparable to AmbiQT, which is based on Spider. Incidentally, our database generation approach could be used  to augment existing text-to-SQL benchmarks, e.g., to assess robustness or out-of-domain generalization.

§ EXPERIMENTS

Below we present an experimental framework for evaluating model performance on  and offer insights into model capabilities and failure modes. Implementation details are provided in Appendix <ref>, and additional experimental results can be found in Appendix <ref>.

Models We benchmark various large language models (LLMs) on   in light of their growing use and good performance on text-to-SQL tasks <cit.>. Our experiments include LLMs of different sizes: OpenChat-7B <cit.>, the model  we used for database generation; the instruction-tuned Llama3-8B and Llama3-70B models from the Llama family <cit.>;  instruction-tuned CodeLlama-70B <cit.> which is trained specifically for code generation tasks; and GPT-3.5 Turbo and GPT-4o models from OpenAI.

offers various options for exploring ambiguity in semantic parsing.
In general, we expect a performant model  to be able to recognize ambiguity in the context of a database and output as many SQL interpretations as applicable. Thus,
our experiments  follow two  scenarios: (1) the model is given  instructions that acknowledge the  potential of ambiguity in the questions and specify that the output should include SQL queries for each possible interpretation; and (2) we provide standard text-to-SQL instructions
but consider  top-5 predictions from a beam of size 5 as does previous work <cit.>.
We refer to the first method as Prompt and the second as Beam.
In both scenarios, models have access to  database context as we  display the  and  statements which fully describe the database schema and content. Databases in  do not have many rows, and as such fit within the context limits of the LLMs we use. We acknowledge that in real-world applications database content can be very large, requiring specific methods to extract  related database entities. However, we leave this to future work.
For Prompt, we conduct experiments with temperature equal to 0.5 and 5 random seeds and average the results for all models except for the OpenAI ones (due to cost constraints). For Beam, experimental results are deterministic since  temperature is fixed to 0.
We reserve 10% of the dataset for few-shot learning.

Evaluation Metrics
A common approach to evaluating text-to-SQL semantic parsing is to compare whether the predicted SQL retrieves the same answer from the database as the gold logical form, typically by measuring execution accuracy. This method accommodates different formulations of the same SQL query and we employ it as well, but in our case, the output for ambiguous questions can be mapped to several correct SQL queries. We report recall and precision, but
our primary focus is recall on ambiguous questions, as it captures the extent to which a model predicts different SQL queries that correspond to all possible interpretations.
Following recent work <cit.>, we also
measure whether all SQL queries are generated for ambiguous questions, i.e., whether recall is 100%. We call this metric AllFound.

§.§ Zero-Shot Results

Recall, precision and AllFound metrics  for different zero-shot LLMs on . Llama3-70B (Prompt) captures ambiguity best, with the highest recall on ambiguous questions. All models are unable to parse multiple interpretations when these exist (see AllFound metric).

2*Model     2*Method     2c% Recall     2c% Precision     1c% AllFound

ambig     unambig     ambig    unambig    ambig
2*OpenChat-7B     Prompt     15.5      36.8      24.7      28.2      0.2

Beam     14.7      37.9
—     —

2*Llama3-8B     Prompt     19.0      47.1      29.6      37.6      0.7

Beam     18.6     46.1
—     —

2*CodeLlama-70B     Prompt
18.7     44.7     34.1     39.9     0.6

Beam     25.4     56.2
—     —

2*Llama3-70B     Prompt     31.5     65.4      43.0     49.3     2.7

Beam     27.8     65.6
—     —
GPT-3.5 Turbo      Prompt     26.7     61.6      40.2      52.1      0.5

GPT-4o     Prompt     27.1      63.4      51.1     59.6     0.4

Table <ref> summarizes the performance of zero-shot models on . We report results
using micro-averaging. The standard deviation of the Prompt method is within 0.3%–1.2% for precision and recall for all models, except for CodeLlama, which varies from 4% to 5%. The standard deviation for AllFound is below 0.3%.
For fairness, we do not report precision  for the Beam method as
it consistently outputs the top-5 predictions, although there are only 1–3 gold SQL queries.

As can be seen, all models demonstrate substantially higher recall on unambiguous questions compared to ambiguous ones,  with differences ranging from 21% for OpenChat-7B to 36% for GPT-4o and 38% for Llama3-70B.  As indicated by the AllFound metric, models generally fail to capture the ambiguity in the question as they rarely predict SQL queries for different interpretations.
They often predict a correct SQL query for  one interpretation only, which is why precision on ambiguous questions is higher than recall. Conversely,  models sometimes predict more than one SQL query for unambiguous questions, which explains  why they have  lower precision than recall. Note that recall for unambiguous questions is the same as execution accuracy in standard text-to-SQL with one gold SQL query. Table <ref> shows that the best recall is only 65.6% (achieved by Llama3-70B), which suggests  has challenging examples even in a standard text-to-SQL setting.

Overall, Llama3-70B (Prompt) captures ambiguity best, with the highest recall on ambiguous questions at nearly 32% and the highest AllFound value of 2.7%  which is admittedly still very low. Llama3-70B performs better with the Prompt method, while CodeLlama-70B appears to perform better when considering top-k predictions (Beam),  however, it never predicts both interpretations in this case.   CodeLlama-70B is very unstable with the Prompt method,
and shows performance comparable to Llama3-8B.
This instability indicates that parsing ambiguous questions is significantly different from other code generation tasks encountered during training, requiring different skills (e.g., explicit instructions or documentation).
GPT-4o performs best in terms of precision, however,  even for this model, precision is lower than recall on unambiguous questions which means that it predicts SQL queries when it should not. Generally, all models fail to provide multiple SQL queries when several interpretations are possible due to ambiguity  but can mistakenly offer more than one SQL query for unambiguous questions.

§.§ Analysis

Breakdown of model performance (zero-shot Llama3-70B)  by ambiguity type shows that attachment ambiguity is most challenging.

2c% Recall
2c% Precision     1c@% AllFound

Prompt             ambig     unambig     ambig
unambig     ambig
Scope            41.5      90.4      52.7     66.4     2.9

Attachment       12.7      24.0      13.7     13.4     0.3

Vague            35.6       69.4     56.7     56.4     4.6

2c% Recall      1@c@% AllFound

Beam             ambig     unambig     ambig
Scope       41.6      91.8      1.1

Attachment       10.3      22.2       0.0

Vague           25.8       69.1      0.3

We perform a more detailed analysis on the zero-shot instruction-tuned Llama 3-70B model, which has demonstrated the highest performance on our dataset.
Table <ref> presents recall, precision, and AllFound metrics for different categories of ambiguous and unambiguous questions. We observe that attachment ambiguity is most challenging. This category involves complex SQL queries (often requiring the  operator) and has the largest diversity in database configurations and corresponding gold SQL queries, which explains poor performance even on  unambiguous questions. In contrast, we obtain best results for scope ambiguity, achieving nearly 92% recall (i.e., execution accuracy) on unambiguous questions,  and 42% on ambiguous ones.
We consider only one database configuration for scope ambiguity, which might be more familiar to LLMs due to the widespread use of many-to-many relationships.   Interestingly, the model captures vagueness better with the Prompt method as evidenced by superior recall and AllFound results.

Table <ref> shows the distribution of model predictions by interpretation type. We consider ambiguous examples that have at least one correctly predicted SQL query. For vague questions, we focus on those with three interpretations and define two types of predictions: those with only one component (interpretations 1 and 2 in Figure <ref>c; Component) and those with all components (interpretation 3 in Figure <ref>c; Full).
There is a clear bias towards one interpretation type.  The distributive interpretation is preferred for scope ambiguity (in 80% of cases) corroborating the  findings of  <cit.>.  High attachment is  chosen in more than 97% of cases for attachment ambiguity, and interpretations involving individual components are also overwhelmingly preferred in the case of vagueness. Overall, the Prompt method leads to more biased predictions compared to Beam.

Distribution of model predictions (zero-shot Llama3-70B) by interpretation type. Across ambiguities, there is a clear bias towards one interpretation type.

2cScope     2cAttachment     2cVague

Method     Collective     Distributive     High    Low    Component     Full
Prompt     16.5      83.5      98.2     1.8      83.5     16.5

Beam     18.6      81.4      97.3      2.7      74.3      25.7

§.§ Few-Shot Results

Recall, precision, and AllFound metrics for zero-shot and few-shot Llama3-70B. Providing one example leads to the most substantial improvement.

Figure <ref> shows the performance of Llama3-70B (Prompt) in a few-shot learning setting.
We selected in-context examples randomly,  each includes ambiguous questions, their unambiguous interpretations, and corresponding SQL queries. We observe largest  improvement in recall and precision over the zero-shot method with one  example. Recall on ambiguous questions improves by 3% but remains substantially worse compared to  unambiguous questions.
Increasing the number of examples helps, but the improvements are not statistically significant given the 2–7% standard deviation.
Further analysis indicates that providing more examples for one type of ambiguity improves performance for that type but may negatively impact others.
Since it is unrealistic to have examples for all possible ambiguities, we consider these results as representing an upper bound.

§ LIMITATIONS

Despite our best efforts to create a high-quality dataset, we cannot guarantee that  is error-free. Recall that we rely on annotators to provide ambiguous questions and their interpretations, both of which may have flaws. Hence, some interpretations may be unclear, failing to disambiguate the question, or unnatural and overly explicit with direct mentions to  database entities.
Our databases generally have simple and clear names, whereas in reality, they might be incomplete, have
abbreviations, and so on. Since our experiments show that LLMs struggle to detect ambiguity and provide interpretations, we believe the current databases are well-suited for our task. However, future work might include augmentations to render them more realistic. When conducting experiments, we display the full database content, which is neither  scalable nor safe for real-world applications.
Consequently, our results can be seen as an upper bound on semantic parsing performance with ambiguous questions.

§ CONCLUSION

In this paper, we present , a novel dataset for parsing  ambiguous questions into SQL database queries across multiple domains.
We populate  with multi-table realistic databases that support ambiguity, having  developed an automatic pipeline for  controlled database generation using key concepts and relations.  covers three types of ambiguity and contains ambiguous questions along with their interpretations in natural language.
Our experiments demonstrate that even the most advanced LLMs struggle to capture ambiguity and provide accurate SQL queries for different interpretations of ambiguous questions, leaving ample room for improvement. We further hope  will spur future research on generalization (e.g., across   domains and ambiguity types). Databases with fixed structures can be also modified to explore other interesting scenarios, including cases where the database context helps clarify originally  ambiguous questions.

§ CHECKLIST

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?
See Section <ref>.

* Did you discuss any potential negative societal impacts of your work?
See Section <ref>.

* Have you read the ethics review guidelines and ensured that your paper conforms to them?

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments (e.g. for benchmarks)...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
See Section <ref>.

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
See Section <ref> and <ref>.

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
See Appendix <ref>.

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?

* Did you mention the license of the assets?
See Appendix <ref>.

* Did you include any new assets either in the supplemental material or as a URL?

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?
See Appendix <ref>.

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
See Appendix <ref>.

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?
See Appendix <ref> and Appendix <ref>.

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
See Appendix <ref>.

§ HOSTING, LICENSING AND MAINTENANCE

The data, instructions, and code used for the creation of  and our benchmarking experiments are publicly available under the CC BY 4.0 license: <ambrosia-benchmark.github.io>.
We plan to update the dataset by correcting any annotation errors as they are identified.
A  datasheet <cit.> for  provides detailed documentation in Appendix <ref>.

§ DOMAINS

The dataset includes the following domains across all categories of ambiguity: Airport, Banking, Education, Entertainment, Filmmaking, Hospitality, Job Postings, and Journalism. Additionally, the Scope and Vague categories include the domains of Agriculture, Traffic, Streaming Services, Healthcare, and the Housing Market. The Vague category alone also encompasses the domains of Construction, Demographics, and Students.

§ PROMPTS FOR KEY CONCEPTS AND RELATIONS GENERATION

Below, we provide the prompts used for generating key concepts and relations. The domain is indicated in grey to be filled in with a specific name. We sample with a temperature of 0.6 and top_p of 0.95 until at least 35 concepts and relations are generated, or the number of attempts exceeds 5.

[colback=white, colframe=purple, left=2pt,  coltitle=black, title=Scope: Key Concepts and Relations Generation, breakable, halign=flush left]
Provide 50 examples on different topics in the  domain in the following format:

Each  has many different . Among them,  is common to many .

Examples in different domains:

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

* Each  has many different . Among them,  is common to many .

Examples in the  domain:

[colback=white, colframe=purple, coltitle=black, parskip=5mm, title=Attachment: Key Concepts and Relations Generation, breakable, halign=flush left]
Provide 50 examples on different topics in the  domain in the following format:

and  are subclasses of . All  and  have property "". There might be a  and a  that both have "" equal to "".

Examples in different domains:

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be an  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and an  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and an  that both have "" equal to "".

*  and  are subclasses of . All  and  have the property "". There might be a  and a  that both have "" equal to "".

Examples in the  domain:

[colback=white, colframe=purple, left=2pt,  coltitle=black, title=Vague: Key Concepts and Relations Generation, breakable, halign=flush left]
Provide 50 examples on different topics in the  domain in the following format:

Question: [Who / What / How / When / Where ...]?     Subject of Inquiry:

Focus:    Possible answer types:

Examples in different domains:

* Question: Who organized the exhibition?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: When was this artefact created?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: What is the capacity of this venue?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: How was this research funded?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: Where was this play published?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: What powers this vehicle?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: When was this phone introduced?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: Who won this contest?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: What are the main features of this region?     Subject of Inquiry:

Focus:      Possible answer types:

* Question: Where is this painting displayed?     Subject of Inquiry:

Focus:      Possible answer types:

Examples in the  domain:

§ DATABASE GENERATION

Database configurations that support attachment ambiguity.

Database configurations that support vagueness.

Figure <ref> shows four possible configurations for attachment ambiguity, each of them corresponding to concepts in Figure <ref>c and supporting the ambiguous request “Show the writers and editors on  work for hire”. Figure <ref> shows two possible configurations for vague questions,  each of them corresponding to concepts in Figure <ref>d and supporting the ambiguous request “Who issued CD Special?”.

Below we provide examples of the prompts  used for the generation of  and  SQL statements for different ambiguity types and configurations. We indicate placeholders we substitute with specific values in grey. For databases with attachment and vague ambiguities, we provide the prompts corresponding to the configurations shown in Figures <ref>a and <ref>a, for the sake of  brevity.

Each configuration has strict conditions we can validate when executing predicted SQL statements. For instance, in the case of scope ambiguity, we automatically discard a database if there is no connection between the “” and “” tables. However, we relaxed some restrictions to generate more efficiently, as they can be satisfied by automatic modifications later. For example, we require a value in “” to be connected to multiple, but not necessarily all, values in “,” as we can later add the missing connections.
[colback=white, colframe=ivory, left=2pt,  coltitle=black, title=Scope: , halign=flush left]
Create multiple connected tables in the  domain via SQLite, including a "" table, a "" table (values: "" and others), and a "" table that joins these two tables (multiple rows of "" are connected to "" and other values of "").

Other tables and columns are arbitrary. Each table must contain at least 3 columns.

Provide multiple  statements in SQLite.

[colback=white, colframe=ivory, coltitle=black, title=Scope: , halign=flush left]
SQLite database in the   domain:

Insert 5 or more rows into each table.

"" table must contain "" and other values. All rows of table "" must be linked to two rows of "", with "" and with another value, through "".

Provide 5 or more  statements for each table in SQLite.

[colback=white, colframe=ivory, left=2pt,  coltitle=black, title=Attachment (a.): , breakable, halign=flush left]
Create multiple connected tables in the  domain via SQLite.  One table contains information about "" and "" (containing values such as "", etc.), and the other table contains information about "" and "".

Other tables and columns are arbitrary. Each table must contain at least 3 columns.

Provide multiple  statements in SQLite.

[colback=white, colframe=ivory, coltitle=black, title=Attachment (a.): , breakable, halign=flush left]
SQLite database in the   domain:

Insert 5 or more rows into each table.

There must be:

* One row with "" and "" equal to "",

* One row with "" and "" different from "",

* One row with "" and "" equal to "",

* One row with "" and "" different from "",

Other rows are arbitrary. Use non-trivial values.

Provide 5 or more  statements for each table in SQLite.

[colback=white, colframe=ivory, left=2pt,  coltitle=black, title=Vague (a.): , breakable, halign=flush left]
Create multiple connected tables in the  domain via SQLite, including a table "", a table "" and a table "" connected to the tables "" and "".

Other tables and columns are arbitrary. Each table must contain at least 3 columns.

Provide multiple  statements in SQLite.

[colback=white, colframe=ivory, coltitle=black, title=Vague (a.): , breakable, halign=flush left]
SQLite database in the   domain:

Insert 5 or more rows into each table.

Provide 5 or more  statements for each table in SQLite.

§ HUMAN ANNOTATION

We recruited Prolific annotators using the following criteria: English as the first and primary language, no language-related disorders or literacy difficulties,  knowledge of SQL and database management, a Prolific approval rate of 100%, and residency in the UK, USA, Ireland, Australia, New Zealand, or Canada. We also created trial tasks to choose the best annotators.
Each annotator was paid £9 per hour. The total budget for annotation was £1,600. All submissions in Prolific are anonymous. All annotators participated voluntarily and had the option to return the task at any time if they chose not to complete it. We only collected  data that was specifically requested from the annotators.

The first group of annotators (20 in total), participated in text-only tasks. These tasks included writing questions with scope and attachment ambiguity and  unambiguous interpretations for all questions, including vague ones. We assigned 3–5 instances per person, which took 10–20 minutes to annotate. The second group (10 annotators in total), wrote SQL queries and corresponding questions that were later rendered vague. Each annotator completed 3 instances, taking an average of 20 minutes per instance.

We provided separate instructions with examples for each task and manually reviewed all submissions to ensure all annotators followed them. We elicited annotations using the Potato tool <cit.>.   An example of the interface is shown in Figure <ref>. The code and detailed instructions used for annotation are available at: <https://github.com/saparina/ambrosia>.

Annotation Interface: example of writing questions with scope ambiguity and their unambiguous interpretations.

§ IMPLEMENTATION DETAILS

Models We implemented the Prompt method using the Text Generation Inference toolkit[<https://github.com/huggingface/text-generation-inference>] for the Llama-family models; for OpenChat <cit.>, we used  the official implementation provided by the authors and the OpenAI API for the GPT models[<https://platform.openai.com/>]. We used latest versions of these models: openchat-3.5-1210, gpt-3.5-turbo-0125, gpt-4o-2024-05-13. The temperature for GPT models is 0. We implemented the Beam method using the vLLM library <cit.>. We used 8-12 CPUs, one NVIDIA-A100  GPU for OpenChat 7B and Llama3-8B and four NVIDIA-A100 GPUs for CodeLlama-70B and Llama3-70B. For the largest models, one zero-shot experiment (with fixed random seed) took one hour and one few-shot experiment with 5 examples took 5 hours.

Below we provide the prompts used for the Prompt and Beam methods. We follow best practices for designing text-to-SQL prompts for LLMs, such as adding instructions like “Do not include any explanations, and do not select extra columns beyond those requested in the question”  to prevent models from generating redundant columns or descriptions <cit.>. However, we found that the models do not always adhere to these instructions. To address this, we implemented a separate extractor for SQL queries from each model's output. Additionally, we observed that models often generate separate SQL queries for attachment ambiguity instead of a single complex query with a  of two subqueries. To resolve this, we added the instruction “Show the results in one table” to each question in this category. When providing examples (Prompt method), we used the same prompt but with
“Some example databases, questions and corresponding SQL queries are provided based on similar problems: …”.

[colback=white, colframe=green, left=2pt,  coltitle=black, title=Prompt, breakable, halign=flush left]
The task is to write SQL queries based on the provided questions in English. Questions can take the form of an instruction or command and can be ambiguous, meaning they can be interpreted in different ways. In such cases, write all possible SQL queries corresponding to different interpretations and separate each SQL query with an empty line. Do not include any explanations, and do not select extra columns beyond those requested in the question.

Given the following SQLite database schema:

Answer the following:

[colback=white, colframe=green, coltitle=black, title=Beam, breakable, halign=flush left]
The task is to write SQL queries based on the provided questions in English. Questions can take the form of an instruction or command. Do not include any explanations, and do not select extra columns beyond those requested in the question.

Given the following SQLite database schema:

Answer the following:

Evaluation To compute evaluation metrics on model predictions, we represent the output of an SQL query as a set of values. This allows matching queries that correspond to the same interpretation and yield the same data even if they are structured differently, e.g., with different column orders or different operations.
For instance, some gold queries in the attachment category use the  operator, but they could also be written using the JOIN operator. Although this relaxation might result in occasional false positives, we found it rarely happens  as different SQL queries produce completely different outputs on our databases. We filter exact duplicates in predictions.

§ ADDITIONAL RESULTS

Extended Analysis Table <ref> presents the number of unique SQL queries predicted by the zero-shot Llama3-70B (Prompt) and the number of unique results obtained upon executing these predictions. As can be seen, the number of execution results is consistently lower than the number of predicted SQL queries, indicating that some queries, despite differing in structure, yield the same results. Typically, both metrics are around 2, implying that the model is capable of generating multiple distinct SQL queries that produce different results.
Although the figures for unambiguous questions are lower than for ambiguous ones, they still do not approach the optimum value of 1.0 (unambiguous questions correspond to a single SQL query).

To further explore how the model captures ambiguity, we consider the task of ambiguity detection. We present the model with a question and ask it to determine whether it  is ambiguous.
Similar to the Prompt method, we provide the model with comprehensive instructions regarding this task:
[colback=white, colframe=green, coltitle=black, title=Ambiguity Detection, breakable, halign=flush left]
The task is to identify ambiguous questions in English that are intended to interact with an SQLite database. Questions can take the form of an instruction or command and can be ambiguous, meaning they can be interpreted in different ways (corresponding to different SQL queries that produce different results). Answer Yes or No, and do not include any explanations.

Given the following SQLite database schema:

Is the following question ambiguous:

Table <ref> shows the accuracy of  zero-shot Llama3-70B in detecting ambiguity. Our results indicate that the model tends to overestimate ambiguity, classifying questions as ambiguous in approximately 80% of cases, regardless of whether they  genuinely are. This is a limitation of prompt-based approaches, which may confuse the model rather than provide helpful guidance.

Number of unique SQL queries predicted by zero-shot Llama3-70B (Prompt) and number of unique results after execution of these queries. The model predicts more than one query even for unambiguous questions.

2c# Unique SQL Queries     2c# Unique Exec Results

ambig     unambig     ambig
2.4    2.3     2.0     1.7

Zero-shot accuracy of the Llama3-70B in detecting ambiguity in questions. The model overestimates ambiguity.

2c% Accuracy

ambig     unambig
81.2       26.1

Error Cases
We conducted an error analysis on the zero-shot Llama3-70B (Prompt) by randomly selecting 50 examples where the model failed. Our analysis revealed errors typical of standard text-to-SQL parsing, as well as errors specifically related to ambiguity. Common text-to-SQL errors included: incorrect output columns (29%), including superfluous columns contrary to our instructions, errors in the filters and joins (17%), failure to include a UNION of two queries (17%, typical for the attachment category), and execution errors (5%). The most frequent error relating to ambiguity is the generation of different variations of the same SQL query, which occurred in 34% of cases, even for unambiguous questions. For ambiguous queries, the model provided only one interpretation in 11% of cases. See examples below:

@L@  L@
2cType: Scope      Domain: Agriculture     Ambiguous? Yes
Question     What kind of crop does each farm offer?

Gold     1.
Comments     All predicted queries are correct but they all provide the same information and correspond to one interpretation (distributive; gold query 2).

@L@  L@
2cType: Scope      Domain: Healthcare     Ambiguous? No
Question     For every program, list the activity descriptions

Comments     The first predicted query has an error in the output columns: it outputs the program description instead of the program name. The second query omits the program name column altogether.
2cType: Attachment      Domain: Filmmaking     Ambiguous? Yes
Question     Provide all sound designs and music scores created by Thomas Newman. Show them in one table.

Gold     1.
Comments     The first two predictions should be subqueries of a single complex query (failure to include a UNION of two queries). The last two queries are correct; however, they provide the same information and correspond to one interpretation (with high attachment; gold query 1).

2cType: Attachment      Domain: Hospitality     Ambiguous? No
Question     Show all youth hostels and also show the hotels, but only those with 50-priced rooms. Show them in one table.

Comments     All predicted queries have redundant output columns. The first two predictions should be subqueries of a single complex query (failure to include a UNION of two queries).

@L@  L@
2cType: Vague      Domain: Journalism     Ambiguous? Yes
Question     List all the articles with their respective editor and author names.

Gold     1.
Comments     All predicted queries are correct and correspond to two different interpretations (Component), but the third interpretation (Full) is missing.
2cType: Vague      Domain: Student     Ambiguous? No
Question     List students born after 1990, along with their field of study.

Comments     The first predicted query has an error in its filter as it incorrectly includes students born in 1990. The second query is not executable as SQLite does not have a built-in  function.

Below we provide a datasheet <cit.> for .

§.§ Motivation

For what purpose was the dataset created?
aims to study ambiguity in semantic parsing, with a focus on text-to-SQL parsing due to its real-world relevance. Large Language Models (LLMs) have demonstrated strong performance on existing unambiguous text-to-SQL benchmarks, allowing us to concentrate specifically on studying ambiguity. Unlike other logical forms, such as the first-order logic used by <cit.>, SQL queries are straightforward to execute and verify, making them ideal for testing. We aim to replicate practical semantic parsing scenarios with diverse databases in various domains that support ambiguity in questions. We also cover three different types of ambiguity and provide human-written interpretations that enable in-depth analysis. This distinguishes  from other benchmarks <cit.>, which rely on augmentations of existing datasets, and are thus less realistic, and lack diversity.

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? It was created by the authors.

Who funded the creation of the dataset? We gratefully acknowledge  the support of the UK Engineering and Physical Sciences Research Council
(grant EP/W002876/1).

§.§ Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?  includes databases, ambiguous questions and requests related to them, their corresponding interpretations and SQL queries.

How many instances are there in total (of each type, if appropriate)?    contains
846 multi-table databases in 16 distinct domains, 1,277 ambiguous questions,  their unambiguous interpretations provided by humans
and complex SQL queries (2,965 in total).

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
The dataset includes all instances we collected.  Our data collection approach allows for future expansion of the dataset.

What data does each instance consist of?
One example in  includes a database that supports ambiguity, an ambiguous question, its possible interpretations, and SQL queries corresponding to these interpretations. Interpretations can be viewed separately as unambiguous questions.

Is there a label or target associated with each instance?
For each question, the targets are the corresponding SQL queries: one query for an unambiguous question and 2–3 queries for an ambiguous one. Additionally, each database is labeled based on the domain it represents, its configuration, and the type of ambiguity present in the relevant questions.

Is any information missing from individual instances? No

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?
Annotators were given databases for context when writing questions, interpretations, and SQL queries for vague questions. Other SQL queries were constructed to query the given database. Annotators provided interpretations when they were aware of the ambiguous question they related to.

Are there recommended data splits (e.g., training, development/validation, testing)?
We reserve 10% for a few-shot learning scenario and recommend to evaluate on the remaining data.

Are there any errors, sources of noise, or redundancies in the dataset?
We manually validate all databases and human annotations. However, there might be annotation errors that we did not notice, such as interpretations that fail to disambiguate the questions or questions that are not ambiguous. The generated database might also contain tables that are irrelevant to the chosen key concepts.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is self-contained.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)?

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?

Does the dataset relate to people?

Does the dataset identify any subpopulations (e.g., by age, gender)?
There might be questions or databases containing information about age or gender, but they do not provide any opinions.

Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?
No, all annotation submissions were fully anonymous.

Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or  memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?
We manually verify that the dataset does not contain any sensitive or harmful information.

§.§ Collection Process

How was the data associated with each instance acquired?

Human annotators wrote ambiguous questions along with their unambiguous interpretations. Databases were generated by the LLM OpenChat (<cit.>), and SQL queries were automatically created using templates to address scope and attachment ambiguity. Annotators were asked to write SQL queries and corresponding questions for databases, which we had modified by merging vague concepts into more general ones, thereby removing ambiguity. After the annotation was complete, we restored the original databases and adjusted the SQL queries. The questions became vague due to the reintroduction of vague concepts. We manually verified questions, their interpretations and databases. SQL queries were executed to validate that they produced non-empty, distinct results. See details in Section 3 and Appendix E.

What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)?
We elicited annotations using the Potato tool <cit.>.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? N/A

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? We recruited annotators through the Prolific crowdsourcing platform. Each annotator was paid £9 per hour.

Over what timeframe was the data collected?
October 2023–April 2024.

Were any ethical review processes conducted (e.g., by an institutional review board)?

Does the dataset relate to people?

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? The data was collected directly.

Were the individuals in question notified about the data collection?
Yes, by requirement process in Prolific.

Did the individuals in question consent to the collection and use of their data?
Yes, by requirement process in Prolific.

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?

Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted?
No, all submissions on Prolific are anonymous, and we did not collect any data beyond what was requested.

§.§ Preprocessing/Cleaning/Labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
We manually validated all generated concepts and databases and human-written submissions.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?

Is the software used to preprocess/clean/label the instances available?
Yes, see <https://github.com/saparina/ambrosia>

§.§ Uses

Has the dataset been used for any tasks already?
Yes, we benchmarked LLMs on the text-to-SQL semantic parsing with ambiguous questions.

Is there a repository that links to any or all papers or systems that use the dataset?

What (other) tasks could the dataset be used for?
It can be used for studying different types of generalization, e.g., across domains or types of questions.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
We cannot guarantee that the dataset is free from errors (see Section 5 for limitations). Additionally, the recruitment of annotators with knowledge of database management and SQL introduces a potential bias. Future research could mitigate this by paraphrasing the questions.

Are there tasks for which the dataset should not be used?

§.§ Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
Yes, the dataset is publicly available.

How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
The dataset is available at  <ambrosia-benchmark.github.io>

When will the dataset be distributed?

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
The dataset is distributed under CC BY 4.0

Have any third parties imposed IP-based or other restrictions on the data associated with the instances?

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?

§.§ Maintenance

Who is supporting/hosting/maintaining the dataset? The authors of this paper.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? The first author of this paper.

Is there an erratum?

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
We plan to update the dataset by correcting any annotation errors as they are identified. The new version will be labeled accordingly.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?

Will older versions of the dataset continue to be supported/hosted/maintained?

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
Yes, the code used for data collection is available at <https://github.com/saparina/ambrosia>.