The Dawn of Natural Language to SQL: Are We Fully Ready?

Translating users' natural language questions into SQL queries (i.e., NL2SQL) significantly lowers the barriers to accessing relational databases. The emergence of Large Language Models has introduced a novel paradigm in NL2SQL tasks, enhancing capabilities dramatically. However, this raises a critical question: Are we fully prepared to deploy NL2SQL models in production? To address the posed questions, we present a multi-angle NL2SQL evaluation framework, <PRE_TAG>NL2SQL360</POST_TAG>, to facilitate the design and test of new <PRE_TAG>NL2SQL methods</POST_TAG> for researchers. Through <PRE_TAG>NL2SQL360</POST_TAG>, we conduct a detailed comparison of leading <PRE_TAG>NL2SQL methods</POST_TAG> across a range of application scenarios, such as different data domains and SQL characteristics, offering valuable insights for selecting the most appropriate <PRE_TAG>NL2SQL methods</POST_TAG> for specific needs. Moreover, we explore the NL2SQL design space, leveraging <PRE_TAG>NL2SQL360</POST_TAG> to automate the identification of an optimal NL2SQL solution tailored to user-specific needs. Specifically, <PRE_TAG>NL2SQL360</POST_TAG> identifies an effective NL2SQL method, SuperSQL, distinguished under the Spdier dataset using the execution accuracy metric. Remarkably, SuperSQL achieves competitive performance with execution accuracy of 87% and 62.66% on the Spider and BIRD test sets, respectively.