utterances among a fixed set of candidates, and are not useful for systems
that generate the utterance directly. The ibAbI dataset mentioned in the next
section has been created based on bAbI to cover several representative multi-
turn QA tasks.

Another interesting resource is the ParlAI framework151515http://parl.ai/ for
dialogue research, as it contains many popular datasets available all in one
place with the goal of sharing, training and evaluating dialogue models across
many tasks (Miller et al., 2017). Some of the dialogue datasets that are
included have been already mentioned (bAbI Dialog tasks and the Ubuntu Dialog
Corpus) but it also contains conversations mined from
OpenSubtitles161616http://opus.lingfil.uu.se/OpenSubtitles.php and Cornell
Movie171717https://www.cs.cornell.edu/~cristian/Cornell_Movie-
Dialogs_Corpus.html.

###  6.3 Datasets for Question Answering Dialogue Systems

Name | Topics | # dialogues | Reference
---|---|---|---
Ubuntu Dialogue Corpus | Ubuntu problems | 930K | (Lowe et al., 2015)
MSDialog | Microsoft products | 35K | (Qu et al., 2018)
ibAbI | Restaurant reservation | - | (Li et al., 2017a)
CoQA | 7 domains | 8,399 | (Reddy et al., 2018)
QuAC | People | 13,594 | (Choi et al., 2018)
DoQA | Cooking | 1,637 | (Campos et al., 2019)
Table 11: Datasets for question answering dialogue systems.

With respect to QA dialogue systems, two datasets have been created based on
human interactions from technical chats or forums. The first one is the Ubuntu
Dialogue Corpus, containing almost one million multi-turn dialogues extracted
from the Ubuntu chat logs, which was used to receive technical support for
various Ubuntu-related problems (Lowe et al., 2015). Similarly, MSDialog
contains dialogues from a forum dedicated to Microsoft products. MSDialog also
contains the user intent of each interaction (Qu et al., 2018).

ibAbI represents another approach for creating multi-turn QA datasets (Li et
al., 2017a). ibAbI interactivity adds to the bAbI dataset that was previously
presented (see Section 6.2) by adding sentences and ambiguous questions with
the corresponding disambiguation question, which should be asked by an
automatic system. The authors evaluate their system regarding the successful
tasks. However, it is unclear how to evaluate a system if it produces a
modified version of the disambiguation question.

Recently, several datasets that are very relevant for the context of QA
dialogue systems have been released. The CoQA (Conversational Question
Answering) dataset contains 8K dialogues and 127K conversation turns (Reddy et
al., 2018). The answers from CoQA are free-form text with their corresponding
evidence highlighted in the passage. It is a multi-domain dataset, as the
passages are selected from several sources, covering seven different domains:
children’s stories, literature, middle and high school English exams, news,
articles from Wikipedia, science and discussions from Reddit. QuAC (Question
Answering in Context) consists of 14K information-seeking QA dialogues (100K
total QA pairs) over sections from Wikipedia articles about people (Choi et
al., 2018). What makes it different from other datasets so far is that some of
the questions are unanswerable and that context is needed in order to answer
some of the questions. Another similar dataset that has unanswerable questions
and its questions are context-dependent is DoQA, a dataset for accessing
domain-specific Frequently Asked Question sites via conversational QA (Campos
et al., 2019). It contains 1,637 information-seeking dialogues on the cooking
domain (7,329 questions in total). An analysis carried out by the authors
showed that in this dataset there are less factoid questions than in the
others, as DoQA focuses on open-ended questions about specific topics. Amazon
Mechanical Turk was used to collect the dialogues for the three datasets.

###  6.4 Evaluation Challenges

We conclude this section by summarizing some of the recent evaluation
challenges that are popular for benchmarking state-of-the-art dialogue
systems. They have an important role in the evaluation of dialogue systems,
not only because they offer a good benchmark scenario to test and compare the
systems on a common platform, but also because they often release the dialogue
datasets for later evaluation.

Perhaps one of the most popular challenges is the Dialog State Tracking
Challenge (DSTC)181818https://www.microsoft.com/en-us/research/event/dialog-
state-tracking-challenge/, which was previously mentioned in this section.
DSTC was started in 2013 in order to provide a common testbed for the task of
dialogue state tracking. It continued on a yearly basis with remarkable
success. For its sixth edition, it was renamed as Dialog System Technology
Challenges due to the interest of the research community in a wider variety of
dialogue-related problems. Various well-known datasets have been produced and
released for every edition: DSTC1 has human-computer dialogues in the bus
timetable domain; DSTC2 and DSTC3 used human-computer dialogues in the
restaurant information domain; DSTC4 dialogues were human-human and in the
tourist information domain; DSTC5 also is from the tourist information domain,
but training dialogues are provided in one language and test dialogues are in
a different language. Finally, as the DSTC6 edition consisted of 3 parallel
tracks, different datasets were released for each track, such as, a
transaction dialogue dataset for the restaurant domain, two datasets that are
part of OpenSubtitles and Twitter datasets, and different chat-oriented
dialogue datasets with dialogue breakdown annotations in Japanese and English.

A more recent challenge that started in 2017 and continued into 2018, with its
second edition being the Conversational Intelligence Challenge
(ConvAI)191919http://convai.io/. This challenge, conducted under the scope of
NIPS, has the aim to unify the community around the task of building systems
capable of intelligent conversations. In its first edition teams were expected
to submit dialogue systems able to carry out intelligent and natural
conversations about specific news articles with humans. The aim of the task of
the second edition has been to model normal conversation when two
interlocutors meet for the first time, and get to know each other. The dataset
of this task consists of 10,981 dialogues with 164,356 utterances, and it is
available in the ParlAI framework mentioned above.

Finally, the Alexa Prize202020https://developer.amazon.com/alexaprize has
attracted mass media and research attention alike. This annual competition for
university teams is dedicated at accelerating the field of conversational AI
in the framework of the Alexa technology. The participants have to create
socialbots that can converse coherently and engagingly with humans on news
events and popular topics such as entertainment, sports, politics, technology
and fashion. Unfortunately, no datasets have been released.

##  7 Challenges and Future Trends

In the introduction, we stated that the goal of the dialogue evaluation is to
find methods that are automated, repeatable, are correlated to human
judgements, capable of differentiating among various dialogue strategies and
explain which features of the dialogue system contribute to its quality. The
main motivation behind this is the need to reduce the human evaluation effort
as much as possible, since human involvement creates high costs and is highly
time-consuming. In this survey, we presented the main concepts regarding
evaluation of dialogue systems and showcased the most important methods.
However, evaluation of dialogue systems is still an area of open research. In
this section, we summarize the current challenges and future trends that we
deem most important.

##### Automation.

The evaluation methods covered in this survey all achieve a certain degree of
automation. However, the automation is achieved with significant engineering
effort, or by loss of correlation to human judgements. Word-overlap metrics
(see Section 4.3.1), which are borrowed from the machine translation and
summarization community, are fully automated. However, they do not correlate
with human judgements on the turn level. On the other hand, BLEU becomes more
competitive when applied on the corpus-level or system-level (Galley et al.,
2015; Lowe et al., 2017a). More recent metrics such as $\Delta$BLEU and ADEM
(see Section 4.3.1) have significantly higher correlations to human judgements
while requiring a significant amount of human annotated data as well as
thorough engineering.

Task-oriented dialogue systems can be evaluated semi-automatically or even
fully automatically. These systems benefit from having a well-defined task,
where success can be measured. Thus, user satisfaction modelling (see Section
3.4.1) as well as user simulations (see Section 3.4.2) exploit this to
automate their evaluation. However, both approaches need a significant amount
of engineering and human annotation: user satisfaction modelling usually
requires prior annotation effort, which is followed by fitting a model that
predicts the judgements. In addition to this effort, the process has to be
potentially repeated for each new domain or new functionality that the
dialogue system incorporates. Although in some cases the model fitted on the
data for one dialogue system can be reused to predict another dialogue system,
this is not always possible.

On the other hand, user simulations require two steps: gathering data to
develop a first version of the simulation, and then building the actual user
simulation. The first step is only required for user simulations that are
based on training corpora (e.g. the neural user simulation). A significant
drawback is that the user simulation is only capable of simulating the
behaviour which is represented in the corpus or the rules. This means that it
cannot cover unseen behaviour well. Furthermore, the user simulation can
hardly be used to train or evaluate dialogue systems for other tasks or
domains.

Automation is thus achieved to a certain degree, but with significant
drawbacks. Hence, finding ways to facilitate the automation of evaluation
methods is clearly an open challenge.

##### High Quality Dialogues.

One major objective for a dialogue system is to deliver high quality
interactions with its users. However, it is often not clear how “high quality”
is defined in this context or how to measure it. For task oriented dialogue
systems, the mostly used definition of quality is often measured by means of
task success and number of dialogue turns (e.g. a reward of 20 for task-
success minus the number of turns needed to achieve the goal). However, this
definition is not applicable to conversational dialogue systems and it might
ignore other aspects of the interaction (e.g. frustration of the user). Thus,
the current trend is to let humans judge the _appropriateness_ of the system
utterances. However, the notion of appropriateness is highly subjective and
entails several finer-grained concepts (e.g. ability to maintain the topic,
the coherence of the utterance, the grammatical correctness of the utterance
itself, etc.). Currently, appropriateness is modelled by means of latent
representations (e.g. ADEM), which are derived again from annotated data.

Other aspects of quality concern the purpose of the dialogue system in
conjunction with the functionality of the system. For instance, Zhou et al.
(2018) define the purpose of their conversational dialogue system to build an
emotional bond between the dialogue system and the user. This goal differs
significantly from the task of training a medical student in the interaction
with patients. Both systems need to be evaluated with respect to their
particular goal. The ability to build an emotional bond can be evaluated by
means of the interaction length (longer interactions are an indicator of a
higher user engagement), whereas training (or e-learning) systems are usually
evaluated regarding their ability of selecting an appropriate utterance for
the given context.

The target audience plays an important role as well. Since quality is mainly a
subjective measure, different user groups prefer different types of
interactions. For instance, depending on the level of domain knowledge, novice
users prefer instructions that use less specialized wording, whereas domain
experts might prefer a more specialized vocabulary.

The notion of quality is thus dependent on a large amount of factors. The
evaluation needs to be adapted to take aspects such as the dialogue system’s
purpose, the target audience, and the dialogue system implementation itself
into account.

##### Lifelong Learning.

The notion of lifelong learning for machine learning systems has gained
traction recently. The main concept of lifelong learning is that a deployed
machine learning system continues to improve by interaction with its
environment (Chen et al., 2016). Lifelong learning for dialogue systems is
motivated by the fact that it is not possible to encounter all possible
situations during training, thus, a component that allows the dialogue system
to retrain itself and adapt its strategy during deployment seems the most
logical solution.

The evaluation step is critical in order to achieve lifelong learning. Since
the dialogue system relies on the ability to automatically find critical
dialogue states where it needs assistance, a module is needed which is able to
evaluate the ongoing dialogue. One step in this direction is done by Hancock
et al. (2019), who present a solution that relies on a satisfaction module
that is able of to classify the current dialogue state as either satisfactory
or not. If this module finds an unsatisfactory dialogue state, a feedback
module asks the user for feedback. The feedback data is then used to improve
the dialogue system.

The aspect of lifelong learning brings a large variety of novel challenges.
Firstly, the lifelong learning system requires a module that self-monitors its
behaviour and notices when a dialogue is going wrong. For this, the module
needs to rely on evaluation methods that work automatically, or at least semi-
automatically. The second challenge lies in the evaluation of the lifelong
learning system itself. The self-monitoring module as well as the adaptive
behaviour need to be evaluated. This brings a new dimension of complexity into
the evaluation procedure.

###  7.1 Conclusion

Evaluation is a critical task when developing and researching dialogue
systems. Over the past decades, many methods and concepts have been proposed.
These methods and concepts are related to the different requirements and
functionalities of the dialogue systems. These are subsequently dependent on
the current development stage of the dialogue system technology. Currently,
the trend is moving towards building end-to-end trainable dialogue systems
based on large amounts of data. These systems have different requirements for
evaluation than a finite state, machine-based system. Thus, the problem of
evaluation is evolving in tandem to the progress of the dialogue system
technology itself. This survey presents the current state-of-the-art research
in evaluation.

## Acknowledgements

We would like to thank Lina Scarborough for proofreading the manuscript. We
would also like to thank our anonymous reviewers for their valuable reviews
that helped improve the quality of this survey.

## Funding

This work was supported by the LIHLITH project in the framework of EU ERA-Net
CHIST-ERA; the Swiss National Science Foundation [20CH21_174237]; the Spanish
Research Agency [PCIN-2017-11, PCIN-2017-085/AEI]; Eneko Agirre and Arantxa
Otegi received the support of the UPV/EHU [grant GIU16/16]; Agence Nationale
pour la Recherche [ANR-17-CHR2-0001-03]

## Conflicts of Interest

There are no conflicts of interest to disclose.

## References

  * Adiwardana et al. (2020) Adiwardana D, Luong MT, So DR, Hall J, Fiedel N, Thoppilan R, Yang Z, Kulshreshtha A, Nemade G, Lu Y, et al. (2020) Towards a human-like open-domain chatbot. arXiv preprint arXiv:200109977
  * Ameixa and Coheur (2013) Ameixa D, Coheur L (2013) From subtitles to human interactions: introducing the SubTle Corpus. In: Technical report, Tech. rep., 2013
  * Austin (1962) Austin JL (1962) How to do things with words. William James Lectures, Oxford University Press
  * Banchs (2012) Banchs RE (2012) Movie-DiC: a Movie Dialogue Corpus for Research and Development. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, pp 203–207
  * Banchs and Li (2012) Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 Demonstrations, Jeju Island, Korea, pp 37–42
  * Bernardi and Kirschner (2010) Bernardi R, Kirschner M (2010) From artificial questions to real user interaction logs: Real challenges for Interactive Question Answering systems. In: Proceedings of Workshop on Web Logs and Question Answering (WLQA’10), Valletta, Malta
  * Black and Eskenazi (2009) Black AW, Eskenazi M (2009) The Spoken Dialogue Challenge. In: Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Stroudsburg, PA, USA, SIGDIAL ’09, pp 337–340
  * Black et al. (2011) Black AW, Burger S, Conkie A, Hastie H, Keizer S, Lemon O, Merigaud N, Parent G, Schubiner G, Thomson B, Williams JD, Yu K, Young S, Eskenazi M (2011) Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results. In: Proceedings of the SIGDIAL 2011 Conference: The 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Portland, Oregon, pp 2–7
  * Bordes et al. (2017) Bordes A, Boureau YL, Weston J (2017) Learning End-to-End Goal-Oriented Dialog. In: International Conference on Learning Representations (ICLR) 2017, Toulon, France
  * Bowman et al. (2016) Bowman SR, Vilnis L, Vinyals O, Dai A, Jozefowicz R, Bengio S (2016) Generating Sentences from a Continuous Space. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, Association for Computational Linguistics, Berlin, Germany, pp 10–21
  * Bruni and Fernandez (2017) Bruni E, Fernandez R (2017) Adversarial evaluation for open-domain dialogue generation. In: Proceedings of the SIGDIAL 2017 Conference: The 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, pp 284–288
  * Budzianowski et al. (2018) Budzianowski P, Wen TH, Tseng BH, Casanueva I, Stefan U, Osman R, Gašić M (2018) MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium
  * Byrne et al. (2019) Byrne B, Krishnamoorthi K, Sankar C, Neelakantan A, Goodrich B, Duckworth D, Yavuz S, Dubey A, Kim K, Cedilnik A (2019) Taskmaster-1: Toward a realistic and diverse dialog dataset. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, pp 4515–4524, DOI 10.18653/v1/D19-1459, URL https://doi.org/10.18653/v1/D19-1459
  * Campos et al. (2019) Campos JA, Otegi A, Soroa A, Deriu J, Cieliebak M, Agirre E (2019) Conversational QA for FAQs. In: 3rd Conversational AI: “Today’s Practice and Tomorrow’s Potential” workshop at NeurIPS 2019
  * Carletta (1996) Carletta J (1996) Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics 22(2):249–254
  * Charras et al. (2016) Charras F, Dubuisson Duplessis G, Letard V, Ligozat AL, Rosset S (2016) Comparing System-response Retrieval Models for Open-domain and Casual Conversational Agent. In: Workshop on Chatbots and Conversational Agent Technologies (WOCHAT)
  * Chen et al. (2017) Chen H, Liu X, Yin D, Tang J (2017) A Survey on Dialogue Systems: Recent Advances and New Frontiers. Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD) Explor Newsl 19(2):25–35
  * Chen et al. (2016) Chen Z, Liu B, Brachman R, Stone P, Rossi F (2016) Lifelong Machine Learning, 1st edn. Morgan & Claypool Publishers
  * Choi et al. (2018) Choi E, He H, Iyyer M, Yatskar M, Yih Wt, Choi Y, Liang P, Zettlemoyer L (2018) QuAC: Question Answering in Context. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Paris, France
  * Chotimongkol and Rudnicky (2001) Chotimongkol A, Rudnicky AI (2001) N-best speech hypotheses reordering using linear regression. In: Dalsgaard P, Lindberg B, Benner H, Tan Z (eds) EUROSPEECH 2001 Scandinavia, 7th European Conference on Speech Communication and Technology, 2nd INTERSPEECH Event, Aalborg, Denmark, September 3-7, 2001, ISCA, pp 1829–1832, URL http://www.isca-speech.org/archive/eurospeech_2001/e01_1829.html
  * Clark and Etzioni (2016) Clark P, Etzioni O (2016) My computer is an honor student but how intelligent is it? standardized tests as a measure of ai. AI Magazine 37(1):5–12, DOI 10.1609/aimag.v37i1.2636, URL https://www.aaai.org/ojs/index.php/aimagazine/article/view/2636
  * Colby (1981) Colby KM (1981) Modeling a paranoid mind. Behavioral and Brain Sciences 4(4):515–534
  * Cole (1999) Cole R (1999) Tools for research and education in speech science. In: Proceedings of the International Conference of Phonetic Sciences, San Francisco, USA, pp 1277–1280
  * Collins et al. (2019) Collins E, Rozanov N, Zhang B (2019) LIDA: lightweight interactive dialogue annotator. In: Padó S, Huang R (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 - System Demonstrations, Association for Computational Linguistics, pp 121–126, DOI 10.18653/v1/D19-3021
  * Danescu and Lee (2011) Danescu C, Lee L (2011) Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs. In: Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, Association for Computational Linguistics, pp 76–87
  * Dethlefs et al. (2013) Dethlefs N, Hastie H, Cuayáhuitl H, Lemon O (2013) Conditional random fields for responsive surface realisation using global features. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, pp 1254–1263
  * DeVault et al. (2011) DeVault D, Leuski A, Sagae K (2011) Toward Learning and Evaluation of Dialogue Policies with Text Examples. In: Proceedings of the SIGDIAL 2011 Conference: The 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Stroudsburg, PA, USA, pp 39–48
  * Devlin et al. (2019) Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171–4186, DOI 10.18653/v1/N19-1423, URL https://www.aclweb.org/anthology/N19-1423
  * Diefenbach et al. (2018) Diefenbach D, Lopez V, Singh K, Maret P (2018) Core Techniques of Question Answering Systems over Knowledge Bases: A Survey. Knowledge and Information Systems 55(3):529–569
  * Do et al. (2017) Do P, Nguyen H, Tran C, Nguyen M, Nguyen M (2017) Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network. arXiv preprint arXiv:170305320 abs/1703.05320
  * Dubuisson Duplessis et al. (2016) Dubuisson Duplessis G, Letard V, Ligozat AL, Rosset S (2016) Purely Corpus-based Automatic Conversation Authoring. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation, European Language Resources Association (ELRA), Paris, France, LREC 2016, URL http://www.lrec-conf.org/proceedings/lrec2016/pdf/396_Paper.pdf
  * Dubuisson Duplessis et al. (2017) Dubuisson Duplessis G, Charras F, Letard V, Ligozat AL, Rosset S (2017) Utterance Retrieval based on Recurrent Surface Text Patterns. In: European Conference on Information Retrieval, Aberdeen, Scotland UK, ECIR 2017, URL https://hal.archives-ouvertes.fr/hal-01436052/document
  * Dušek et al. (2020) Dušek O, Novikova J, Rieser V (2020) Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech and Language 59:123–156, DOI 10.1016/j.csl.2019.06.009
  * Dušek and Jurcicek (2016) Dušek O, Jurcicek F (2016) Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, ACL 2016, pp 45–51
  * Engel et al. (2005) Engel Y, Mannor S, Meir R (2005) Reinforcement Learning with Gaussian Processes. In: Proceedings of the 22nd International Conference on Machine Learning, ACM, Bonn, Germany, ICML ’05, pp 201–208
  * Engelbrecht et al. (2008) Engelbrecht KP, Möller S, Schleicher R, Wechsung I (2008) Analysis of paradise models for individual users of a spoken dialog system. In: Electronic Speech Signal Processing, Proceedings of the 19th Conference, Frankfurt am Main, Germany, ESSV 2008, pp 86–93, URL https://d-nb.info/990359174/04
  * Engelbrecht et al. (2009a) Engelbrecht KP, Gödde F, Hartard F, Ketabdar H, Möller S (2009a) Modeling User Satisfaction with Hidden Markov Model. In: Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, London, UK, SIGDIAL ’09, pp 170–177, URL http://dl.acm.org/citation.cfm?id=1708376.1708402
  * Engelbrecht et al. (2009b) Engelbrecht KP, Quade M, Möller S (2009b) Analysis of a New Simulation Approach to Dialog System Evaluation. Speech Communication 51(12):1234–1252, URL http://dx.doi.org/10.1016/j.specom.2009.06.007
  * Eric et al. (2017) Eric M, Krishnan L, Charette F, Manning CD (2017) Key-Value Retrieval Networks for Task-Oriented Dialogue. In: Proceedings of the SIGDIAL 2017 Conference: The 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrücken, Germany, SIGDIAL’17, pp 37–49, DOI 10.18653/v1/W17-5506, URL http://aclweb.org/anthology/W17-5506
  * Evanini et al. (2008) Evanini K, Hunter P, Liscombe J, Suendermann D, Dayanidhi K, Pieraccini R (2008) Caller Experience: A method for evaluating dialog systems and its automatic prediction. In: 2008 IEEE Spoken Language Technology Workshop, Goa, India, pp 129–132, DOI 10.1109/SLT.2008.4777857
  * Fader et al. (2013) Fader A, Zettlemoyer L, Etzioni O (2013) Paraphrase-driven learning for open question answering. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Sofia, Bulgaria, pp 1608–1618, URL https://www.aclweb.org/anthology/P13-1158
  * Fleiss (1971) Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychological bulletin 76(5):378–382, DOI http://dx.doi.org/10.1037/h0031619
  * Furlanello et al. (2018) Furlanello T, Lipton ZC, Tschannen M, Itti L, Anandkumar A (2018) Born-again neural networks. In: Dy JG, Krause A (eds) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, PMLR, Proceedings of Machine Learning Research, vol 80, pp 1602–1611, URL http://proceedings.mlr.press/v80/furlanello18a.html
  * Gašić et al. (2011) Gašić M, Jurčíček F, Thomson B, Yu K, Young S (2011) On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In: 2011 IEEE Workshop on Automatic Speech Recognition Understanding, pp 312–317, DOI 10.1109/ASRU.2011.6163950
  * Galley et al. (2015) Galley M, Brockett C, Sordoni A, Ji Y, Auli M, Quirk C, Mitchell M, Gao J, Dolan B (2015) deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, ACL 2015, pp 445–450, URL http://www.aclweb.org/anthology/P15-2073
  * Gandhe and Traum (2016) Gandhe S, Traum D (2016) A Semi-automated Evaluation Metric for Dialogue Model Coherence, Springer International Publishing, Cham, pp 217–225. URL https://doi.org/10.1007/978-3-319-21834-2_19
  * Gandhe and Traum (2013) Gandhe S, Traum DR (2013) Surface text based dialogue models for virtual humans. In: Proceedings of the SIGDIAL 2013 Conference: The 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Metz, France, SIGDIAL 2013
  * Gandhe et al. (2009) Gandhe S, Whitman N, Traum D, Artstein R (2009) An integrated authoring tool for tactical questioning dialogue systems. In: 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, Pasadena Conference Center, California, USA., pp 10–18
  * Gasic et al. (2013) Gasic M, Breslin C, Henderson M, Kim D, Szummer M, Thomson B, Tsiakoulis P, Young S (2013) POMDP-based dialogue manager adaptation to extended domains. In: Proceedings of the SIGDIAL 2013 Conference: The 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Metz, France, SIGDIAL 2013, pp 214–222, URL http://www.aclweb.org/anthology/W13-4035
  * Gasic et al. (2014) Gasic M, Kim D, Tsiakoulis P, Breslin C, Henderson M, Szummer M, Thomson B, Young SJ (2014) Incremental on-line adaptation of POMDP-based dialogue managers to extended domains. In: 15th Annual Conference of the International Speech Communication Association, Singapore, INTERSPEECH 2014, pp 140–144, URL http://www.isca-speech.org/archive/interspeech_2014/i14_0140.html
  * Ghazvininejad et al. (2018) Ghazvininejad M, Brockett C, Chang MW, Dolan B, Gao J, Yih Wt, Galley M (2018) A knowledge-grounded neural conversation model. In: Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, AAAI 2018, pp 5110–5117
  * Godfrey et al. (1992) Godfrey JJ, Holliman EC, McDaniel J (1992) SWITCHBOARD: telephone speech corpus for research and development. In: [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, San Francisco, CA, USA, vol 1, pp 517–520, DOI 10.1109/ICASSP.1992.225858
  * Goodfellow et al. (2014) Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in Neural Information Processing Systems 27, NIPS 27, Curran Associates, Inc., pp 2672–2680, URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
  * Gunasekara et al. (2019) Gunasekara C, Kummerfeld JK, Polymenakos L, Lasecki WS (2019) DSTC7 Task 1: Noetic End-to-End Response Selection. In: 7th Edition of the Dialog System Technology Challenges at AAAI 2019, URL http://workshop.colips.org/dstc7/papers/dstc7_task1_final_report.pdf
  * Guo et al. (2014) Guo D, Tur G, Yih Wt, Zweig G (2014) Joint semantic utterance classification and slot filling with recursive neural networks. In: 2014 IEEE Spoken Language Technology Workshop (SLT), South Lake Tahoe, California, USA, IEEE 2014, pp 554–559, URL https://www.microsoft.com/en-us/research/wp-content/uploads/2014/12/SLT2014-daniel.pdf
  * Guo et al. (2018) Guo F, Metallinou A, Khatri C, Raju A, Venkatesh A, Ram A (2018) Topic-based evaluation for conversational bots. arXiv preprint arXiv:180103622
  * Gupta et al. (2019) Gupta P, Mehri S, Zhao T, Pavel A, Eskenazi M, Bigham JP (2019) Investigating evaluation of open-domain dialogue systems with human generated multiple references. 1907.10568
  * Hahn et al. (2010) Hahn S, Dinarelli M, Raymond C, Lefèvre F, Lehen P, De Mori R, Moschitti A, Ney H, Riccardi G (2010) Comparing Stochastic Approaches to Spoken Language Understanding in Multiple Languages. IEEE Transactions on Audio, Speech and Language Processing (TASLP) 16:1569–1583, URL https://hal.archives-ouvertes.fr/file/index/docid/746965/filename/plugin-05639034.pdf
  * Hancock et al. (2019) Hancock B, Bordes A, Mazare PE, Weston J (2019) Learning from Dialogue after Deployment: Feed Yourself, Chatbot! In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, ACL 2019, pp 3667–3684, URL https://www.aclweb.org/anthology/P19-1358
  * Hara (2010) Hara S (2010) Estimation method of user satisfaction using N-gram-based dialog history model for spoken dialog system. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation, Valletta, Malta, LREC’10, pp 78–83, URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/579_Paper.pdf
  * Henderson et al. (2013a) Henderson M, Thomson B, Williams J (2013a) Dialog State Tracking Challenge 2 & 3\. Tech. rep.
  * Henderson et al. (2013b) Henderson M, Thomson B, Young S (2013b) Deep neural network approach for the dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference: The 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Metz, France, pp 467–471, URL http://www.aclweb.org/anthology/W13-4073
  * Henderson et al. (2014) Henderson M, Thomson B, Williams J (2014) The Second Dialog State Tracking Challenge. In: Proceedings of the SIGDIAL 2014 Conference: The 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Philadelphia, PA, USA, pp 263––272, URL https://www.microsoft.com/en-us/research/publication/the-second-dialog-state-tracking-challenge/
  * Higashinaka et al. (2010) Higashinaka R, Minami Y, Dohsaka K, Meguro T (2010) Issues in Predicting User Satisfaction Transitions in Dialogues: Individual Differences, Evaluation Criteria, and Prediction Models. In: Lee GG, Mariani J, Minker W, Nakamura S (eds) Second International Workshop on Spoken Dialogue Systems Technology: Spoken Dialogue Systems for Ambient Environments, Springer Berlin Heidelberg, Gotemba, Shizuoka, Japan, IWSDS 2010, pp 48–60
  * Hirschman et al. (1990) Hirschman L, Dahl DA, McKay DP, Norton LM, Linebarger MC (1990) Beyond class A: A proposal for automatic evaluation of discourse. In: In Proceedings of the Speech and Natural Language Workshop, Hidden Valley, Pennsylvania, USA, HLT, pp 109–113
  * Hochreiter and Schmidhuber (1997) Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Computation pp 1735–1780
  * Hu et al. (2017) Hu Z, Yang Z, Liang X, Salakhutdinov R, Xing EP (2017) Toward Controlled Generation of Text. In: Proceedings of the 34th International Conference on Machine Learning, International Convention Centre, Sydney, Australia, ICML, pp 1587–1596, URL http://proceedings.mlr.press/v70/hu17e.html
  * Huang et al. (2019) Huang HY, Choi E, tau Yih W (2019) FlowQA: Grasping flow in history for conversational machine comprehension. In: International Conference on Learning Representations, URL https://openreview.net/forum?id=ByftGnR9KX
  * Iyyer et al. (2017a) Iyyer M, Yih Wt, Chang MW (2017a) Search-based Neural Structured Learning for Sequential Question Answering. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, ACL, pp 1821–1831, DOI 10.18653/v1/P17-1167, URL http://www.aclweb.org/anthology/P17-1167
  * Iyyer et al. (2017b) Iyyer M, Yih Wt, Chang MW (2017b) Search-based neural structured learning for sequential question answering. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, pp 1821–1831, DOI 10.18653/v1/P17-1167, URL https://www.aclweb.org/anthology/P17-1167
  * Joshi et al. (2017) Joshi M, Choi E, Weld D, Zettlemoyer L (2017) TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, pp 1601–1611, DOI 10.18653/v1/P17-1147, URL https://www.aclweb.org/anthology/P17-1147
  * Ju et al. (2019) Ju Y, Zhao F, Chen S, Zheng B, Yang X, Liu Y (2019) Technical report on conversational question answering. 1909.10772
  * Jurafsky and Martin (2017) Jurafsky D, Martin JH (2017) Speech and Language Processing, draft of 3rd edition edn, chap Dialog Systems and Chatbots
  * Jurcícek et al. (2011) Jurcícek F, Keizer S, Gasic M, Mairesse F, Thomson B, Yu K, Young SJ (2011) Real User Evaluation of Spoken Dialogue Systems Using Amazon Mechanical Turk. In: 12th Annual Conference of the International Speech Communication Association, Florence, Italy, INTERSPEECH, pp 3061–3064
  * Kannan and Vinyals (2016) Kannan A, Vinyals O (2016) Adversarial evaluation of dialogue models. Workshop on Adversarial Training at Neural Information Processing Systems 2016
  * Kelly et al. (2009) Kelly D, Kantor PB, Morse EL, Scholtz J, Sun Y (2009) Questionnaires for eliciting evaluation data from users of interactive question answering systems. Natural Language Engineering 15(1):119–141
  * Kenny et al. (2009) Kenny PG, Parsons TD, Rizzo AA (2009) Human Computer Interaction in Virtual Standardized Patient Systems. In: Proceedings of the 13th International Conference on Human-Computer Interaction. Part IV: Interacting in Various Application Domains, Springer-Verlag, Berlin, Heidelberg, pp 514–523, URL http://dx.doi.org/10.1007/978-3-642-02583-9_56
  * Kim et al. (2016) Kim S, D’Haro LF, Banchs RE, Williams JD, Henderson M, Yoshino K (2016) The fifth dialog state tracking challenge. In: 2016 IEEE Spoken Language Technology Workshop (SLT), pp 511–517, DOI 10.1109/SLT.2016.7846311
  * Kočiský et al. (2018) Kočiský T, Schwarz J, Blunsom P, Dyer C, Hermann KM, Melis G, Grefenstette E (2018) The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics 6:317–328, DOI 10.1162/tacl˙a˙00023, URL https://www.aclweb.org/anthology/Q18-1023, 1712.07040
  * Kolomiyets and Moens (2011) Kolomiyets O, Moens MF (2011) A Survey on Question Answering Technology from an Information Retrieval Perspective. Information Sciences 181(24):5412–5434, DOI 10.1016/j.ins.2011.07.047, URL https://doi.org/10.1016/j.ins.2011.07.047
  * Konstantinova and Orasan (2013) Konstantinova N, Orasan C (2013) Interactive Question Answering. In: Emerging Applications of Natural Language Processing: Concepts and New Research, pp 149–169
  * Kreyssig et al. (2018) Kreyssig F, Casanueva I, Budzianowski P, Gasic M (2018) Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems. arXiv preprint arXiv:180506966
  * Lai et al. (2017) Lai G, Xie Q, Liu H, Yang Y, Hovy E (2017) RACE: Large-scale ReAding Comprehension Dataset From Examinations. EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings pp 785–794, URL http://arxiv.org/abs/1704.04683, 1704.04683
  * Lamel et al. (2000) Lamel L, Rosset S, Gauvain JL, Bennacef S, Garnier-Rizet M, Prouts B (2000) The limsi arise system. Speech Communication 31(4):339–353
  * Lan et al. (2019) Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: A lite bert for self-supervised learning of language representations. 1909.11942
  * Larson et al. (2019) Larson S, Mahendran A, Peper JJ, Clarke C, Lee A, Hill P, Kummerfeld JK, Leach K, Laurenzano MA, Tang L, Mars J (2019) An evaluation dataset for intent classification and out-of-scope prediction. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, pp 1311–1316, DOI 10.18653/v1/D19-1131, URL https://www.aclweb.org/anthology/D19-1131
  * Lavie and Denkowski (2009) Lavie A, Denkowski MJ (2009) The Meteor Metric for Automatic Evaluation of Machine Translation. Machine Translation 23(2-3):105–115, URL http://dx.doi.org/10.1007/s10590-009-9059-4
  * Lee et al. (2009) Lee C, Jung S, Kim S, Lee GG (2009) Example-based dialog modeling for practical multi-domain dialog system. Speech Communication 51(5):466–484
  * Lee et al. (2019) Lee S, Schulz H, Atkinson A, Gao J, Suleman K, Asri LE, Adada M, Huang M, Sharma S, Tay W, Li X (2019) Multi-Domain Task-Completion Dialog Challenge. Dialog System Technology Challenges 8
  * Leech (1993) Leech GN (1993) 100 million words of english: the british national corpus (BNC). English Today 28:9–15, DOI doi:10.1017/S0266078400006854
  * Lemon and Pietquin (2012) Lemon O, Pietquin O (2012) Data-Driven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces. Springer Publishing Company, Incorporated
  * Levenshtein (1966) Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady 10(8):707–710, doklady Akademii Nauk SSSR, V163 No4 845-848 1965
  * Levin et al. (1998) Levin E, Pieraccini R, Eckert W (1998) Using Markov decision process for learning dialogue strategies. In: Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, Seattle, WA, USA, ICASSP, vol 1, pp 201–204, DOI 10.1109/ICASSP.1998.674402
  * Li et al. (2017a) Li H, Min MR, Ge Y, Kadav A (2017a) A Context-aware Attention Network for Interactive Question Answering. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, NY, USA, KDD ’17, pp 927–935, URL http://doi.acm.org/10.1145/3097983.3098115
  * Li et al. (2016a) Li J, Galley M, Brockett C, Gao J, Dolan B (2016a) A Diversity-Promoting Objective Function for Neural Conversation Models. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, San Diego, California, pp 110–119, URL http://www.aclweb.org/anthology/N16-1014
  * Li et al. (2016b) Li J, Monroe W, Ritter A, Jurafsky D, Galley M, Gao J (2016b) Deep Reinforcement Learning for Dialogue Generation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, EMNLP ’16, pp 1192–1202, DOI 10.18653/v1/D16-1127, URL http://www.aclweb.org/anthology/D16-1127
  * Li et al. (2017b) Li X, Chen YN, Li L, Gao J, Celikyilmaz A (2017b) End-to-End Task-Completion Neural Dialogue Systems. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan, IJCNLP, pp 733–743, URL http://aclweb.org/anthology/I17-1074
  * Li et al. (2017c) Li Y, Su H, Shen X, Li W, Cao Z, Niu S (2017c) DailyDialog: A manually labelled multi-turn dialogue dataset. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan, pp 986–995, URL https://www.aclweb.org/anthology/I17-1099
  * Lin (2004) Lin CY (2004) ROUGE: A Package for Automatic Evaluation of Summaries. In: Marie-Francine Moens SS (ed) Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, Association for Computational Linguistics, Barcelona, Spain, pp 74–81, URL http://www.aclweb.org/anthology/W04-1013
  * Liu et al. (2018) Liu B, Tür G, Hakkani-Tür D, Shah P, Heck L (2018) Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, USA, NAACL-HLT ’18, pp 2060–2069, URL http://aclweb.org/anthology/N18-1187
  * Liu et al. (2016) Liu CW, Lowe R, Serban I, Noseworthy M, Charlin L, Pineau J (2016) How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, pp 2122–2132, DOI 10.18653/v1/D16-1230, URL http://www.aclweb.org/anthology/D16-1230
  * Liu et al. (2019) Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: A robustly optimized bert pretraining approach. 1907.11692
  * Lowe et al. (2016) Lowe R, Serban IV, Noseworthy M, Charlin L, Pineau J (2016) On the Evaluation of Dialogue Systems with Next Utterance Classification. In: Proceedings of the SIGDIAL 2016 Conference: The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Los Angeles, CA, USA, pp 264–269, URL http://www.aclweb.org/anthology/W16-3634
  * Lowe et al. (2017a) Lowe R, Noseworthy M, Serban IV, Angelard-Gontier N, Bengio Y, Pineau J (2017a) Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, ACL ’17, pp 1116–1126, DOI 10.18653/v1/P17-1103, URL http://www.aclweb.org/anthology/P17-1103
  * Lowe et al. (2017b) Lowe R, Pow N, Serban IV, Charlin L, Liu CW, Pineau J (2017b) Training End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus. Dialogue & Discourse 8(1):31–65, URL http://dad.uni-bielefeld.de/index.php/dad/article/view/3698
  * Lowe et al. (2015) Lowe RJ, Pow N, Serban I, Pineau J (2015) The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In: Proceedings of the SIGDIAL 2015 Conference: The 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Prague, Czech Republic, pp 285–294, URL http://aclweb.org/anthology/W15-4640
  * Lu (2012) Lu X (2012) The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal 96(2):190–208, DOI 10.1111/j.1540-4781.2011.01232“˙1.x
  * Mairesse et al. (2010) Mairesse F, Gašić M, Jurčíček F, Keizer S, Thomson B, Yu K, Young S (2010) Phrase-based statistical language generation using graphical models and active learning. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, ACL ’10, pp 1552–1561, URL https://www.aclweb.org/anthology/P10-1157
  * Mazza et al. (2018) Mazza R, Ambrosini L, Catenazzi N, Vanini S, Tuggener D, Tavarnesi G (2018) Behavioural Simulator For Professional Training Based On Natural Language Interaction. In: 10th International Conference on Education and New Learning Technologies, Palma, Mallorca, Spain, EDULEARN18, pp 3204–3214, URL http://repository.supsi.ch/9776/1/edulearn18-paper-lifelike.pdf
  * McTear et al. (2005) McTear M, O’Neill I, Hanna P, Liu X (2005) Handling errors and determining confirmation strategies—an object-based approach. Speech Communication 45(3):249–269, URL https://www.aclweb.org/anthology/N16-1086
  * Mei et al. (2016) Mei H, Bansal M, Walter MR (2016) What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, NAACL-HLT, pp 720–730, URL https://www.aclweb.org/anthology/N16-1086
  * Mesnil et al. (2015) Mesnil G, Dauphin Y, Yao K, Bengio Y, Deng L, Hakkani-Tur D, He X, Heck L, Tur G, Yu D, Zweig G (2015) Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding. IEEE/ACM Trans Audio, Speech and Language Processing 23(3):530–539, URL http://www.iro.umontreal.ca/~lisa/pointeurs/taslp_RNNSLU_final_doubleColumn.pdf
  * Metallinou et al. (2013) Metallinou A, Bohus D, Williams J (2013) Discriminative state tracking for spoken dialog systems. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Sofia, Bulgaria, pp 466–475, URL http://www.aclweb.org/anthology/P13-1046
  * Miller et al. (2017) Miller A, Feng W, Batra D, Bordes A, Fisch A, Lu J, Parikh D, Weston J (2017) ParlAI: A Dialog Research Software Platform. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP ’17, pp 79–84, URL https://www.aclweb.org/anthology/D17-2014
  * Mishra and Jain (2016) Mishra A, Jain SK (2016) A Survey on Question Answering Systems with Classification. Journal of King Saud University - Computer and Information Sciences 28(3):345–361, URL https://doi.org/10.1016/j.jksuci.2014.10.007
  * Möller et al. (2004) Möller S, Krebber J, Raake A, Smeele P, Rajman M, Melichar M, Pallotta V, Tsakou G, Kladis B, Vovos A, Hoonhout J, Schuchardt D, Fakotakis N, Ganchev T, Potamitis I (2004) INSPIRE: Evaluation of a smart-home system for infotainment management and device control. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), European Language Resources Association (ELRA), Lisbon, Portugal, URL http://www.lrec-conf.org/proceedings/lrec2004/pdf/12.pdf
  * Möller et al. (2006) Möller S, Englert R, Engelbrecht K, Hafner V, Jameson A, Oulasvirta A, Raake A, Reithinger N (2006) MeMo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In: Ninth International Conference on Spoken Language Processing, INTERSPEECH - ICSLP 2006, pp 1786–1789, URL https://www.isca-speech.org/archive/interspeech_2006/i06_1131.html
  * Mrkšić et al. (2017) Mrkšić N, Ó Séaghdha D, Wen TH, Thomson B, Young S (2017) Neural Belief Tracker: Data-Driven Dialogue State Tracking. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, ACL ’17, pp 1777–1788, DOI 10.18653/v1/P17-1163, URL http://aclweb.org/anthology/P17-1163
  * Novikova et al. (2017) Novikova J, Dušek O, Rieser V (2017) The E2E Dataset: New Challenges for End-to-End Generation. In: Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrücken, Germany, SIGDIAL ’17, pp 201––206, URL https://www.aclweb.org/anthology/W17-5525, arXiv:1706.09254
  * Paek (2006) Paek T (2006) Reinforcement learning for spoken dialogue systems: Comparing strengths and weaknesses for practical deployment. In: Proc. Dialog-on-Dialog Workshop, Interspeech, Pittsburgh, PA, USA, URL http://www.ling.helsinki.fi/~kjokinen/ICSLP06-DoD/Programme/PaekTim.pdf
  * Papineni et al. (2002) Papineni K, Roukos S, Ward T, Zhu WJ (2002) Bleu: a Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, ACL ’02, pp 311–318, URL http://www.aclweb.org/anthology/P02-1040
  * Peñas et al. (2012) Peñas A, Magnini B, Forner P, Sutcliffe R, Rodrigo Á, Giampiccolo D (2012) Question answering at the cross-language evaluation forum 2003-2010. Language Resources and Evaluation 46(2):177–217, DOI 10.1007/s10579-012-9177-0, URL https://doi.org/10.1007/s10579-012-9177-0
  * Perez et al. (2017) Perez J, Boureau YL, Bordes A (2017) Dialog System and Technology Challenge 6 Overview of Track 1 - End-to-End Goal-Oriented Dialog learning. Tech. rep.
  * Peskov et al. (2019) Peskov D, Clarke N, Krone J, Fodor B, Zhang Y, Youssef A, Diab M (2019) Multi-Domain Goal-Oriented Dialogues (MultiDoGO): Strategies toward Curating and Annotating Large Scale Dialogue Data. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, pp 4526–4536, DOI 10.18653/v1/D19-1460, URL https://www.aclweb.org/anthology/D19-1460
  * Pietquin and Hastie (2013) Pietquin O, Hastie H (2013) A survey on metrics for the evaluation of user simulations. The Knowledge Engineering Review 28(1):59–73, DOI 10.1017/S0269888912000343
  * Powers (2012) Powers DMW (2012) The Problem with Kappa. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, EACL ’13, pp 345–355, URL http://www.aclweb.org/anthology/E12-1035
  * Qu et al. (2018) Qu C, Yang L, Croft WB, Trippas JR, Zhang Y, Qiu M (2018) Analyzing and Characterizing User Intent in Information-seeking Conversations. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA, SIGIR 2018, pp 989–992, DOI 10.1145/3209978.3210124, URL http://doi.acm.org/10.1145/3209978.3210124
  * Qu et al. (2019) Qu C, Yang L, Qiu M, Zhang Y, Chen C, Croft WB, Iyyer M (2019) Attentive history selection for conversational question answering. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA, CIKM ’19, p 1391–1400, DOI 10.1145/3357384.3357905, URL https://doi.org/10.1145/3357384.3357905
  * Qu and Green (2002) Qu Y, Green N (2002) A constraint-based approach for cooperative information-seeking dialogue. In: Proceedings of the International Natural Language Generation Conference, Harriman, New York, USA, INLG, pp 136–143
  * Rajpurkar et al. (2016) Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, pp 2383–2392, DOI 10.18653/v1/D16-1264, URL https://www.aclweb.org/anthology/D16-1264
  * Rajpurkar et al. (2018) Rajpurkar P, Jia R, Liang P (2018) Know What You Don’t Know: Unanswerable Questions for SQuAD. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Melbourne, Australia, pp 784–789, DOI 10.18653/v1/P18-2124, URL https://www.aclweb.org/anthology/P18-2124, 1806.03822
  * Rambow et al. (2001) Rambow O, Bangalore S, Walker M (2001) Natural Language Generation in Dialog Systems. In: Proceedings of the first international conference on Human language technology (HLT) research, San Diego, USA, pp 67–73
  * Rastogi et al. (2019) Rastogi A, Zang X, Sunkara S, Gupta R, Khaitan P (2019) Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset. arXiv preprint arXiv:190905855
  * Reddy et al. (2018) Reddy S, Chen D, Manning CD (2018) CoQA: A Conversational Question Answering Challenge. Transactions of the Association for Computational Linguistics 7:249–266, URL https://www.aclweb.org/anthology/Q19-1016"
  * Richardson et al. (2013) Richardson M, Burges CJ, Renshaw E (2013) MCTest: A challenge dataset for the open-domain machine comprehension of text. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Seattle, Washington, USA, pp 193–203, URL https://www.aclweb.org/anthology/D13-1020
  * Rieser and Lemon (2009) Rieser V, Lemon O (2009) Does this list contain what you were searching for? Learning adaptive dialogue strategies for interactive question answering. Natural Language Engineering 15(1):55––72, DOI 10.1017/S1351324908004907
  * Ritter et al. (2010) Ritter A, Cherry C, Dolan B (2010) Unsupervised Modeling of Twitter Conversations. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Stroudsburg, PA, USA, HLT ’10, pp 172–180, URL http://dl.acm.org/citation.cfm?id=1857999.1858019
  * Ritter et al. (2011) Ritter A, Cherry C, Dolan WB (2011) Data-driven Response Generation in Social Media. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK., EMNLP ’11, pp 583–593, URL http://dl.acm.org/citation.cfm?id=2145432.2145500
  * Rodrigo et al. (2018) Rodrigo A, Peñas A, Miyao Y, Kando N (2018) Do systems pass university entrance exams? Information Processing & Management 54(4):564–575, DOI 10.1016/J.IPM.2018.03.002, URL https://www.sciencedirect.com/science/article/abs/pii/S0306457317305344
  * Rogers et al. (2020a) Rogers A, Kovaleva O, Downey M, Rumshisky A (2020a) Getting closer to ai complete question answering: A set of prerequisite real tasks
  * Rogers et al. (2020b) Rogers A, Kovaleva O, Rumshisky A (2020b) A Primer in BERTology: What we know about how BERT works URL http://arxiv.org/abs/2002.12327, 2002.12327
  * Saha et al. (2018) Saha A, Pahuja V, Khapra MM, Sankaranarayanan K, Chandar S (2018) Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In: McIlraith SA, Weinberger KQ (eds) Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, AAAI Press, pp 705–713, URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17181
  * Sai et al. (2019) Sai AB, Gupta MD, Khapra MM, Srinivasan M (2019) Re-evaluating adem: A deeper look at scoring dialogue responses. In: Proceedings of the thirty-third AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA, AAAI’19, vol 33, pp 6220–6227, URL https://aaai.org/ojs/index.php/AAAI/article/view/4581
  * Sarrouti and Ouatik El Alaoui (2017) Sarrouti M, Ouatik El Alaoui S (2017) A Passage Retrieval Method Based on Probabilistic Information Retrieval Model and UMLS Concepts in Biomedical Question Answering. J of Biomedical Informatics 68(C):96–103, DOI 10.1016/j.jbi.2017.03.001, URL https://doi.org/10.1016/j.jbi.2017.03.001
  * Schatzmann et al. (2006) Schatzmann J, Weilhammer K, Stuttle M, Young S (2006) A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review 21(2):97–126, URL https://search.proquest.com/docview/217517188?accountid=15920
  * Schatzmann et al. (2007) Schatzmann J, Thomson B, Weilhammer K, Ye H, Young S (2007) Agenda-based User Simulation for Bootstrapping a POMDP Dialogue System. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, Rochester, New York, NAACL-Short ’07, pp 149–152, URL http://dl.acm.org/citation.cfm?id=1614108.1614146
  * Schatztnann et al. (2005) Schatztnann J, Stuttle MN, Weilhammer K, Young S (2005) Effects of the user model on simulation-based learning of dialogue strategies. In: IEEE Workshop on Automatic Speech Recognition and Understanding, San Juan, Puerto Rico, ASRU, pp 220–225, URL https://ieeexplore.ieee.org/document/1566539
  * Schmitt and Ultes (2015) Schmitt A, Ultes S (2015) Interaction Quality: Assessing the quality of ongoing spoken dialog interaction by experts—And how it relates to user satisfaction. Speech Communication 74:12–36, URL https://dl.acm.org/doi/10.1016/j.specom.2015.06.003
  * Schmitt et al. (2012) Schmitt A, Ultes S, Minker W (2012) A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System. In: Chair) NCC, Choukri K, Declerck T, Doğan MU, Maegaard B, Mariani J, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), European Language Resources Association (ELRA), Istanbul, Turkey
  * van Schooten et al. (2007) van Schooten B, Rosset S, Galibert O, Max A, op den Akker R, Illouz G (2007) Handling speech input in the Ritel QA dialogue system. In: 8th Annual Conference of the International Speech Communication Association, Antwerp, Belgium, INTERSPEECH 2007, pp 126–129, URL https://www.isca-speech.org/archive/interspeech_2007/i07_0126.html
  * Schrading (2015) Schrading JN (2015) Analyzing domestic abuse using natural language processing on social media data. Master’s thesis, Rochester Institute of Technology, URL http://scholarworks.rit.edu/theses
  * Searle (1969) Searle JR (1969) Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge, London
  * Searle (1975) Searle JR (1975) Indirect speech acts. In: Cole P, Morgan J (eds) Syntax and Semantics 3: Speech Acts, Academic Press, New York, pp 59–82
  * Semeniuta et al. (2017) Semeniuta S, Severyn A, Barth E (2017) A Hybrid Convolutional Variational Autoencoder for Text Generation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, EMNLP, pp 627–637, URL https://www.aclweb.org/anthology/D17-1066
  * Serban et al. (2016) Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J (2016) Building End-to-end Dialogue Systems Using Generative Hierarchical Neural Network Models. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI Press, Phoenix, Arizona, USA, AAAI’16, pp 3776–3783, URL http://dl.acm.org/citation.cfm?id=3016387.3016435
  * Serban et al. (2017a) Serban IV, Klinger T, Tesauro G, Talamadupula K, Zhou B, Bengio Y, Courville AC (2017a) Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California, USA, AAAI ’17, pp 3288–3294, URL http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14571
  * Serban et al. (2017b) Serban IV, Sankar C, Germain M, Zhang S, Lin Z, Subramanian S, Kim T, Pieper M, Chandar S, Ke NR, et al. (2017b) A deep reinforcement learning chatbot. arXiv preprint arXiv:170902349
  * Serban et al. (2017c) Serban IV, Sordoni A, Lowe R, Charlin L, Pineau J, Courville A, Bengio Y (2017c) A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California USA, AAAI’17, pp 3295–3301, URL https://dl.acm.org/doi/10.5555/3298023.3298047
  * Serban et al. (2018) Serban IV, Lowe R, Henderson P, Charlin L, Pineau J (2018) A Survey of Available Corpora for Building Data-Driven Dialogue Systems: The Journal Version. Dialogue & Discourse 1(9)
  * Shang et al. (2015) Shang L, Lu Z, Li H (2015) Neural Responding Machine for Short-Text Conversation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China, ACL - IJCNLP ’15, pp 1577–1586, URL http://www.aclweb.org/anthology/P15-1152
  * Singh et al. (2000) Singh SP, Kearns MJ, Litman DJ, Walker MA (2000) Reinforcement Learning for Spoken Dialogue Systems. In: Solla SA, Leen TK, Müller K (eds) Advances in Neural Information Processing Systems 12, MIT Press, pp 956–962, URL http://papers.nips.cc/paper/1775-reinforcement-learning-for-spoken-dialogue-systems.pdf
  * Sordoni et al. (2015) Sordoni A, Galley M, Auli M, Brockett C, Ji Y, Mitchell M, Nie JY, Gao J, Dolan B (2015) A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Beijing, China, ACL - IJCNLP ’15, pp 196–205, DOI 10.3115/v1/N15-1020, URL http://www.aclweb.org/anthology/N15-1020
  * Stent et al. (2004) Stent A, Prasad R, Walker M (2004) Trainable Sentence Planning for Complex Information Presentation in Spoken Dialog Systems. In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, ACL ’04, pp 79–86, URL https://www.aclweb.org/anthology/P04-1011
  * Sugiyama et al. (2019) Sugiyama H, Meguro T, Higashinaka R (2019) Automatic Evaluation of Chat-Oriented Dialogue Systems Using Large-Scale Multi-references, Springer International Publishing, Cham, pp 15–25. DOI 10.1007/978-3-319-92108-2˙2, URL https://doi.org/10.1007/978-3-319-92108-2_2
  * Sutskever et al. (2014) Sutskever I, Vinyals O, Le QV (2014) Sequence to Sequence Learning with Neural Networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, MIT Press, Cambridge, MA, USA, NIPS’14, pp 3104–3112, URL http://dl.acm.org/citation.cfm?id=2969033.2969173
  * Talmor and Berant (2018) Talmor A, Berant J (2018) The web as a knowledge-base for answering complex questions. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, pp 641–651, DOI 10.18653/v1/N18-1059, URL https://www.aclweb.org/anthology/N18-1059
  * Tao et al. (2018) Tao C, Mou L, Zhao D, Yan R (2018) Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16179/15752
  * Tiedemann (2009) Tiedemann J (2009) News from OPUS : A Collection of Multilingual Parallel Corpora with Tools and Interfaces. In: Recent Advances in Natural Language Processing V, vol V, John Benjamins, pp 237–248
  * Tiedemann (2012) Tiedemann J (2012) Parallel Data, Tools and Interfaces in OPUS. In: Chair) NCC, Choukri K, Declerck T, Doğan MU, Maegaard B, Mariani J, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), European Language Resources Association (ELRA)
  * Traum (1999) Traum DR (1999) Speech Acts for Dialogue Agents, Springer Netherlands, Dordrecht, pp 169–201. URL https://doi.org/10.1007/978-94-015-9204-8_8
  * Trischler et al. (2017) Trischler A, Wang T, Yuan X, Harris J, Sordoni A, Bachman P, Suleman K (2017) NewsQA: A machine comprehension dataset. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, Association for Computational Linguistics, Vancouver, Canada, pp 191–200, DOI 10.18653/v1/W17-2623, URL https://www.aclweb.org/anthology/W17-2623
  * Tur and De Mori (2011) Tur G, De Mori R (2011) Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons
  * Tur and Mori (2011) Tur G, Mori RD (2011) Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley & Sons
  * Turing (1950) Turing AM (1950) Computing Machinery and Intelligence. Mind LIX(236):433–460, DOI 10.1093/mind/LIX.236.433, URL http://dx.doi.org/10.1093/mind/LIX.236.433
  * Ultes et al. (2013) Ultes S, Schmitt A, Minker W (2013) On quality ratings for spoken dialogue systems–experts vs. users. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, USA, NAACL - HLT’13, pp 569–578, URL https://www.aclweb.org/anthology/N13-1064
  * Ultes et al. (2017) Ultes S, Rojas Barahona LM, Su PH, Vandyke D, Kim D, Casanueva In, Budzianowski P, Mrkšić N, Wen TH, Gasic M, Young S (2017) PyDial: A Multi-domain Statistical Dialogue System Toolkit. In: Proceedings of ACL 2017, System Demonstrations, Vancouver, Canada, pp 73–78
  * Vinyals and Le (2015) Vinyals O, Le Q (2015) A neural conversational model. arXiv preprint arXiv:150605869
  * Voorhees (2006) Voorhees EM (2006) Evaluating Question Answering System Performance, Springer Netherlands, Dordrecht, pp 409–430. DOI 10.1007/978-1-4020-4746-6˙13, URL https://doi.org/10.1007/978-1-4020-4746-6_13
  * Walker et al. (1997) Walker MA, Litman DJ, Kamm CA, Abella A (1997) PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics, Madrid, Spain, EACL ’97, pp 271–280, URL https://doi.org/10.3115/979617.979652
  * Walker et al. (2000) Walker MA, Kamm CA, Litman DJ (2000) Towards developing general models of usability with PARADISE. Natural Language Engineering 6(3-4):363–377, DOI https://doi.org/10.1017/S1351324900002503
  * Wang et al. (2018) Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S (2018) GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In: Proceedings of the 2018 {EMNLP} Workshop {B}lackbox{NLP}: Analyzing and Interpreting Neural Networks for {NLP}, Association for Computational Linguistics, Brussels, Belgium, pp 353–355, DOI 10.18653/v1/W18-5446, URL https://www.aclweb.org/anthology/W18-5446
  * Wang et al. (2015) Wang Z, Wen TH, Su PH, Stylianou Y (2015) Learning Domain-Independent Dialogue Policies via Ontology Parameterisation. In: Proceedings of the SIGDIAL 2015 Conference: The 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Prague, Czech Republic, SIGDIAL ’15, pp 412–416, DOI 10.18653/v1/W15-4654, URL http://www.aclweb.org/anthology/W15-4654
  * Weizenbaum (1966) Weizenbaum J (1966) ELIZA - a Computer Program for the Study of Natural Language Communication Between Man and Machine. Communications of the ACM 9(1):36–45, URL http://doi.acm.org/10.1145/365153.365168
  * Wen et al. (2015) Wen TH, Gašić M, Mrkšić N, Su PH, Vandyke D, Young S (2015) Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, EMNLP ’15
  * Wen et al. (2016) Wen TH, Gašić M, Mrkšić N, Rojas-Barahona LM, Su PH, Vandyke D, Young S (2016) Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, NAACL -HLT ’16, pp 120–129
  * Wen et al. (2017) Wen TH, Vandyke D, Mrkšić N, Gasic M, Rojas Barahona LM, Su PH, Ultes S, Young S (2017) A Network-based End-to-End Trainable Task-oriented Dialogue System. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain, EACL ’17, pp 438–449, URL http://aclweb.org/anthology/E17-1042
  * Williams et al. (2013) Williams J, Raux A, Ramachandran D, Black A (2013) The Dialog State Tracking Challenge. In: Proceedings of the SIGDIAL 2013 Conference, Association for Computational Linguistics, Metz, France, pp 404–413
  * Williams et al. (2016) Williams J, Raux A, Henderson M (2016) The Dialog State Tracking Challenge Series: A Review. Dialogue & Discourse URL https://www.microsoft.com/en-us/research/publication/the-dialog-state-tracking-challenge-series-a-review/
  * Xing et al. (2017) Xing C, Wu W, Wu Y, Liu J, Huang Y, Zhou M, Ma W (2017) Topic Aware Neural Response Generation. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California, USA, AAAI ’17, pp 3351–3357, URL http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14563
  * Yang et al. (2015) Yang Y, Yih Wt, Meek C (2015) WikiQA: A challenge dataset for open-domain question answering. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Lisbon, Portugal, pp 2013–2018, DOI 10.18653/v1/D15-1237, URL https://www.aclweb.org/anthology/D15-1237
  * Yang et al. (2019) Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: Generalized autoregressive pretraining for language understanding. In: Advances in neural information processing systems, pp 5754–5764
  * Yao et al. (2014) Yao K, Peng B, Zhang Y, Yu D, Zweig G, Shi Y (2014) Spoken language understanding using long short-term memory neural networks. In: Spoken Language Technology Workshop (SLT), IEEE, South Lake Tahoe, NV, USA, IEEE 2014, pp 189–194, DOI 10.1109/SLT.2014.7078572, URL https://ieeexplore.ieee.org/document/7078572
  * Yeh and Chen (2019) Yeh YT, Chen YN (2019) FlowDelta: Modeling flow information gain in reasoning for conversational machine comprehension. In: Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Association for Computational Linguistics, Hong Kong, China, pp 86–90, DOI 10.18653/v1/D19-5812, URL https://www.aclweb.org/anthology/D19-5812
  * Young (2007) Young S (2007) CUED standard dialogue acts. Report, Cambridge University, Engineering Department URL http://mi.eng.cam.ac.uk/research/dialogue/LocalDocs/dastd.pdf
  * Young et al. (2007) Young S, Schatzmann J, Weilhammer K, Ye H (2007) The Hidden Information State Approach to Dialog Management. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, HI, USA, ICASSP ’07, vol 4, pp 149–152, URL http://svr-ftp.eng.cam.ac.uk/~sjy/papers/yswy07.pdf
  * Young et al. (2010) Young S, Gašić M, Keizer S, Mairesse F, Schatzmann J, Thomson B, Yu K (2010) The Hidden Information State Model: A Practical Framework for POMDP-based Spoken Dialogue Management. Computer Speech and Language 24(2):150–174, DOI 10.1016/j.csl.2009.04.001, URL http://dx.doi.org/10.1016/j.csl.2009.04.001
  * Young et al. (2013) Young S, Gašić M, Thomson B, Williams JD (2013) POMDP-Based Statistical Spoken Dialog Systems: A Review. Proceedings of the IEEE 101(5):1160–1179, DOI 10.1109/JPROC.2012.2225812
  * Zhang and Wang (2016) Zhang X, Wang H (2016) A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, New York, USA, IJCAI’16, pp 2993–2999, URL https://www.ijcai.org/Proceedings/16/Papers/425.pdf
  * Zhao and Eskenazi (2016) Zhao T, Eskenazi M (2016) Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning. In: Proceedings of the SIGDIAL 2016 Conference: The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Los Angeles, CA, USA, SIGDIAL’16, pp 1–10, DOI 10.18653/v1/W16-3601, URL http://www.aclweb.org/anthology/W16-3601
  * Zhao et al. (2017) Zhao T, Zhao R, Eskenazi M (2017) Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, pp 654–664, DOI 10.18653/v1/P17-1061, URL https://www.aclweb.org/anthology/P17-1061
  * Zhao et al. (2011) Zhao WX, Jiang J, Weng J, He J, Lim EP, Yan H, Li X (2011) Comparing Twitter and Traditional Media Using Topic Models. In: Proceedings of the 33rd European Conference on Advances in Information Retrieval, Springer-Verlag, Berlin, Heidelberg, ECIR’11, pp 338–349, URL http://dl.acm.org/citation.cfm?id=1996889.1996934
  * Zhou et al. (2018) Zhou L, Gao J, Li D, Shum HY (2018) The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. arXiv preprint arXiv:181208989