Conversation Learner provides a simple way of learning from dialog examples.
Thus, developers can focus on collecting or creating scripts of typical
conversations. To improve robustness, Conversation Learner integrates
reinforcement learning techniques to predict the best next action in a
conversation. This probabilistic next action prediction can transfer flow
within a dialog-flow in a non-linear way – likewise it can be combined with
rules to override probabilistic control and obtain more deterministic behavior
in certain contexts. Extending reinforcement learning to update dialog models
from user interactions is an ongoing research area (e.g., see Section 6.5);
thus we expect to see commercial toolkits to continue to evolve their
offerings and capabilities when it comes to learning from implicit feedback.

###  8.2 Commercial Applications

It’s true. I am unhappy. Do you think coming here will help you
not to be unhappy?
I need some help, that much seems certain. What would it mean to you if you
got some help?
Perhaps I could learn to get along with my mother. Tell me more about your
family.
My mother takes care of me. Who else in your family takes care of you?
My father. Your father.
You are like my father in some ways. What resemblance do you see?

Figure 8.2: Example of actual interaction with ELIZA excerpted from
Weizenbaum, [1966]. ELIZA simulated a Rogerian psychotherapist that primarily
echoes back statements as questions. Conversational input turns by the human
user of the system are in normal font. Responses from ELIZA are indented and
in teletype font.

####  8.2.1 Chatbots

As mentioned in early chapters, the focus of this book is primarily on
information-seeking. However, because chatbots were one of the first domains
where significant progress was made to a more open-domain NL interaction, we
briefly review several popular chatbots whose primary goal was to serve as AI
companions. Overall, the literature and industry demonstrate a trend toward
bots that blend together various functions of retrieval, recommendation, chit-
chat, and task-completion. Increasingly, there is a belief among practitioners
that conversational systems most demonstrate both EQ (emotional quotient) and
IQ (intelligent quotient). The demonstration of a “personality” can play a
critical role in user acceptance of the system – our discussion of
Microsofts’s XiaoIce [Zhou et al.,, 2020] highlights some of the reasons why
this belief is gaining popularity. Thus, understanding the history of
different chatbots can help current designers and researchers regardless of
the primary goals of the system. In this section we review a brief history of
well-known chatbots over time as further background for making design
decisions.

##### Influential Historical Chatbots

We review three well-known historical chatbots: ELIZA [Weizenbaum,, 1966],
PARRY [Colby,, 1975], and ALICE [Wallace,, 2009]. ELIZA was a chatbot based on
a model of Rogerian psychotherapy. Figure 8.2 shows an example excerpt of an
interaction with ELIZA originally published in 1966. ELIZA relied on very
simplistic dialog management – canned phrases and simple re-write rules of the
user’s input with catch-all fallback responses to handle unmatched input
phrases. Despite the simplicity and limitations of ELIZA, some users engaged
in long conversations with ELIZA. Ultimately, ELIZA highlighted the potential
power of chatbots while the limitations would indicate the need to support
more flexible conversations.

ELIZA: Good Evening. Tell me your problems. Please
terminate input with a period or a question mark.
PARRY: People get on my nerves sometimes.
ELIZA: I am not sure I understand you fully.
PARRY: You should pay more attention.
ELIZA: Suppose you should pay more attention.
PARRY: You’re entitled to your own opinion.
ELIZA: What makes you think I am entitled to my own opinion?
PARRY: It bothers me just to be around people in general.
ELIZA: Please go on.
PARRY: I went to the track at Bay Meadows a while back.
ELIZA: What does that suggest to you?
PARRY: I went to the races.
ELIZA: What else would you like to discuss?

Figure 8.3: PARRY was an attempt to automatically simulate a paranoid
schizophrenic patient with a chatbot to help understand more complex human
conditions. In one of the earliest multi-agent interaction systems, Vint Cerf
connected ELIZA and PARRY to have a conversation via the ARPANET (excerpt from
[CERF, Request for Comments: 439, 1973]). ELIZA’s turns are noted in teletype
font with no indentation. PARRY’s turns are noted in italics with indentation.

Less well-known than ELIZA, PARRY [Colby,, 1975] is another notable chatbot
that was created a decade after ELIZA. PARRY was designed to be a simulation
of mental states. In particular, PARRY was meant to simulate a paranoid
schizophrenic patient to help researchers and therapists understand more
complex human conditions. In this regard, PARRY stands as notable as a case
outside of information-seeking, task-completion, etc. where the primary goal
is truly to understand humans through simulation. More relevant to current
research was a humorous attempt by Vint Cerf to get PARRY and ELIZA to engage
in a conversation with each other (see excerpt in Figure 8.3). While the
conversation excerpt may not be the most stunning example, it is one of the
earliest cases of completely automated multi-agent system interaction. As
chatbots for task completion become more common, this ELIZA-PARRY interaction
is also one of the earliest examples of a more nascent and speculative line of
research on multi-agent systems [Baarslag et al.,, 2017]. In particular, this
is an early example of the possible future where autonomous chatbots might
negotiate on behalf of a specific user and be able to handle negotiations with
other autonomous agents – whether they are humans or bots. Again, the example
interaction highlights the brittle nature of early chatbot systems.

Before neural end-to-end chatbot systems started to become more prevalent, one
of the most successful chatbots of the early systems based on trigger-response
pairs is ALICE (Artificial Linguistic Internet Computer Entity) [Wallace,,
2009]. By the time of ALICE’s creation and usage, the evaluation of chatbots
had evolved significantly, and one of ALICE’s most notable achievements was to
win the Loebner prize version of the Turing test in 2004. In this version of
Turing’s imitation game, there are both humans and bots which judges must
interact with in a conversational way and then determine whether the actor on
the other end is a bot or a human. Figure 8.4 presents an excerpt of one
transcript from the Loebner prize [Shah,, 2006].

Judge 1: Do you still want to know about my life in New York? ALICE: Yes,
please tell me.
Judge 1: I’m a grad student and do not have much free time. I study a lot.
ALICE: You are a grad student and do not have much free time? Do you get to
meet a lot of people? What subject are you studying?
Judge 1: I meet a lot of my colleagues. I study computer science. What do you
do? ALICE: Do you make long term friends that way? Is computer science a hard
subject?

Figure 8.4: Excerpt from transcript of Loebner 2004 Contest of Turing’s
Imitation Game where ALICE won the gold medal (as reported in Shah, [2006]).

In terms of technology at the core of ALICE, the dialog is driven by a very
large set of manually curated triples of (input question, output response,
optional context). Matching of the input question and optional context is used
to determine and generate the appropriate response. The dialog strategies
going back to ELIZA of relying on questions as a part of dialog are evident
here. Also, as the granularity of the matching is much higher in ALICE, the
excerpt also demonstrates early capabilities for a bot to build one-on-one
rapport with a user, which systems like XiaoIce later use as a strategy. We
also see an early example of external knowledge being brought into the dialog
where mention of being a grad student and time-demands results in a new
introduced topic appropriate to the context, “Do you get to meet a lot of
people?”

The curated triples of ALICE are grown incrementally through past dialog
failures – specific cases of failures are used to expand and refine the rules.
Here again we see a relatively early example of now-common machine teaching
strategies that use natural dialog to drive system design. Finally, on a note
of influence in popular culture, Spike Jonze also cited ALICE as the
inspiration for the screenplay of the movie Her [Morais,, 2019].

##### Modern Chatbots

We now turn from a more historical perspective on chatbots to more modern
chatbots. In contrast to the origin of chatbots, most modern chatbots provide
a blend of information seeking and retrieval support, task completion, and
chit-chat functionalities. To demonstrate this trend in evolution of chatbots,
we now dive into XiaoIce (“Little Ice”) [Zhou et al.,, 2020] as an example of
the modern class of chatbots. After its release in 2014, XiaoIce became one of
the most popular social chatbots in the world reaching more than 660 million
active users.

While XiaoIce blends multiple functionalities, the design focuses on creating
an engaging conversation. This is predicated on the philosophy that the
primary reason we have a conversation is because it is enjoyable rather than
to get something out of it. This is in contrast to the bots that simply focus
on task completion where the goal is typically to get the task completed with
as few conversational turns as possible. The main metric for measuring long-
term user engagement in XiaoIce is conversational-turns per session (CPS).
That is, a longer engaging conversation will be considered typically better in
XiaoIce. This is similar to a viewpoint taken in other settings such as the
Alexa Challenge [Ram et al., 2018a, ] which considered conversation length as
a signal for positive engagement while also considering other factors such as
coherence, topic diversity, etc. In order to prevent simple methods of
extending a conversation that fail to provide much depth (e.g. like the ELIZA
system above), XiaoIce also considers the number of active users (NAU) to
compute the expected CPS. This is analogous to information systems which
consider sessions per unique user as a sign of long-term engagement while the
short-term engagement signal is the number of turns per session. As reported
in Zhou et al., [2020] XiaoIce ultimately reached an average of 23 CPS across
– one of the highest reported lengths of engagement for chatbots.

To demonstrate how XiaoIce achieves such deep engagement, we provide several
examples taken from different sessions of the same user as reported by Zhou et
al., [2020]. Figure 8.5 demonstrates how initial sessions (here Session 1)
focus on building rapport and a sense of personal connection. Note that
XiaoIce’s immediate response to a generic greeting is to “observe” something
personal (the user’s profile picture) and proactively engage in finding out
more about it.

Figure 8.5: Excerpt from Session 1 of XiaoIce-User conversation. In early
sessions, conversation focuses on building rapport and a sense of personal
connection (e.g. see XiaoIce’s response in Turn 1). Figure credit: Zhou et
al., [2020].

As the user returns in future sessions, XiaoIce continues to both build
rapport and extend interactions beyond simple chit-chat. Figure 8.6 shows an
excerpt from Session 6 of the same user. Early chatbot systems reviewed
previously in this chapter would have likely responded to the direct questions
in Turns 1 and 2 with little more than simple Yes/No responses (e.g. “Yes, I
have” or “No, why don’t you tell me about it.”). These questions are implicit
information-seeking questions; that is, the questions suggest that information
beyond a simple Yes/No is desired. XiaoIce not only responds to the question
affirmatively, but extends the conversation in a more human-like way by
responding to the implicit information request with details about the comic
and character which were not requested.

Figure 8.6: Excerpt from Session 6 of XiaoIce-User conversation (same user as
Figure 8.5). In Turns 1 and 2, rather than simply answer the question, XiaoIce
responds to an implicit information request demonstrating external knowledge.
Figure credit: Zhou et al., [2020].

Given XiaoIce’s flexibility and large-coverage of different skills,
information-seeking, and recommendation capabilities, users often continue
their engagement with XiaoIce over many sessions. As such it is important to
consider how XiaoIce extends beyond functionality and also supports ethical
and responsible design. In Figure 8.7, we see an excerpt of Session 42 with
the same user. Here, we see in a discussion of bed time, XiaoIce responds in
Turn 4 to say that XiaoIce itself is “going to bed”. This is an example of
XiaoIce’s overall approach to Responsible AI222Microsoft Responsible AI:
https://www.microsoft.com/en-us/ai/responsible-ai for bots – in particular in
this case the design intends to encourage social norms. For example, in this
case, the bot can demonstrate tendencies to encourage the same norms of
standard diurnal rhythms in a contextually appropriate way. That is, at a late
hour, the bot can gently encourage sleep based on that being a social norm for
the time of day rather than a prescriptive decision of “what is best for the
user” – a much more complex issue to determine and which the bot avoids taking
a direct stance on.

More specifically, XiaoIce’s design was evolved and refined based on the
guidelines described in Microsoft’s guidelines for responsible bots
[Microsoft,, 2018]. In today’s complex ecosystem of bot design, bot designers
should consider early in the design process what guidelines they will use in
designing the system. Additionally, whatever guidelines chosen, these
guidelines may need to evolve as system capabilities and new challenges
emerge. In fact, some of the 10 guidelines described in Microsoft, [2018]
evolved out of the need for better approaches to design when earlier chatbot
systems gave rise to new challenges in real-world situations. The ten
guidelines described in that work include:

  1. 1.

Articulate the purpose of your bot and take special care if your bot will
support consequential use cases.

  2. 2.

Be transparent about the fact that you use bots as part of your product or
service.

  3. 3.

Ensure a seamless hand-off to a human where the human-bot exchange leads to
interactions that exceed the bot’s competence.

  4. 4.

Design your bot so that it respects relevant cultural norms and guards against
misuse.

  5. 5.

Ensure your bot is reliable.

  6. 6.

Ensure your bot treats people fairly.

  7. 7.

Ensure your bot respects user privacy.

  8. 8.

Ensure your bot handles data securely.

  9. 9.

Ensure your bot is accessible.

  10. 10.

Accept responsibility.

Interested readers should refer to the article for examples and contexts of
how to apply the guidelines. These guidelines are applied in XiaoIce in a
number of specific ways. For example, with respect to privacy, XiaoIce has
models of “topic sensitivity” that are applied in determining when it is
reasonable to try to form a group of users (one capability of XiaoIce). While
seeking to design from a point-of-view where the user is in control, to guard
against misuse, XiaoIce is also designed with the capability to not respond.
This may be appropriate in situations where a question has been asked and
responding in any way could force XiaoIce to take an opinion on a sensitive
and delicate topic. To help protect against misuse, XiaoIce can simply not
respond. To respect relevant cultural norms and avoid imposing the norms of
the bot designers on others, public conversations specific to the culture and
locale are labeled as within-norm or outside-of-norm for each specific culture
and locale and used to train XiaoIce to adhere to local cultural norms. These
are just a few of the ways in which the guidelines are manifested in XiaoIce’s
operation.

Figure 8.7: Excerpt from Session 42 of XiaoIce-User conversation (same user as
Figure 8.5). In Turn 4, XiaoIce encourages social norms as part of its general
responsible AI approach. Figure credit: Zhou et al., [2020].

The examples above and overall statistics demonstrate that XiaoIce is capable
of creating long-running engagement with users over many sessions while
navigating complex ethical issues. It is worth also considering the basic
functionality of why users find XiaoIce to be engaging. Fang et al., [2018];
Li et al., [2016] provide evidence that generic responses (e.g., “I don’t
understand, what do you mean?”) yield longer sessions but lead to overall user
attrition as measured by NAU. To help ensure that topics are extended based on
the topic, XiaoIce’s topic selection model (see Section 7.3) takes into
account five factors: (1) contextual relevance: novel information that is
still related to the discussion thus far; (2) freshness: trending in the news
or other external sources; (3) personal interests: likely of interest to the
user based on engagement in past sessions; (4) popularity: high overall
attention online; (5) acceptance: past interaction with the topic from other
users has been high. These dimensions of topicality provide five broad ways in
which XiaoIce can bring new information into the conversation. In terms of
more general functionality, XiaoIce like other modern chatbots has a large
extensive skillset where specific “skills” or modes can be engaged through
contextual triggers. Focusing simply on the chat capabilities itself, XiaoIce
fuses two common modern styles of chatbot systems: (1) information retrieval
based chat which uses retrieval from past conversations across users filtered
for appropriateness; (2) a neural based chat model which is trained on
filtered query-context-response pairs. While the complexity of the dialog
policy and mitigation go beyond the scope of this article, XiaoIce can be seen
more generally as how chatbots have evolved from social chit-chat to include
social chit-chat as simply one capability within personalized and extensible
capabilities for search, recommendation, and task completion.

Figure 8.8: Screenshot of Microsoft’s Bing search engine (July 23, 2020). The
search engine result page includes instant answers and perspectives (main
column inset box with panes), an entity pane for exploring related attributes
(right), and suggested follow-up questions that are useful assuming the
question is answered (main column “People Also Ask” box).

####  8.2.2 Conversational Search Engine

With the advances in CIR, even standard GUI interfaces to search have become
“more conversational”. These changes can be seen as part of a broader trend
acknowledging that a search engine site will continue to be useful in many
contexts where voice-based search is inconvenient; however, it may still be
desirable to introduce analogous affordances on a Search Engine site for
conversational techniques for disambiguation, suggesting new topics, etc. Such
trends are evidenced on a number of major search engines including Google and
Bing as well as other search engines. Since some of the technology behind
these capabilities have been discussed in earlier chapters. We focus on how
these capabilities present in the user experience and how the capabilities
relate to a conversational style.

Figure 8.8 shows a screenshot of Microsoft’s Bing Search Engine in response to
the query “is the ford mustang fast”. The inset box provides an aggregate
instant answer (“Yes”) with perspectives justifying the answer (“two source”).
The same capability can be used to present perspectives which disagree on more
subjective questions. On the right side, there is an entity pane with
“attributes” that lead to new topics to explore. This entity pane has become
quite common in many search engines and can be thought of as a similar
approach to XiaoIce’s strategy to create engagement with novel contextually
relevant information. Another conversational capability seen in the same
screenshot, is the “People Also Ask” feature which focuses on suggesting new
useful questions focused on natural follow-up questions under the assumption
that the instant answer or other results will answer the main query. For an
in-depth discussion of techniques which can power such experiences, the reader
should see Section 7.2.

To illustrate another conversational capability, Figure 8.9 shows a screenshot
of Bing in response to the query “convert string to int”. Note that here the
search engine has responded with a clarification question [Zamani et al.,
2020a, ; Zamani et al., 2020c, ] “What programming language are you looking
for?” with a number of options. Beyond simply being a clarification question
in a more conversational style, the use of “programming language” is again an
example of a similar technique used in chatbots like XiaoIce. By introducing
external knowledge (the query never mentioned “programming language”), the
search engine introduces new information into the conversation rather than
using a generic prompt such as “which of these would you like?” Similar to the
studies on chatbots, Zamani et al., 2020c  provide evidence that the more
specific question leads to higher engagement than the more general question.
The reader are referred to Section 7.1 for an in-depth discussion of
techniques.

Figure 8.9: Screenshot of Microsoft’s Bing search engine (July 23, 2020). The
clarification question demonstrates understanding by specifying “programming
language” in the clarification question rather than asking a generic question
such as “Which of these would you like?”

Search engines are continuously experimenting with such features. Some may
only last a short time as not all conversational techniques work as well in a
website as a chatbot or voice assistant, other changes become longer lasting.
One research challenge that still persists in this space is to make search
engines more contextual across multiple queries and answers in a search
session. This is a case where the technology to power such experiences in
chatbots and voice-assistants have evolved well beyond the web experience.
Nonetheless, we believe that search engines will continue this trend of
evolving to become more conversational and contextual.

####  8.2.3 Productivity-Focused Agents

Some of the first CIR experiences developed around the notion of a personal
assistant that could act on a person’s private information store. These
assistants were often focused towards productivity tasks such as calendar
management, task management, contact management, etc. Typically, these
assistants blend both information-seeking and task-completion oriented
behaviors. For example, a user may seek to query information from their
calendar before taking action to set up a new meeting, move a previous one,
etc. We will review these productivity-focused personal assistants.

##### From PAL to Device-Based Assistants

The RADAR / CALO personal assistant projects both were part of DARPA’s broader
”Personal Assistants that Learn” (PAL) program. While neither was directly a
commercial system, they are known to have strongly influenced the development
of Apple’s Siri and provide an interesting historical study in the space of
personal productivity assistants.

Figure 8.10 provides a broad sketch of the RADAR system. The RADAR system was
designed for an assistive scenario centered on modern email and calendaring
systems. Here a user is focused on processing their inbox, sending new mails,
and making scheduling and resourcing decisions.

Figure 8.10: The RADAR system’s conception of a personal assistant focused
heavily on extracting task lists from email and calendaring. Figure credit:
Freed et al., [2008].

As such the system’s functionality focused on three basic AI-assisted
capabilities.

  1. 1.

Calendar management [Berry et al.,, 2011, 2006; Modi et al.,, 2004].

  2. 2.

Dealing with uncertain resources in scheduling [Fink et al.,, 2006].

  3. 3.

Task management [Freed et al.,, 2008].

Specific capabilities included automatic extraction of tasks from emails,
finding rooms under uncertain conditions (range of audience sizes, A/V
capabilities), calendar optimization etc.

CALO covered a similar range of functionality and extended to other scenarios
such as AI-assisted authoring of documents and QA against a personal store of
past experiences (e.g. web pages viewed, etc.). More importantly CALO was
designed with more of an extensible architecture of skills in mind. Both
CALO’s architecture and the learnings from both RADAR / CALO and the more
general PAL program helped researchers recognize the need for unifying
architectures. The result was a new extensible architecture proposed by
Guzzoni et al., [2007] and shown in Figure 8.11 with a vision of creating a
“do engine” rather than a “search engine”. Guzzoni et al.,’s design first led
building a consumer-focused version of CALO at Stanford Research Institute
(SRI). One of the authors then moved from SRI to co-found a startup Siri
before Apple later acquired that company in
2010333http://www.morgenthaler.com/press-
releases/Siri%20Named%20Top%2010%20Emerging%20Tech%20of%202009.pdf.

Figure 8.11: The research influence on DARPA’s PAL program to Siri can be
traced to the architecture Guzzoni et al. designed to unify the services in
PAL (primarily CALO) and enable easy extensibility. Figure credit: Guzzoni et
al., [2007].

Historically, this brings us from simple productivity-focused assistants with
more generally device-based assistants. These assistants typically offer CIR
integrated throughout their functionality and often are integrated with
personal devices (e.g. mobile phones, situated speakers). In addition to the
examples like Apple’s Siri, this includes assistants such as Google Assistant,
and Amazon Alexa. These voice-based conversational assistants must go beyond
the chatbot interactions described above and tackle challenges of voice
recognition in noisy environments and increasingly multi-modal and multi-party
interactions. As an example Alexa [Ram et al., 2018b, ; Khatri et al., 2018b,
; Gopalakrishnan et al.,, 2019; Goel et al.,, 2018; Paul et al.,, 2019; Yavuz
et al.,, 2019; Khatri et al., 2018a, ; Kollar et al.,, 2018] highlights an
interesting trend where a situated speaker interacts with multiple-parties
(e.g., all of the members of a household) rather than a single party as is the
case with a personal productivity-agent. This has both given rise to an
increase in interest in speaker identification as well as other authentication
mechanisms to be able to provide personal productivity functionality as well
as information-seeking, chit-chat, recommendation, and task-completion. While
a full discussion of device-based assistants is beyond the scope of this book,
we see these devices not only continuing to become more conversational, but
also see an increasing trend in research to customize the experience to
different types of users (e.g. kids vs. adults) different situations (e.g.
theme parks vs. offices) and different scenarios (e.g. entertainment vs.
education).

####  8.2.4 Hybrid-Intelligence Assistants

Finally, we conclude this chapter with a brief discussion of hybrid-
intelligence assistants. These assistants typically blend automated systems
with crowd-powered systems to provide capabilities that are beyond automated
systems’ current capabilities but with an overall impression to the user that
the entire experience is completely AI-powered. Examples of conversational
assistants of this type include Facebook’s M and Microsoft’s Calendar.help.
Facebook’s M (now defunct) was an assistant within Facebook’s Messenger.
During the time that M was available, a user could perform tasks within
Messenger through M such as booking a restaurant, buying a gift, etc. External
stories indicated that much of M’s experience was done with crowd-sourced
computation where unhandled cases were escalated to a person who could
complete the task – the dialog and the set of low-level operations the human
carried out could then become training data. A similar tactic was taken in
Calendar.help [Cranshaw et al.,, 2017] which was designed to help people
automatically find time to meet without the time-consuming negotiation of
finding mutual time availability. Initial versions of Calendar.help were based
on a handful of high-precision rules that could do automated extraction from
the email of a meeting request with a fall-back to crowd-powered workflows
when a general intent to meet was detected and specifics were uncertain. This
provided efficient basic functionality of the system and again provided a
steady source of natural language training data and ground-truth labels from
the crowd-powered workflow. The goal of such training data is to reliably,
robustly train more general models through a self-sustainable process. While
scheduling remains an everyday challenge for many people, the technology in
Calendar.help has grown well beyond its original hybrid-intelligence design
and has influenced the development of Microsoft’s Scheduler which is a product
in early preview release as of the date of this writing.

##  Chapter 9 Conclusions and Research Trends

We conclude the book with a brief discussion of research trends and areas for
future work. While we have colored a promising picture of CIR, the field is
still at a research exploration stage in that a commercial CIR system that
goes beyond popular search engines (e.g., Google and Bing) to effectively meet
users’ information needs of wide diversity and complexity is yet to be
developed. In the previous chapters, we describe the recent progress of
individual CIR modules or sub-systems. But these components are yet to be
pieced together to build a system. Many research questions remain open. What
is the best system architecture? How to evaluate the effectiveness of the
system in real-world settings? How to continually improve the system
performance after its deployment? What is a good design for effective and
responsible Human-AI Interaction?

The reference architecture of CIR systems, shown in Figure 1.4, is a proposal
yet to be empirically verified. It can be viewed as following the schema of
_society of mind_ [Minsky,, 1988]. The AI system is made of many small
modules. Each by itself can only do simple things with little intelligence.
But combining them in some special ways can lead to human-like intelligence.
The CIR system consists of many modules that are coordinated by a global
dialog manager, and are collectively optimized for a long-term reward, such as
the increase of the number of active users in three months.

The interactive nature of conversational search makes it challenging to
develop robust and reusable evaluation methods for system development.
Although off-policy evaluation based on pre-collected datasets is widely used
to measure the individual CIR modules, such as query understanding, document
retrieval and ranking, question answering, and so on, as described in the
previous chapters, it is insufficient to measure how effective a system
_interacts_ with users to accomplish a search task. On the other hand, online
system evaluation methods based on simulated users and A/B test are too costly
to perform frequently, especially at the stage of research exploration. Thus,
how to combine the strengths of different evaluation methods to seek the best
trade-off between low cost and high reliability is a challenge not only for
CIR, but also for all interactive AI systems.

In terms of application-oriented research, our understanding of how to measure
and optimize long-term engagement and satisfaction in a variety of settings is
evolving. In general chatbot settings, there is a need to continue to define
long-term evaluation metrics for engagement that go beyond conversational-
turns per session (CPS) and Number of Active Users (NAU) and consider task-
completion, goal-support, etc. For example, recommendation technology
increasingly is explored in aspirational goal-setting (e.g. “I want to lose
weight”, “I want to eat healthy”, “I want to learn more about deep learning”)
– clearly in these settings achieving the goal while creating engagement to
get users to return must be considered (presumably being more likely to return
also makes the user more likely to achieve their goal). Similarly when
conversational agents are primarily providing social companionship, there are
theories of linguistic accommodation or coordination [Danescu-Niculescu-Mizil
et al.,, 2011] which describe human-human interactions and may form a key part
of metrics to know when AI-based social companions are performing well.

Building a conversational AI system, like CIR, also imposes a challenge
regarding the development of continual learning methods that allow the system
to adapt in a dynamic environment. Most AI models, including the CIR modules
described in this book, are developed using independent and identically
distributed (IID) learning methods that assume that the world is static, and
the data of the tasks that the models need to perform can be pre-collected.
However, a conversational system has to work in a non-stationary world, where
the tasks and settings change constantly. For example, XiaoIce has to release
new dialog skills every week since its first release in 2014 to keep and grow
its user base, and many of these skills are short lived [Zhou et al.,, 2020].
Ideally, the system should be able to adapt itself to the environment
efficiently with little supervision of new tasks or settings without
catastrophic forgetting, where the performance on old tasks degrades sharply
when the system adapts to new tasks. Recent neuroscience studies show that
humans can adapt much more efficiently than machines because humans are born
with rich innate knowledge inherited from humans’ long evolutionary history
and have the ability of transferring knowledge efficiently in two directions
in our life-long learning: in forward transfer, previously learned tasks help
improve the performance of new tasks, and in backward transfer, learning a new
task improves the performance of old tasks [Hadsell et al.,, 2020]. Inspired
by these neuroscience studies, continual learning with neural networks becomes
an active research field [e.g., Parisi et al.,, 2019; Hadsell et al.,, 2020].

The learning efficiency can also be significantly improved by making better
use of knowledge. Kahneman, [2011] introduces a metaphor to interpret how
humans use knowledge for problem solving. It says that your brain has a
_System 1_ that uses implicit knowledge for quick, instinctive responses, and
a _System 2_ that uses explicit knowledge for slower, more thoughtful
processing. We can find many AI systems that work similarly to Systems 1 and
2, respectively. For example, a close-book QA system based on GPT-3 [Brown et
al.,, 2020] is like System 1. It assumes that all the knowledge has been
encoded in the model parameters, and the system can rely solely on the model
to generate answers without accessing any external knowledge, e.g., by
retrieving relevant Web documents. An open-book QA system based on a
retriever-reader pipeline (Chapter 5) works like System 2. Given an input
query, the system first retrieves relevant documents, entities, and other
forms of explicit knowledge from document collections or knowledge bases, and
then generates the answer from the retrieved knowledge. There have been recent
works on integrating explicit and implicit knowledge for open-domain question
answering [e.g., Marino et al.,, 2021; Gui et al.,, 2021].

There is also the possibility of accelerating progress within the field with
richer paradigms of supervision as well as learning that is less dependent on
supervision. Cases of richer supervision include teaching-by-demonstration
where the ability to tie language to procedural tasks could unlock potential
for conversational systems to help truly bring about the Natural User
Interface (NUI). It also includes extending machine teaching to dialog-based
interaction [Peng et al.,, 2021; Gao et al., 2020a, ]. Reducing the dependency
on supervision includes research trends into large neural language models that
are providing progress toward few-shot and zero-shot learning in many
language-related tasks and increasingly in conversational and search tasks.

There is the potential for other forms of reasoning and intelligence that
humans demonstrate and which apply in conversational intelligence. For
example, humans demonstrate social awareness and can interpret attention in
settings to understand what was meant by combining an understanding of where
the speaker is looking with the words they are saying – human can then reason
if the language applies to a person or an object. Humans demonstrate emotional
intelligence in a number of ways; these capabilities may be even more critical
in healthcare situation such as general well-being bots or bots that can
engage in conversational therapy such as cognitive behavioral therapy
[Ghandeharioun et al.,, 2019]. There are a large number of possibilities to
continue to extend capabilities for conversational search and interaction.
While there has been major progress over the last decade, we believe that we
are still only beginning to see the broad impact that conversation will have
on search, question answering, and human-AI interaction more generally.

Within the space of conversational understanding some of the most salient
challenges are in grounded language generation and understanding. That is, a
conversational system requires being able to reason about how the language and
the state of the world correspond. Challenges in this space range from
abstractive summarization where a question answering system would want to
avoid _hallucinating_ phrases and facts which did not exist in the full
document to guide process completion where the agent should have a model of
the state of the world relative to the process through which the user is being
guided.

For device-based assistants and physically situated assistants (e.g.,
intelligent buildings, elevators, etc.), challenges include integrating
observations from multiple modalities including voice, vision, and text. For
example in multiple-speaker situations inferring attention to determine which
person is interacting with the device versus the people who are interacting
with each other. Likewise, increasingly it is important to be able to identify
and authenticate people in multi-person scenarios. Other areas where research
is needed is in physical understanding and monitoring that should modulate the
behavior of conversational interaction – Amazon Alexa’s _whisper mode_ is a
simple example of physical understanding. More general cases might involve
understanding anomalies (e.g., a kitchen fire), adjusting to ambient noise, or
using vision to understand gestures and objects.

In terms of Human-AI Interaction Design there are still quite fundamental
challenges for conversational interaction. Two that often come up are: (1)
flexibility in specifying goals; and (2) knowing what the system can do. While
conversational systems have increased their flexibility by leaps-and-bounds
over early systems, we still have not reached systems that provide the
flexibility of humans. For example, if a young girl walks up to a librarian
and says, “I am looking for a fun book to read,” the librarian can quickly
narrow in on suitable recommendations. Our current systems have not yet
reached this point of flexibility and this is a simpler case of goal
specification which we are likely to achieve in the next few years. Turning to
the second major challenge within Human-AI Interaction Design, the problem of
conveying to a user all of the skills and capabilities of the system is tied
together with a more general Human Computer Interaction (HCI) problem termed
the gulf-of-evaluation. That is, even when a system has new capabilities it is
difficult for users to discover and know these capabilities exist; as a
result, the user may never learn the best ways to interact with the system. In
an area such as conversational search where capabilities are constantly
evolving, finding ways to regularly and efficiently communicate new
capabilities to the user is a key part of designing an overall system that
works well.

Finally, another key ongoing challenge is around Responsible AI. As we pointed
out in Section 8.2.1, Responsible AI is a complex and evolving topic. New
challenges are evolving on a regular basis and the need to continually evolve
new guidelines for Human-AI interaction [Amershi et al.,, 2019] evolves with
those challenges. One of the key challenges currently in Responsible AI for
conversational systems is the need to defend against harms and mitigate
potential toxicity and bias [Bhat et al.,, 2021; Breitfeller et al.,, 2019;
Zhang et al., 2018a, ] – either in training data, introduced through feedback
and usage, or simply inappropriate due to local culture or context. In terms
of privacy, as device-based assistants are extended to have personal
information capabilities, even with authentication, the systems may need to
defer disclosing personal information if other parties are present. This is
similar to challenges for privacy reported on email notifications when a
screen is being shared [Kim et al.,, 2019].

## Acknowledgments

## References

  * Alain et al., [2015] Alain, G., Lamb, A., Sankar, C., Courville, A., and Bengio, Y. (2015).  Variance reduction in SGD by distributed importance sampling.  arXiv preprint arXiv:1511.06481.
  * Aliannejadi et al., [2020] Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., and Burtsev, M. (2020).  Convai3: Generating clarifying questions for open-domain dialogue systems (clariq).  arXiv preprint arXiv:2009.11352.
  * Aliannejadi et al., [2019] Aliannejadi, M., Zamani, H., Crestani, F., and Croft, W. B. (2019).  Asking clarifying questions in open-domain information-seeking conversations.  In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 475–484.
  * Allahyari et al., [2017] Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E. D., Gutierrez, J. B., and Kochut, K. (2017).  Text summarization techniques: a brief survey.  arXiv preprint arXiv:1707.02268.
  * Allen et al., [1993] Allen, R. B., Obry, P., and Littman, M. (1993).  An interface for navigating clustered document sets returned by queries.  In Proceedings of the conference on Organizational computing systems, pages 166–171.
  * Amershi et al., [2019] Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019).  Guidelines for human-ai interaction.  In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1–13.
  * Anantha et al., [2020] Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., and Chappidi, S. (2020).  Open-domain question answering goes conversational via question rewriting.  arXiv preprint arXiv:2010.04898.
  * Andreas et al., [2016] Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016).  Learning to compose neural networks for question answering.  arXiv preprint arXiv:1601.01705.
  * Asri et al., [2017] Asri, L. E., Schulz, H., Sharma, S., Zumer, J., Harris, J., Fine, E., Mehrotra, R., and Suleman, K. (2017).  Frames: a corpus for adding memory to goal-oriented dialogue systems.  arXiv preprint arXiv:1704.00057.
  * Auer et al., [2007] Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., and Ives, Z. (2007).  DBpedia: A nucleus for a web of open data.  In The semantic web, pages 722–735. Springer.
  * Aumüller et al., [2017] Aumüller, M., Bernhardsson, E., and Faithfull, A. (2017).  Ann-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms.  In International Conference on Similarity Search and Applications, pages 34–49. Springer.
  * Austin, [1975] Austin, J. L. (1975).  How to do things with words, volume 88.  Oxford university press.
  * Baarslag et al., [2017] Baarslag, T., Kaisers, M., Gerding, E., Jonker, C. M., and Gratch, J. (2017).  When will negotiation agents be able to represent us? the challenges and opportunities for autonomous negotiators.  International Joint Conferences on Artificial Intelligence.
  * Baeza-Yates et al., [2011] Baeza-Yates, R., Ribeiro-Neto, B., et al. (2011).  Modern information retrieval (second edition).  Pearson Education Limited.
  * Bahdanau et al., [2015] Bahdanau, D., Cho, K., and Bengio, Y. (2015).  Neural machine translation by jointly learning to align and translate.  In Proc. of ICLR.
  * Bajaj et al., [2016] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016).  Ms marco: A human generated machine reading comprehension dataset.  arXiv preprint arXiv:1611.09268.
  * Barto and Mahadevan, [2003] Barto, A. G. and Mahadevan, S. (2003).  Recent advances in hierarchical reinforcement learning.  Discrete event dynamic systems, 13(1-2):41–77.
  * Bates, [1979] Bates, M. J. (1979).  Information search tactics.  Journal of the American Society for information Science, 30(4):205–214.
  * Bates, [1989] Bates, M. J. (1989).  The design of browsing and berrypicking techniques for the online search interface.  Online review, 13(5):407–424.
  * Bengio et al., [2003] Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003).  A neural probabilistic language model.  The journal of machine learning research, 3:1137–1155.
  * Berg-Kirkpatrick et al., [2011] Berg-Kirkpatrick, T., Gillick, D., and Klein, D. (2011).  Jointly learning to extract and compress.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 481–490.
  * Berry et al., [2006] Berry, P., Peintner, B., Conley, K., Gervasio, M., Uribe, T., and Yorke-Smith, N. (2006).  Deploying a personalized time management agent.  In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pages 1564–1571. ACM.
  * Berry et al., [2011] Berry, P. M., Gervasio, M., Peintner, B., and Yorke-Smith, N. (2011).  Ptime: Personalized assistance for calendaring.  ACM Transactions on Intelligent Systems and Technology (TIST), 2(4):40.
  * Bhat et al., [2021] Bhat, M. M., Hosseini, S., Awadallah, A. H., Bennett, P., and Li, W. (2021).  Say ‘yes’ to positivity: Detecting toxic language in workplace communications.  In EMNLP 2021.
  * Biega et al., [2018] Biega, A. J., Gummadi, K. P., and Weikum, G. (2018).  Equity of attention: Amortizing individual fairness in rankings.  In The 41st international acm sigir conference on research & development in information retrieval, pages 405–414.
  * Bisk et al., [2020] Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020).  Experience grounds language.  arXiv preprint arXiv:2004.10151.
  * Bojanowski et al., [2017] Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017).  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146.
  * Bollacker et al., [2008] Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008).  Freebase: a collaboratively created graph database for structuring human knowledge.  In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. ACM.
  * Breitfeller et al., [2019] Breitfeller, L., Ahn, E., Jurgens, D., and Tsvetkov, Y. (2019).  Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664–1674.
  * Broder, [2002] Broder, A. (2002).  A taxonomy of web search.  In ACM Sigir forum, volume 36, pages 3–10. ACM New York, NY, USA.
  * Brooks et al., [1986] Brooks, H. M., Daniels, P. J., and Belkin, N. J. (1986).  Research on information interaction and intelligent information provision mechanisms.  Journal of Information Science, 12(1-2):37–44.
  * Brown et al., [1983] Brown, G., Brown, G. D., Brown, G. R., Yule, G., and Gillian, B. (1983).  Discourse analysis.  Cambridge university press.
  * Brown et al., [2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165.
  * Budzianowski et al., [2017] Budzianowski, P., Ultes, S., Su, P.-H., Mrkšić, N., Wen, T.-H., Casanueva, I., Rojas-Barahona, L., and Gašić, M. (2017).  Sub-domain modelling for dialogue management with hierarchical reinforcement learning.  arXiv preprint arXiv:1706.06210.
  * Budzianowski et al., [2018] Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., and Gasic, M. (2018).  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In Proceedings of EMNLP.
  * Byrne et al., [2019] Byrne, B., Krishnamoorthi, K., Sankar, C., Neelakantan, A., Duckworth, D., Yavuz, S., Goodrich, B., Dubey, A., Cedilnik, A., and Kim, K.-Y. (2019).  Taskmaster-1: Toward a realistic and diverse dialog dataset.  arXiv preprint arXiv:1909.05358.
  * Cao et al., [2008] Cao, G., Nie, J.-Y., Gao, J., and Robertson, S. (2008).  Selecting good expansion terms for pseudo-relevance feedback.  In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 243–250.
  * Carbonell and Goldstein, [1998] Carbonell, J. and Goldstein, J. (1998).  The use of mmr, diversity-based reranking for reordering documents and producing summaries.  In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336.
  * Carpineto et al., [2009] Carpineto, C., Osiński, S., Romano, G., and Weiss, D. (2009).  A survey of web clustering engines.  ACM Computing Surveys (CSUR), 41(3):1–38.
  * Casanueva et al., [2018] Casanueva, I., Budzianowski, P., Su, P.-H., Ultes, S., Rojas-Barahona, L. M., Tseng, B.-H., and Gašić, M. (2018).  Feudal reinforcement learning for dialogue management in large domains.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 714–719.
  * Celikyilmaz et al., [2018] Celikyilmaz, A., Bosselut, A., He, X., and Choi, Y. (2018).  Deep communicating agents for abstractive summarization.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662–1675.
  * Celikyilmaz et al., [2020] Celikyilmaz, A., Clark, E., and Gao, J. (2020).  Evaluation of text generation: A survey.  arXiv preprint arXiv:2006.14799.
  * Chakraborty et al., [2019] Chakraborty, N., Lukovnikov, D., Maheshwari, G., Trivedi, P., Lehmann, J., and Fischer, A. (2019).  Introduction to neural network based approaches for question answering over knowledge graphs.  arXiv preprint arXiv:1907.09361.
  * Chang et al., [2021] Chang, Y., Narang, M., Suzuki, H., Cao, G., Gao, J., and Bisk, Y. (2021).  Webqa: Multihop and multimodal qa.  arXiv preprint arXiv:2109.00590.
  * Chapelle et al., [2012] Chapelle, O., Joachims, T., Radlinski, F., and Yue, Y. (2012).  Large-scale validation and analysis of interleaved search evaluation.  ACM Transactions on Information Systems (TOIS), 30(1):1–41.
  * Chapelle and Li, [2012] Chapelle, O. and Li, L. (2012).  An empirical evaluation of Thompson sampling.  In Advances in Neural Information Processing Systems 24 (NIPS), pages 2249–2257.
  * [47] Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017a).  Reading Wikipedia to answer open-domain questions.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * [48] Chen, L., Zhou, X., Chang, C., Yang, R., and Yu, K. (2017b).  Agent-aware dropout DQN for safe and efficient on-line dialogue policy learning.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2454–2464.
  * [49] Chen, W., Chen, J., Qin, P., Yan, X., and Wang, W. Y. (2019a).  Semantically conditioned dialog response generation via hierarchical disentangled self-attention.  arXiv preprint arXiv:1905.12866.
  * [50] Chen, Y., Wu, L., and Zaki, M. J. (2019b).  Graphflow: Exploiting conversation flow with graph neural networks for conversational machine comprehension.  arXiv preprint arXiv:1908.00059.
  * Chen and Bansal, [2018] Chen, Y.-C. and Bansal, M. (2018).  Fast abstractive summarization with reinforce-selected sentence rewriting.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686.
  * Chen and Gao, [2017] Chen, Y.-N. and Gao, J. (2017).  Open-domain neural dialogue systems.  In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Tutorial Abstracts), pages 6–10.
  * Cheng et al., [2021] Cheng, H., Shen, Y., Liu, X., He, P., Chen, W., and Gao, J. (2021).  Unitedqa: A hybrid approach for open domain question answering.  arXiv preprint arXiv:2101.00178.
  * Cheng and Lapata, [2016] Cheng, J. and Lapata, M. (2016).  Neural summarization by extracting sentences and words.  In 54th Annual Meeting of the Association for Computational Linguistics, pages 484–494. Association for Computational Linguistics.
  * Cheyer and Guzzoni, [2014] Cheyer, A. and Guzzoni, D. (2014).  Method and apparatus for building an intelligent automated assistant.  US Patent 8,677,377.
  * Choi et al., [2018] Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-t., Choi, Y., Liang, P., and Zettlemoyer, L. (2018).  Quac: Question answering in context.  arXiv preprint arXiv:1808.07036.
  * Chopra et al., [2016] Chopra, S., Auli, M., and Rush, A. M. (2016).  Abstractive sentence summarization with attentive recurrent neural networks.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98.
  * Christmann et al., [2019] Christmann, P., Saha Roy, R., Abujabal, A., Singh, J., and Weikum, G. (2019).  Look before you hop: Conversational question answering over knowledge graphs using judicious context expansion.  In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 729–738.
  * Clark et al., [2020] Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020).  Electra: Pre-training text encoders as discriminators rather than generators.  arXiv preprint arXiv:2003.10555.
  * Cleverdon et al., [1966] Cleverdon, C. W., Mills, J., and Keen, E. M. (1966).  Factors determining the performance of indexing systems,(volume 1: Design).  Cranfield: College of Aeronautics, 28.
  * Colby, [1975] Colby, K. M. (1975).  Artificial Paranoia: A Computer Simulation of Paranoid Processes.  Elsevier Science Inc., New York, NY, USA.
  * Collins-Thompson et al., [2017] Collins-Thompson, K., Hansen, P., and Hauff, C. (2017).  Search as learning (dagstuhl seminar 17092).  In Dagstuhl reports, volume 7. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  * Cranshaw et al., [2017] Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., and Monroy-Hernández, A. (2017).  Calendar. help: Designing a workflow-based scheduling agent with humans in the loop.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 2382–2393.
  * Craswell et al., [2020] Craswell, N., Mitra, B., Yilmaz, E., Campos, D., and Voorhees, E. M. (2020).  Overview of the TREC 2019 deep learning track.  CoRR.
  * Croft and Thompson, [1987] Croft, W. and Thompson, R. (1987).  I3r: A new approach to the design of document retrieval systems.  J. Am. Soc. Inf. Sci., 38:389–404.
  * Croft, [1978] Croft, W. B. (1978).  Organizing and searching large files of document descriptions.  PhD thesis, University of Cambridge.
  * Croft, [2019] Croft, W. B. (2019).  The importance of interaction for information retrieval.  In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1–2, New York, NY, USA. ACM.
  * Croft and Leouski, [1996] Croft, W. B. and Leouski, A. V. (1996).  An evaluation of techniques for clustering search results.  Computer Science Department Faculty Publication Series, page 36.
  * Croft et al., [2010] Croft, W. B., Metzler, D., and Strohman, T. (2010).  Search engines: Information retrieval in practice, volume 520.  Addison-Wesley Reading.
  * Culpepper et al., [2018] Culpepper, J. S., Diaz, F., and Smucker, M. (2018).  Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018).  SIGIR Forum, 52(1):34–90.
  * Cutting et al., [1992] Cutting, D., Pedersen, J. O., Karger, D., and Tukey, J. (1992).  Scatter/gather: a cluster-based approach to browsing large document collections.  In SIGIR ’92.
  * Dai et al., [2018] Dai, B., Shaw, A., He, N., Li, L., and Song, L. (2018).  Boosting the actor with dual critic.  In Proceedings of the Sixth International Conference on Learning Representations (ICLR).
  * Dalton et al., [2019] Dalton, J., Xiong, C., and Callan, J. (2019).  Overview of the trec 2019 conversational assistance track.
  * Dalton et al., [2020] Dalton, J., Xiong, C., and Callan, J. (2020).  Trec cast 2019: The conversational assistance track overview.  arXiv preprint arXiv:2003.13624.
  * Dalton et al., [2021] Dalton, J., Xiong, C., and Callan, J. (2021).  Trec cast 2020: The conversational assistance track overview.  TREC.
  * Danescu-Niculescu-Mizil et al., [2011] Danescu-Niculescu-Mizil, C., Gamon, M., and Dumais, S. (2011).  Mark my words! linguistic style accommodation in social media.  In Proceedings of the 20th international conference on World wide web, pages 745–754.
  * Dang, [2005] Dang, H. T. (2005).  Overview of duc 2005.  In Proceedings of the document understanding conference, volume 2005, pages 1–12.
  * Deerwester et al., [1990] Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990).  Indexing by latent semantic analysis.  Journal of the American society for information science, 41(6):391–407.
  * Devlin et al., [2018] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Dhingra et al., [2017] Dhingra, B., Li, L., Li, X., Gao, J., Chen, Y.-N., Ahmed, F., and Deng, L. (2017).  Towards end-to-end reinforcement learning of dialogue agents for information access.  In ACL (1), pages 484–495.
  * Dietz et al., [2017] Dietz, L., Verma, M., Radlinski, F., and Craswell, N. (2017).  Trec complex answer retrieval overview.  In TREC.
  * Dinan et al., [2019] Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., et al. (2019).  The second conversational intelligence challenge (convai2).  arXiv preprint arXiv:1902.00098.
  * Ding et al., [2020] Ding, Y. Q. Y., Liu, J., Liu, K., Ren, R., Zhao, X., Dong, D., Wu, H., and Wang, H. (2020).  Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering.  arXiv preprint arXiv:2010.08191.
  * Dong et al., [2019] Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. (2019).  Unified language model pre-training for natural language understanding and generation.  In Advances in Neural Information Processing Systems, pages 13063–13075.
  * Dumais et al., [2014] Dumais, S., Jeffries, R., Russell, D. M., Tang, D., and Teevan, J. (2014).  Understanding user behavior through log data and analysis.  In Ways of Knowing in HCI, pages 349–372. Springer.
  * Durmus et al., [2020] Durmus, E., He, H., and Diab, M. (2020).  Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070.
  * Elgohary et al., [2019] Elgohary, A., Peskov, D., and Boyd-Graber, J. L. (2019).  Can you unpack that? learning to rewrite questions-in-context.  In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of EMNLP, pages 5917–5923.
  * Elgohary et al., [2018] Elgohary, A., Zhao, C., and Boyd-Graber, J. (2018).  Dataset and baselines for sequential open-domain question answering.  In Empirical Methods in Natural Language Processing.
  * Erkan and Radev, [2004] Erkan, G. and Radev, D. R. (2004).  Lexrank: Graph-based lexical centrality as salience in text summarization.  Journal of artificial intelligence research, 22:457–479.
  * Fang et al., [2018] Fang, H., Cheng, H., Sap, M., Clark, E., Holtzman, A., Choi, Y., Smith, N. A., and Ostendorf, M. (2018).  Sounding board: A user-centric and content-driven social chatbot.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 96–100.
  * Fink et al., [2006] Fink, E., Bardak, U., Rothrock, B., and Carbonell, J. G. (2006).  Scheduling with uncertain resources: Collaboration with the user.  In 2006 IEEE International Conference on Systems, Man and Cybernetics, volume 1, pages 11–17. IEEE.
  * Fox et al., [2005] Fox, S., Karnawat, K., Mydland, M., Dumais, S., and White, T. (2005).  Evaluating implicit measures to improve web search.  ACM Transactions on Information Systems (TOIS), 23(2):147–168.
  * Freed et al., [2008] Freed, M., Carbonell, J. G., Gordon, G. J., Hayes, J., Myers, B. A., Siewiorek, D. P., Smith, S. F., Steinfeld, A., and Tomasic, A. (2008).  Radar: A personal assistant that learns to reduce email overload.  In AAAI, volume 8, pages 1287–1293.
  * French, [1999] French, R. M. (1999).  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135.
  * Gabriel et al., [2020] Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., and Gao, J. (2020).  Go figure! a meta evaluation of factuality in summarization.  arXiv preprint arXiv:2010.12834.
  * Gallistel, [1990] Gallistel, C. R. (1990).  The organization of learning. The MIT Press.
  * Gallistel and King, [2011] Gallistel, C. R. and King, A. P. (2011).  Memory and the computational brain: Why cognitive science will transform neuroscience, volume 6.  John Wiley & Sons.
  * Gao et al., [2019] Gao, J., Galley, M., Li, L., et al. (2019).  Neural approaches to conversational ai.  volume 13, pages 127–298. Now Publishers, Inc.
  * Gao et al., [2014] Gao, J., Pantel, P., Gamon, M., He, X., and Deng, L. (2014).  Modeling interestingness with deep neural networks.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2–13.
  * [100] Gao, J., Peng, B., Li, C., Li, J., Shayandeh, S., Liden, L., and Shum, H.-Y. (2020a).  Robust conversational ai with grounded text generation.  arXiv preprint arXiv:2009.03457.
  * [101] Gao, J., Xiong, C., and Bennett, P. (2020b).  Recent advances in conversational information retrieval.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2424.
  * [102] Gao, S., Chen, X., Ren, Z., Zhao, D., and Yan, R. (2020c).  From standard summarization to new tasks and beyond: Summarization with manifold information.  arXiv preprint arXiv:2005.04684.
  * Gasic et al., [2010] Gasic, M., Jurcicek, F., Keizer, S., Mairesse, F., Thomson, B., Yu, K., and Young, S. (2010).  Gaussian processes for fast policy optimisation of pomdp-based dialogue managers.  In Proceedings of the SIGDIAL 2010 Conference, pages 201–204.
  * Gašić et al., [2014] Gašić, M., Kim, D., Tsiakoulis, P., Breslin, C., Henderson, M., Szummer, M., Thomson, B., and Young, S. (2014).  Incremental on-line adaptation of pomdp-based dialogue managers to extended domains.  In Fifteenth Annual Conference of the International Speech Communication Association.
  * Gašic et al., [2014] Gašic, M., Kim, D., Tsiakoulis, P., Breslin, C., Henderson, M., Szummer, M., Thomson, B., and Young, S. (2014).  Incremental on-line adaptation of POMDP-based dialogue managers to extended domains.  In Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 140–144.
  * Gehrmann et al., [2018] Gehrmann, S., Deng, Y., and Rush, A. M. (2018).  Bottom-up abstractive summarization.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.
  * Ghandeharioun et al., [2019] Ghandeharioun, A., McDuff, D., Czerwinski, M., and Rowan, K. (2019).  Emma: an emotion-aware wellbeing chatbot.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–7. IEEE.
  * Goel et al., [2018] Goel, R., Paul, S., Chung, T., Lecomte, J., Mandal, A., and Hakkani-Tur, D. (2018).  Flexible and scalable state tracking framework for goal-oriented dialogue systems.  arXiv preprint arXiv:1811.12891.
  * Gong et al., [2012] Gong, Y., Lazebnik, S., Gordo, A., and Perronnin, F. (2012).  Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.  IEEE transactions on pattern analysis and machine intelligence, 35(12):2916–2929.
  * Gopalakrishnan et al., [2019] Gopalakrishnan, K., Hedayatnia, B., Chen, Q., Gottardi, A., Kwatra, S., Venkatesh, A., Gabriel, R., Hakkani-Tür, D., and AI, A. A. (2019).  Topical-chat: Towards knowledge-grounded open-domain conversations.  Proc. Interspeech 2019, pages 1891–1895.
  * Gu et al., [2017] Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., and Levine, S. (2017).  Q-Prop: Sample-efficient policy gradient with an off-policy critic.  In Proceedings of the 5th International Conference on Learning Representations (ICLR).
  * Gu et al., [2020] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. (2020).  Domain-specific language model pretraining for biomedical natural language processing.  arXiv preprint arXiv:2007.15779.
  * Gui et al., [2021] Gui, L., Wang, B., Huang, Q., Hauptmann, A., Bisk, Y., and Gao, J. (2021).  Kat: A knowledge augmented transformer for vision-and-language.  arXiv preprint arXiv:2112.08614.
  * Guo et al., [2018] Guo, D., Tang, D., Duan, N., Zhou, M., and Yin, J. (2018).  Dialog-to-action: conversational question answering over a large-scale knowledge base.  In Advances in Neural Information Processing Systems, pages 2942–2951.
  * Gupta et al., [2019] Gupta, S., Kohavi, R., Tang, D., Xu, Y., Andersen, R., Bakshy, E., Cardin, N., Chandran, S., Chen, N., Coey, D., et al. (2019).  Top challenges from the first practical online controlled experiments summit.  ACM SIGKDD Explorations Newsletter, 21(1):20–35.
  * Gupta et al., [2020] Gupta, S., Rawat, B. P. S., and Yu, H. (2020).  Conversational machine comprehension: a literature review.  arXiv preprint arXiv:2006.00671.
  * Guu et al., [2020] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.-W. (2020).  REALM: Retrieval-augmented language model pre-training.  arXiv preprint arXiv:2002.08909.
  * Guzzoni et al., [2007] Guzzoni, D., Baur, C., and Cheyer, A. (2007).  Modeling human-agent interaction with active ontologies.  In AAAI Spring Symposium: Interaction Challenges for Intelligent Assistants, pages 52–59.
  * Hadsell et al., [2020] Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. (2020).  Embracing change: Continual learning in deep neural networks.  Trends in cognitive sciences.
  * Ham et al., [2020] Ham, D., Lee, J.-G., Jang, Y., and Kim, K.-E. (2020).  End-to-end neural pipeline for goal-oriented dialogue system using gpt-2.  ACL.
  * He et al., [2013] He, K., Wen, F., and Sun, J. (2013).  K-means hashing: An affinity-preserving quantization method for learning binary compact codes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2938–2945.
  * He et al., [2020] He, P., Liu, X., Gao, J., and Chen, W. (2020).  Deberta: Decoding-enhanced bert with disentangled attention.  arXiv preprint arXiv:2006.03654.
  * Hearst, [2009] Hearst, M. (2009).  Search user interfaces.  Cambridge university press.
  * Hearst, [1995] Hearst, M. A. (1995).  Tilebars: visualization of term distribution information in full text information access.  In CHI ’95.
  * Heimonen and Jhaveri, [2005] Heimonen, T. and Jhaveri, N. (2005).  Visualizing query occurrence in search result lists.  Ninth International Conference on Information Visualisation (IV’05), pages 877–882.
  * Henderson et al., [2008] Henderson, J., Lemon, O., and Georgila, K. (2008).  Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets.  Computational Linguistics, 34(4):487–511.
  * Hermann et al., [2015] Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. (2015).  Teaching machines to read and comprehend.  In Advances in Neural Information Processing Systems, pages 1693–1701.
  * Hinton et al., [2015] Hinton, G., Vinyals, O., and Dean, J. (2015).  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531.
  * Hinton and Van Camp, [1993] Hinton, G. E. and Van Camp, D. (1993).  Keeping the neural networks simple by minimizing the description length of the weights.  In Proceedings of the sixth annual conference on Computational learning theory, pages 5–13. ACM.
  * Hoeber and Yang, [2006] Hoeber, O. and Yang, X. (2006).  A comparative user study of web search interfaces: Hotmap, concept highlighter, and google.  2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI’06), pages 866–874.
  * Hosseini-Asl et al., [2020] Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., and Socher, R. (2020).  A simple language model for task-oriented dialogue.  arXiv preprint arXiv:2005.00796.
  * Huang et al., [2018] Huang, H.-Y., Choi, E., and Yih, W.-t. (2018).  Flowqa: Grasping flow in history for conversational machine comprehension.  arXiv preprint arXiv:1810.06683.
  * Huang et al., [2013] Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., and Heck, L. (2013).  Learning deep structured semantic models for web search using clickthrough data.  In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management (CIKM 2013), pages 2333–2338. ACM.
  * Ingwersen, [1992] Ingwersen, P. (1992).  Information retrieval interaction, volume 246.  Taylor Graham London.
  * Ingwersen and Järvelin, [2006] Ingwersen, P. and Järvelin, K. (2006).  The turn: Integration of information seeking and retrieval in context, volume 18.  Springer Science & Business Media.
  * Ingwersen and Wormell, [1989] Ingwersen, P. and Wormell, I. (1989).  Modern indexing and retrieval techniques matching different types of information needs.  In International forum on information and documentation, volume 14, pages 17–22.
  * Iyyer et al., [2017] Iyyer, M., Yih, W.-t., and Chang, M.-W. (2017).  Search-based neural structured learning for sequential question answering.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831.
  * Izacard and Grave, [2020] Izacard, G. and Grave, E. (2020).  Leveraging passage retrieval with generative models for open domain question answering.  arXiv preprint arXiv:2007.01282.
  * Jegou et al., [2010] Jegou, H., Douze, M., and Schmid, C. (2010).  Product quantization for nearest neighbor search.  IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128.
  * Joachims, [2002] Joachims, T. (2002).  Optimizing search engines using clickthrough data.  In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133–142.
  * Johnson et al., [2017] Johnson, J., Douze, M., and Jégou, H. (2017).  Billion-scale similarity search with gpus.  arXiv preprint arXiv:1702.08734.
  * Joshi et al., [2017] Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017).  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  arXiv preprint arXiv:1705.03551.
  * Ju et al., [2019] Ju, Y., Zhao, F., Chen, S., Zheng, B., Yang, X., and Liu, Y. (2019).  Technical report on conversational question answering.  arXiv preprint arXiv:1909.10772.
  * Kahneman, [2011] Kahneman, D. (2011).  Thinking, fast and slow.  Macmillan.
  * Kakade, [2001] Kakade, S. (2001).  A natural policy gradient.  In Advances in Neural Information Processing Systems 13 (NIPS), pages 1531–1538.
  * Kamath and Das, [2018] Kamath, A. and Das, R. (2018).  A survey on semantic parsing.  arXiv preprint arXiv:1812.00978.
  * Karpukhin et al., [2020] Karpukhin, V., Oguz, B., Min, S., Lewis, P. S. H., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020).  Dense passage retrieval for open-domain question answering.  In Proceedings of EMNLP, pages 6769–6781.
  * Katharopoulos and Fleuret, [2018] Katharopoulos, A. and Fleuret, F. (2018).  Not all samples are created equal: Deep learning with importance sampling.  In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, pages 2530–2539. PMLR.
  * Keskar et al., [2019] Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. (2019).  Ctrl: A conditional transformer language model for controllable generation.  arXiv preprint arXiv:1909.05858.
  * [150] Khatri, C., Goel, R., Hedayatnia, B., Metanillou, A., Venkatesh, A., Gabriel, R., and Mandal, A. (2018a).  Contextual topic modeling for dialog systems.  In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 892–899. IEEE.
  * [151] Khatri, C., Hedayatnia, B., Venkatesh, A., Nunn, J., Pan, Y., Liu, Q., Song, H., Gottardi, A., Kwatra, S., Pancholi, S., et al. (2018b).  Advancing the state of the art in open domain dialog systems through the alexa prize.  arXiv preprint arXiv:1812.10757.
  * Kim et al., [2019] Kim, Y., Fourney, A., and Kamar, E. (2019).  Studying preferences and concerns about information disclosure in email notifications.  In The World Wide Web Conference, pages 874–885.
  * Kim et al., [2016] Kim, Y., Jernite, Y., Sontag, D., and Rush, A. (2016).  Character-aware neural language models.  In Proceedings of the AAAI conference on artificial intelligence, volume 30.
  * Kirkpatrick et al., [2017] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017).  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526.
  * Kiseleva et al., [2016] Kiseleva, J., Williams, K., Jiang, J., Hassan Awadallah, A., Crook, A. C., Zitouni, I., and Anastasakos, T. (2016).  Understanding user satisfaction with intelligent assistants.  In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, pages 121–130.
  * Knight and Marcu, [2000] Knight, K. and Marcu, D. (2000).  Statistics-based summarization-step one: Sentence compression.  AAAI/IAAI, 2000:703–710.
  * Kollar et al., [2018] Kollar, T., Berry, D., Stuart, L., Owczarzak, K., Chung, T., Mathias, L., Kayser, M., Snow, B., and Matsoukas, S. (2018).  The alexa meaning representation language.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 177–184.
  * Konda and Tsitsiklis, [1999] Konda, V. R. and Tsitsiklis, J. N. (1999).  Actor-critic algorithms.  In Advances in Neural Information Processing Systems 12 (NIPS), pages 1008–1014.
  * Kryściński et al., [2019] Kryściński, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. (2019).  Neural text summarization: A critical evaluation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540–551.
  * Kulkarni et al., [2020] Kulkarni, S., Chammas, S., Zhu, W., Sha, F., and Ie, E. (2020).  Aquamuse: Automatically generating datasets for query-based multi-document summarization.  arXiv preprint arXiv:2010.12694.
  * Kulkarni et al., [2021] Kulkarni, S., Chammas, S., Zhu, W., Sha, F., and Ie, E. (2021).  Comsum and sibert: A dataset and neural model for query-based multi-document summarization.  In International Conference on Document Analysis and Recognition, pages 84–98. Springer.
  * Kwiatkowski et al., [2019] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. (2019).  Natural questions: a benchmark for question answering research.  Transactions of the Association of Computational Linguistics.
  * Langkilde and Knight, [1998] Langkilde, I. and Knight, K. (1998).  Generation that exploits corpus-based statistical knowledge.  In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.
  * [164] Lee, K., Chang, M.-W., and Toutanova, K. (2019a).  Latent retrieval for weakly supervised open domain question answering.  In Proceedings of ACL, pages 6086–6096.
  * [165] Lee, S., Zhu, Q., Takanobu, R., Zhang, Z., Zhang, Y., Li, X., Li, J., Peng, B., Li, X., Huang, M., et al. (2019b).  Convlab: Multi-domain end-to-end dialog system platform.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 64–69.
  * Lesk and Salton, [1969] Lesk, M. E. and Salton, G. (1969).  Interactive search and retrieval methods using automatic information displays.  In Proceedings of the May 14-16, 1969, spring joint computer conference, pages 435–446. ACM.
  * Lewis et al., [2020] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020).  Retrieval-augmented generation for knowledge-intensive nlp tasks.  arXiv preprint arXiv:2005.11401.
  * Li et al., [2020] Li, C., Zhang, M., Andersen, D. G., and He, Y. (2020).  Improving approximate nearest neighbor search through learned adaptive early termination.  In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 2539–2554.
  * Li et al., [2016] Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., and Jurafsky, D. (2016).  Deep reinforcement learning for dialogue generation.  arXiv preprint arXiv:1606.01541.
  * Li et al., [2014] Li, L., He, H., and Williams, J. D. (2014).  Temporal supervised learning for inferring a dialog policy from example conversations.  In Proceedings of the 2014 IEEE Spoken Language Technology Workshop (SLT), pages 312–317.
  * Li et al., [2018] Li, R., Kahou, S., Schulz, H., Michalski, V., Charlin, L., and Pal, C. (2018).  Towards deep conversational recommendations.  arXiv preprint arXiv:1812.07617.
  * Li et al., [2017] Li, X., Chen, Y.-N., Li, L., Gao, J., and Celikyilmaz, A. (2017).  End-to-end task-completion neural dialogue systems.  In Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP), pages 733–743.
  * Lin, [2004] Lin, C.-Y. (2004).  ROUGE: A package for automatic evaluation of summaries.  In ACL workshop, pages 74–81.
  * [174] Lin, J., Nogueira, R., and Yates, A. (2020a).  Pretrained transformers for text ranking: Bert and beyond.  arXiv preprint arXiv:2010.06467.
  * [175] Lin, S.-C., Yang, J.-H., Nogueira, R., Tsai, M.-F., Wang, C.-J., and Lin, J. (2020b).  Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting.  arXiv preprint arXiv:2005.02230.
  * Lipton et al., [2018] Lipton, Z. C., Gao, J., Li, L., Li, X., Ahmed, F., and Deng, L. (2018).  BBQ-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems.  In AAAI, pages 5237–5244.
  * Liu and Lane, [2017] Liu, B. and Lane, I. (2017).  Iterative policy learning in end-to-end trainable task-oriented neural dialog models.  In Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 482–489.
  * [178] Liu, H., Feng, Y., Mao, Y., Zhou, D., Peng, J., and Liu, Q. (2018a).  Action-depedent control variates for policy optimization via Stein’s identity.  In Proceedings of the 6th International Conference on Learning Representations (ICLR).
  * [179] Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. (2018b).  Generating wikipedia by summarizing long sequences.  arXiv preprint arXiv:1801.10198.
  * [180] Liu, Q., Chen, B., Liu, H., Jian-Guang, L., Fang, L., Zhou, B., and Zhang, D. (2019a).  A split-and-recombine approach for follow-up query analysis.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5319–5329.
  * [181] Liu, Q., Chen, B., Lou, J.-G., Jin, G., and Zhang, D. (2019b).  Fanda: A novel approach to perform follow-up query analysis.  arXiv preprint arXiv:1901.08259.
  * Liu et al., [2009] Liu, T.-Y. et al. (2009).  Learning to rank for information retrieval.  Foundations and Trends® in Information Retrieval, 3(3):225–331.
  * [183] Liu, Y. and Lapata, M. (2019a).  Hierarchical transformers for multi-document summarization.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070–5081.
  * [184] Liu, Y. and Lapata, M. (2019b).  Text summarization with pretrained encoders.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740.
  * [185] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019c).  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * Luan et al., [2020] Luan, Y., Eisenstein, J., Toutanova, K., and Collins, M. (2020).  Sparse, dense, and attentional representations for text retrieval.  arXiv preprint arXiv:2005.00181.
  * Lucy and Gauthier, [2017] Lucy, L. and Gauthier, J. (2017).  Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning.  arXiv preprint arXiv:1705.11168.
  * Luhn, [1958] Luhn, H. P. (1958).  The automatic creation of literature abstracts.  IBM Journal of research and development, 2(2):159–165.
  * Mani and Maybury, [1999] Mani, I. and Maybury, M. T. (1999).  Advances in automatic text summarization mit press.
  * Marchionini, [2006] Marchionini, G. (2006).  Exploratory search: from finding to understanding.  Communications of the ACM, 49(4):41–46.
  * Marcus, [2020] Marcus, G. (2020).  The next decade in ai: four steps towards robust artificial intelligence.  arXiv preprint arXiv:2002.06177.
  * Marino et al., [2021] Marino, K., Chen, X., Parikh, D., Gupta, A., and Rohrbach, M. (2021).  Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111–14121.
  * Mehrotra et al., [2017] Mehrotra, R., Anderson, A., Diaz, F., Sharma, A., Wallach, H., and Yilmaz, E. (2017).  Auditing search engines for differential satisfaction across demographics.  In Proceedings of the 26th international conference on World Wide Web companion, pages 626–633.
  * Microsoft, [2018] Microsoft (2018).  Responsible bots: 10 guidelines for developers of conversational ai.
  * Mikolov et al., [2013] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013).  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * Min et al., [2020] Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. (2020).  Ambigqa: Answering ambiguous open-domain questions.  arXiv preprint arXiv:2004.10645.
  * Minsky, [1988] Minsky, M. (1988).  Society of mind.  Simon and Schuster.
  * Minsky, [2007] Minsky, M. (2007).  The emotion machine: Commonsense thinking, artificial intelligence, and the future of the human mind.  Simon and Schuster.
  * Mintz et al., [2009] Mintz, M., Bills, S., Snow, R., and Jurafsky, D. (2009).  Distant supervision for relation extraction without labeled data.  In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003–1011.
  * Mitra et al., [2018] Mitra, B., Craswell, N., et al. (2018).  An introduction to neural information retrieval.  Now Foundations and Trends.
  * Mnih et al., [2016] Mnih, V., Adrià, Badia, P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.  In Proceedings of the 33rd International Conference on Machine Learning (ICML), pages 1928–1937.
  * Mnih et al., [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015).  Human-level control through deep reinforcement learning.  Nature, 518:529–533.
  * Modi et al., [2004] Modi, P. J., Veloso, M., Smith, S. F., and Oh, J. (2004).  Cmradar: A personal assistant agent for calendar management.  In International Bi-Conference Workshop on Agent-Oriented Information Systems, pages 169–181. Springer.
  * Morais, [2019] Morais, B. (2019).  Can humans fall in love with bots?  The New Yorker, Annals of Technology.
  * Müller et al., [2019] Müller, T., Piccinno, F., Nicosia, M., Shaw, P., and Altun, Y. (2019).  Answering conversational questions on structured data without logical forms.  arXiv preprint arXiv:1908.11787.
  * Nallapati et al., [2016] Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., and Xiang, B. (2016).  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290.
  * Nema et al., [2017] Nema, P., Khapra, M. M., Laha, A., and Ravindran, B. (2017).  Diversity driven attention model for query-based abstractive summarization.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072.
  * Nguyen et al., [2016] Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. (2016).  MS MARCO: A human generated machine reading comprehension dataset.  arXiv preprint arXiv:1611.09268.
  * Nogueira and Cho, [2019] Nogueira, R. and Cho, K. (2019).  Passage re-ranking with bert.  arXiv preprint arXiv:1901.04085.
  * Nothman et al., [2008] Nothman, J., Curran, J. R., and Murphy, T. (2008).  Transforming wikipedia into named entity training data.  In Proceedings of the Australasian Language Technology Association Workshop 2008, pages 124–132.
  * Oddy, [1977] Oddy, R. N. (1977).  Information retrieval through man-machine dialogue.  Journal of documentation, 33(1):1–14.
  * Ohsugi et al., [2019] Ohsugi, Y., Saito, I., Nishida, K., Asano, H., and Tomita, J. (2019).  A simple but effective method to incorporate multi-turn context with bert for conversational machine comprehension.  arXiv preprint arXiv:1905.12848.
  * Ou and Lin, [2020] Ou, W. and Lin, Y. (2020).  A clarifying question selection system from ntes_along in convai3 challenge.  arXiv preprint arXiv:2010.14202.
  * Parisi et al., [2019] Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019).  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71.
  * Pasunuru et al., [2021] Pasunuru, R., Celikyilmaz, A., Galley, M., Xiong, C., Zhang, Y., Bansal, M., and Gao, J. (2021).  Data augmentation for abstractive query-focused multi-document summarization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13666–13674.
  * Pasupat and Liang, [2015] Pasupat, P. and Liang, P. (2015).  Compositional semantic parsing on semi-structured tables.  arXiv preprint arXiv:1508.00305.
  * Pathak et al., [2017] Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017).  Curiosity-driven exploration by self-supervised prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16–17.
  * Paul et al., [2019] Paul, S., Goel, R., and Hakkani-Tür, D. (2019).  Towards universal dialogue act tagging for task-oriented dialogues.  arXiv preprint arXiv:1907.03020.
  * Paulevé et al., [2010] Paulevé, L., Jégou, H., and Amsaleg, L. (2010).  Locality sensitive hashing: A comparison of hash function types and querying mechanisms.  Pattern recognition letters, 31(11):1348–1358.
  * [220] Peng, B., Li, C., Li, J., Shayandeh, S., Liden, L., and Gao, J. (2020a).  Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model.  arXiv preprint arXiv:2005.05298.
  * Peng et al., [2021] Peng, B., Li, C., Li, J., Shayandeh, S., Liden, L., and Gao, J. (2021).  Soloist: Buildingtask bots at scale with transfer learning and machine teaching.  Transactions of the Association for Computational Linguistics, 9:807–824.
  * [222] Peng, B., Li, C., Zhang, Z., Zhu, C., Li, J., and Gao, J. (2020b).  Raddle: An evaluation benchmark and analysis platform for robust task-oriented dialog systems.  arXiv preprint arXiv:2012.14666.
  * Peng et al., [2018] Peng, B., Li, X., Gao, J., Liu, J., and Wong, K.-F. (2018).  Deep dyna-q: Integrating planning for task-completion dialogue policy learning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2182–2192.
  * Peng et al., [2017] Peng, B., Li, X., Li, L., Gao, J., Celikyilmaz, A., Lee, S., and Wong, K.-F. (2017).  Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning.  In EMNLP, pages 2231–2240.
  * [225] Peng, B., Zhu, C., Li, C., Li, X., Li, J., Zeng, M., and Gao, J. (2020c).  Few-shot natural language generation for task-oriented dialog.  arXiv preprint arXiv:2002.12328.
  * Pennington et al., [2014] Pennington, J., Socher, R., and Manning, C. (2014).  GloVe: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * Peters et al., [2005] Peters, J., Vijayakumar, S., and Schaal, S. (2005).  Natural actor-critic.  In Proceedings of the 16th European Conference on Machine Learning (ECML), pages 280–291.
  * Pirolli and Card, [2005] Pirolli, P. and Card, S. (2005).  The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis.  In Proceedings of international conference on intelligence analysis, volume 5, pages 2–4. McLean, VA, USA.
  * Qin et al., [2019] Qin, L., Galley, M., Brockett, C., Liu, X., Gao, X., Dolan, W. B., Choi, Y., and Gao, J. (2019).  Conversing by reading: Contentful neural conversation with on-demand machine reading.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5427–5436.
  * Qu et al., [2020] Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., and Iyyer, M. (2020).  Open-retrieval conversational question answering.  In Proceedings of SIGIR, pages 539–548.
  * Qu et al., [2019] Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W. B., and Iyyer, M. (2019).  Attentive history selection for conversational question answering.  In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1391–1400.
  * Radev et al., [2002] Radev, D. R., Hovy, E., and McKeown, K. (2002).  Introduction to the special issue on summarization.  Computational linguistics, 28(4):399–408.
  * Radford et al., [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019).  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9.
  * Radlinski et al., [2019] Radlinski, F., Balog, K., Byrne, B., and Krishnamoorthi, K. (2019).  Coached conversational preference elicitation: A case study in understanding movie preferences.  In SIGdial.
  * Radlinski and Craswell, [2017] Radlinski, F. and Craswell, N. (2017).  A theoretical framework for conversational search.  In Proceedings of the 2017 conference on conference human information interaction and retrieval, pages 117–126. ACM.
  * Raffel et al., [2019] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019).  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683.
  * Rajpurkar et al., [2018] Rajpurkar, P., Jia, R., and Liang, P. (2018).  Know what you don’t know: Unanswerable questions for SQuAD.  arXiv 1806.03822.
  * Rajpurkar et al., [2016] Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016).  SQuAD: 100,000+ questions for machine comprehension of text.  arXiv preprint arXiv:1606.05250.
  * [239] Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q., Nunn, J., Hedayatnia, B., Cheng, M., Nagar, A., King, E., Bland, K., Wartick, A., Pan, Y., Song, H., Jayadevan, S., Hwang, G., and Pettigrue, A. (2018a).  Conversational AI: the science behind the alexa prize.  CoRR, abs/1801.03604.
  * [240] Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q., Nunn, J., Hedayatnia, B., Cheng, M., Nagar, A., King, E., Bland, K., Wartick, A., Pan, Y., Song, H., Jayadevan, S., Hwang, G., and Pettigrue, A. (2018b).  Conversational AI: the science behind the alexa prize.  CoRR, abs/1801.03604.
  * Ranzato et al., [2015] Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. (2015).  Sequence level training with recurrent neural networks.  arXiv preprint arXiv:1511.06732.
  * Rao and Daumé III, [2018] Rao, S. and Daumé III, H. (2018).  Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746.
  * Rao and Daumé III, [2019] Rao, S. and Daumé III, H. (2019).  Answer-based adversarial training for generating clarification questions.  arXiv preprint arXiv:1904.02281.
  * Reddy et al., [2018] Reddy, S., Chen, D., and Manning, C. D. (2018).  CoQA: A conversational question answering challenge.  arXiv preprint arXiv:1808.07042.
  * Reddy et al., [2019] Reddy, S., Chen, D., and Manning, C. D. (2019).  Coqa: A conversational question answering challenge.  Transactions of the Association for Computational Linguistics, 7:249–266.
  * [246] Ren, G., Ni, X., Malik, M., and Ke, Q. (2018a).  Conversational query understanding using sequence to sequence modeling.  In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1715–1724. International World Wide Web Conferences Steering Committee.
  * [247] Ren, L., Xie, K., Chen, L., and Yu, K. (2018b).  Towards universal dialogue state tracking.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2780–2786.
  * Roberts et al., [2020] Roberts, A., Raffel, C., and Shazeer, N. (2020).  How much knowledge can you pack into the parameters of a language model?  arXiv preprint arXiv:2002.08910.
  * Robertson and Zaragoza, [2009] Robertson, S. and Zaragoza, H. (2009).  The probabilistic relevance framework: BM25 and beyond.  Now Publishers Inc.
  * Robertson, [1977] Robertson, S. E. (1977).  The probability ranking principle in ir.  Journal of documentation.
  * Robertson and Jones, [1976] Robertson, S. E. and Jones, K. S. (1976).  Relevance weighting of search terms.  Journal of the American Society for Information science, 27(3):129–146.
  * Rocchio, [1971] Rocchio, J. (1971).  Relevance feedback in information retrieval.  The Smart retrieval system-experiments in automatic document processing, pages 313–323.
  * Roller et al., [2020] Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020).  Open-domain conversational agents: Current progress, open problems, and future directions.  arXiv preprint arXiv:2006.12442.
  * Rosset et al., [2020] Rosset, C., Xiong, C., Song, X., Campos, D., Craswell, N., Tiwary, S., and Bennett, P. (2020).  Leading conversational search by suggesting useful questions.  In Proceedings of The Web Conference 2020, pages 1160–1170.
  * Rush et al., [2015] Rush, A. M., Chopra, S., and Weston, J. (2015).  A neural attention model for abstractive sentence summarization.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.
  * Russo et al., [2018] Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., and Wen, Z. (2018).  A tutorial on Thompson sampling.  Foundations and Trends in Machine Learning, 11(1):1–96.
  * Saeidi et al., [2018] Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., and Riedel, S. (2018).  Interpretation of natural language rules in conversational machine reading.  In Proceedings of EMNLP, pages 2087–2097.
  * Saha et al., [2018] Saha, A., Pahuja, V., Khapra, M. M., Sankaranarayanan, K., and Chandar, S. (2018).  Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph.  arXiv preprint arXiv:1801.10314.
  * Schick and Schütze, [2020] Schick, T. and Schütze, H. (2020).  It’s not just size that matters: Small language models are also few-shot learners.  arXiv preprint arXiv:2009.07118.
  * SchManning, Christopher D and Raghavan, Prabhakar and ütze, [2008] SchManning, Christopher D and Raghavan, Prabhakar and ütze, H. (2008).  Introduction to information retrieval.  Cambridge University Press Cambridge.
  * Schneiderman, [1992] Schneiderman, B. (1992).  Designing the user interface. strategies for effective human-computer interaction.
  * [262] Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. (2015a).  Trust region policy optimization.  In Proceedings of the Thirty-Second International Conference on Machine Learning (ICML), pages 1889–1897.
  * [263] Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2015b).  High-dimensional continuous control using generalized advantage estimation.  arXiv:1506.02438.
  * Scialom et al., [2019] Scialom, T., Lamprier, S., Piwowarski, B., and Staiano, J. (2019).  Answers unite! unsupervised metrics for reinforced summarization models.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246–3256.
  * See et al., [2017] See, A., Liu, P. J., and Manning, C. D. (2017).  Get to the point: Summarization with pointer-generator networks.  arXiv preprint arXiv:1704.04368.
  * Sennrich et al., [2015] Sennrich, R., Haddow, B., and Birch, A. (2015).  Neural machine translation of rare words with subword units.  arXiv preprint arXiv:1508.07909.
  * Sennrich et al., [2016] Sennrich, R., Haddow, B., and Birch, A. (2016).  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725.
  * Seo et al., [2016] Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2016).  Bidirectional attention flow for machine comprehension.  arXiv preprint arXiv:1611.01603.
  * Shah, [2006] Shah, H. (2006).  Alice: an ace in digitaland.  tripleC: Communication, Capitalism & Critique. Open Access Journal for a Global Sustainable Information Society, 4(2):284–292.
  * Shen et al., [2014] Shen, Y., He, X., Gao, J., Deng, L., and Mesnil, G. (2014).  A latent semantic model with convolutional-pooling structure for information retrieval.  In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pages 101–110.
  * Shen et al., [2016] Shen, Y., Huang, P., Chang, M., and Gao, J. (2016).  Implicit ReasoNet: Modeling large-scale structured relationships with shared memory.  CoRR, abs/1611.04642.
  * Shneiderman et al., [1997] Shneiderman, B., Byrd, D., and Croft, W. B. (1997).  Clarifying search: A user-interface framework for text searches.  D-lib magazine, 3(1):18–20.
  * Shukla et al., [2020] Shukla, S., Liden, L., Shayandeh, S., Kamal, E., Li, J., Mazzola, M., Park, T., Peng, B., and Gao, J. (2020).  Conversation learner-a machine teaching tool for building dialog managers for task-oriented dialog systems.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 343–349.
  * Silverstein et al., [1999] Silverstein, C., Marais, H., Henzinger, M., and Moricz, M. (1999).  Analysis of a very large web search engine query log.  In Acm sigir forum, volume 33, pages 6–12. ACM New York, NY, USA.
  * Singh and Joachims, [2018] Singh, A. and Joachims, T. (2018).  Fairness of exposure in rankings.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2219–2228.
  * Singh et al., [2010] Singh, S., Lewis, R. L., Barto, A. G., and Sorg, J. (2010).  Intrinsically motivated reinforcement learning: An evolutionary perspective.  IEEE Transactions on Autonomous Mental Development, 2(2):70–82.
  * Sordoni et al., [2015] Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Grue Simonsen, J., and Nie, J. (2015).  A hierarchical recurrent encoder-decoder for generative context-aware query suggestion.  In CIKM.
  * Speer et al., [2017] Speer, R., Chin, J., and Havasi, C. (2017).  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Proceedings of AAAI, pages 4444–4451.
  * Stolcke et al., [2000] Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., and Meteer, M. (2000).  Dialogue act modeling for automatic tagging and recognition of conversational speech.  Computational linguistics, 26(3):339–373.
  * [280] Su, P.-H., Gasic, M., Mrkšić, N., Barahona, L. M. R., Ultes, S., Vandyke, D., Wen, T.-H., and Young, S. (2016a).  On-line active reward learning for policy optimisation in spoken dialogue systems.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2431–2441.
  * [281] Su, P.-H., Gašić, M., Mrkšić, N., Rojas-Barahona, L., Ultes, S., Vandyke, D., Wen, T.-H., and Young, S. (2016b).  Continuously learning neural dialogue management.  arXiv preprint: 1606.02689.
  * Su et al., [2018] Su, S.-Y., Li, X., Gao, J., Liu, J., and Chen, Y.-N. (2018).  Discriminative deep dyna-q: Robust planning for dialogue policy learning.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3813–3823.
  * Sutcliffe and Ennis, [1998] Sutcliffe, A. and Ennis, M. (1998).  Towards a cognitive theory of information retrieval.  Interacting with computers, 10(3):321–351.
  * Sutskever et al., [2014] Sutskever, I., Vinyals, O., and Le, Q. V. (2014).  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112.
  * Sutton, [1990] Sutton, R. S. (1990).  Integrated architectures for learning, planning, and reacting based on approximating dynamic programming.  In Proceedings of the seventh international conference on machine learning, pages 216–224.
  * Sutton and Barto, [2018] Sutton, R. S. and Barto, A. G. (2018).  Reinforcement Learning: An Introduction.  MIT Press, 2nd edition edition.
  * Sutton et al., [1999] Sutton, R. S., McAllester, D., Singh, S. P., and Mansour, Y. (1999).  Policy gradient methods for reinforcement learning with function approximation.  In Advances in Neural Information Processing Systems 12 (NIPS), pages 1057–1063.
  * Tang et al., [2018] Tang, D., Li, X., Gao, J., Wang, C., Li, L., and Jebara, T. (2018).  Subgoal discovery for hierarchical dialogue policy learning.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2298––2309.
  * Tang et al., [2019] Tang, J., Zhao, T., Xiong, C., Liang, X., Xing, E. P., and Hu, Z. (2019).  Target-guided open-domain conversation.  arXiv preprint arXiv:1905.11553.
  * Tenney et al., [2019] Tenney, I., Das, D., and Pavlick, E. (2019).  Bert rediscovers the classical nlp pipeline.  arXiv preprint arXiv:1905.05950.
  * Thomas et al., [2018] Thomas, P., Czerwinski, M., McDuff, D., Craswell, N., and Mark, G. (2018).  Style and alignment in information-seeking conversation.  In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, pages 42–51.
  * Thomas et al., [2017] Thomas, P., McDuff, D., Czerwinski, M., and Craswell, N. (2017).  Misc: A data set of information-seeking conversations.  In 1st SIGIR workshop on conversational approaches to information retrieval, volume 5.
  * Thompson, [1933] Thompson, W. R. (1933).  On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.  Biometrika, 25(3–4):285–294.
  * Tombros et al., [1998] Tombros, A., Sanderson, M., and Gray, P. (1998).  Advantages of query biased summaries in information retrieval.  In SIGIR, volume 98, pages 2–10.
  * Trippas et al., [2018] Trippas, J. R., Spina, D., Cavedon, L., Joho, H., and Sanderson, M. (2018).  Informing the design of spoken conversational search: Perspective paper.  In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, pages 32–41.
  * Trippas et al., [2017] Trippas, J. R., Spina, D., Cavedon, L., and Sanderson, M. (2017).  How do people interact in conversational speech-only search tasks: A preliminary analysis.  In Conference on conference human information interaction and retrieval.
  * Vakulenko et al., [2021] Vakulenko, S., Longpre, S., Tu, Z., and Anantha, R. (2021).  Question rewriting for conversational question answering.  In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355–363.
  * Vasilyev et al., [2020] Vasilyev, O., Dharnidharka, V., and Bohannon, J. (2020).  Fill in the blanc: Human-free quality estimation of document summaries.  In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11–20.
  * Vaswani et al., [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017).  Attention is all you need.  In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
  * Voorhees et al., [2005] Voorhees, E. M., Harman, D. K., et al. (2005).  Trec: Experiment and evaluation in information retrieval.  63\.
  * Voskarides et al., [2020] Voskarides, N., Li, D., Ren, P., Kanoulas, E., and de Rijke, M. (2020).  Query resolution for conversational search with limited supervision.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 921–930.
  * Vtyurina et al., [2017] Vtyurina, A., Savenkov, D., Agichtein, E., and Clarke, C. L. (2017).  Exploring conversational search with humans, assistants, and wizards.  In Proceedings of the 2017 chi conference extended abstracts on human factors in computing systems, pages 2187–2193.
  * Wallace, [2009] Wallace, R. S. (2009).  The Anatomy of A.L.I.C.E., pages 181–210.  Springer Netherlands, Dordrecht.
  * Wan, [2008] Wan, X. (2008).  Using only cross-document relationships for both generic and topic-focused multi-document summarizations.  Information Retrieval, 11(1):25–49.
  * Wang and Bramer, [2006] Wang, X. and Bramer, M. (2006).  Exploring web search results clustering.  In International Conference on Innovative Techniques and Applications of Artificial Intelligence, pages 393–397. Springer.
  * Weizenbaum, [1966] Weizenbaum, J. (1966).  ELIZA: a computer program for the study of natural language communication between man and machine.  Commun. ACM, 9(1):36–45.
  * Wen et al., [2015] Wen, T.-H., Gasic, M., Mrksic, N., Su, P.-H., Vandyke, D., and Young, S. (2015).  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  arXiv preprint arXiv:1508.01745.
  * Wen et al., [2017] Wen, T.-H., Vandyke, D., Mrkšić, N., Gasic, M., Barahona, L. M. R., Su, P.-H., Ultes, S., and Young, S. (2017).  A network-based end-to-end trainable task-oriented dialogue system.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438–449.
  * White and Roth, [2009] White, R. W. and Roth, R. A. (2009).  Exploratory search: Beyond the query-response paradigm.  Synthesis lectures on information concepts, retrieval, and services, 1(1):1–98.
  * Williams et al., [2017] Williams, J. D., Asadi, K., and Zweig, G. (2017).  Hybrid code networks: Practical and efficient end-to-end dialog control with supervised and reinforcement learning.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pages 665–677.
  * Williams, [1992] Williams, R. J. (1992).  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 8:229–256.
  * Wu et al., [2018] Wu, B., Xiong, C., Sun, M., and Liu, Z. (2018).  Query suggestion with feedback memory network.  In Proceedings of the 2018 World Wide Web Conference, pages 1563–1571.
  * Wu et al., [2015] Wu, J., Li, M., and Lee, C.-H. (2015).  A probabilistic framework for representing dialog systems and entropy-based dialog management through dynamic stochastic state evolution.  IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 23(11):2026–2035.
  * Wu et al., [2010] Wu, Q., Burges, C. J., Svore, K. M., and Gao, J. (2010).  Adapting boosting for information retrieval measures.  Information Retrieval, 13(3):254–270.
  * [315] Wu, W., Guo, Z., Zhou, X., Wu, H., Zhang, X., Lian, R., and Wang, H. (2019a).  Proactive human-machine conversation with explicit conversation goals.  arXiv preprint arXiv:1906.05572.
  * [316] Wu, Y., Li, X., Liu, J., Gao, J., and Yang, Y. (2019b).  Switch-based active deep dyna-q: Efficient adaptive planning for task-completion dialogue policy learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7289–7296.
  * Wu et al., [2016] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. (2016).  Google’s neural machine translation system: Bridging the gap between human and machine translation.  CoRR, abs/1609.08144.
  * Wu et al., [2017] Wu, Y., Wu, W., Xing, C., Zhou, M., and Li, Z. (2017).  Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496–505.
  * Wu et al., [2021] Wu, Z., Galley, M., Brockett, C., Zhang, Y., Gao, X., Quirk, C., Koncel-Kedziorski, R., Gao, J., Hajishirzi, H., Ostendorf, M., et al. (2021).  A controllable model of grounded response generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14085–14093.
  * Xie et al., [2021] Xie, H., Liu, Z., Xiong, C., Liu, Z., and Copestake, A. (2021).  Tiage: A benchmark for topic-shift aware dialog modeling.  arXiv preprint arXiv:2109.04562.
  * Xiong and Callan, [2015] Xiong, C. and Callan, J. (2015).  Query expansion with freebase.  In Proceedings of the 2015 international conference on the theory of information retrieval, pages 111–120.
  * Xiong et al., [2017] Xiong, C., Dai, Z., Callan, J., Liu, Z., and Power, R. (2017).  End-to-end neural ad-hoc ranking with kernel pooling.  In Proceedings of the 40th annual international ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2017), pages 55–64. ACM.
  * Xiong et al., [2021] Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P. N., Ahmed, J., and Overwikj, A. (2021).  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In Proceedings of ICLR.
  * Xu et al., [2021] Xu, H., Zhengyan, Z., Ning, D., Yuxian, G., Xiao, L., Yuqi, H., Jiezhong, Q., Liang, Z., Wentao, H., Minlie, H., et al. (2021).  Pre-trained models: Past, present and future.  arXiv preprint arXiv:2106.07139.
  * Xu et al., [2019] Xu, J., Wang, Y., Tang, D., Duan, N., Yang, P., Zeng, Q., Zhou, M., and Sun, X. (2019).  Asking clarification questions in knowledge-based question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1618–1629, Hong Kong, China. Association for Computational Linguistics.
  * Xu and Lapata, [2020] Xu, Y. and Lapata, M. (2020).  Coarse-to-fine query focused multi-document summarization.  In Proceedings of the 2020 Conference on empirical methods in natural language processing (EMNLP), pages 3632–3645.
  * Yang et al., [2019] Yang, J.-H., Lin, S.-C., Lin, J., Tsai, M.-F., and Wang, C.-J. (2019).  Query and answer expansion from conversation history.
  * Yang et al., [2015] Yang, Y., Yih, W.-t., and Meek, C. (2015).  Wikiqa: A challenge dataset for open-domain question answering.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013–2018.
  * Yao et al., [2013] Yao, X., Van Durme, B., Callison-Burch, C., and Clark, P. (2013).  Answer extraction as sequence tagging with tree edit distance.  In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 858–867.
  * Yatskar, [2018] Yatskar, M. (2018).  A qualitative comparison of coqa, squad 2.0 and quac.  arXiv preprint arXiv:1809.10735.
  * Yavuz et al., [2019] Yavuz, S., Rastogi, A., Chao, G.-L., and Hakkani-Tur, D. (2019).  Deepcopy: Grounded response generation with hierarchical pointer networks.  arXiv preprint arXiv:1908.10731.
  * Yeh and Chen, [2019] Yeh, Y.-T. and Chen, Y.-N. (2019).  Flowdelta: Modeling flow information gain in reasoning for conversational machine comprehension.  arXiv preprint arXiv:1908.05117.
  * Yih et al., [2015] Yih, S. W.-t., Chang, M.-W., He, X., and Gao, J. (2015).  Semantic parsing via staged query graph generation: Question answering with knowledge base.
  * Young et al., [2013] Young, S., Gašić, M., Thomson, B., and Williams, J. D. (2013).  POMDP-based statistical spoken dialog systems: A review.  Proceedings of the IEEE, 101(5):1160–1179.
  * [335] Yu, D., Cohn, M., Yang, Y. M., Chen, C. Y., Wen, W., Zhang, J., Zhou, M., Jesse, K., Chau, A., Bhowmick, A., et al. (2019a).  Gunrock: A social bot for complex and engaging long conversations.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 79–84.
  * Yu et al., [2020] Yu, S., Liu, J., Yang, J., Xiong, C., Bennett, P., Gao, J., and Liu, Z. (2020).  Few-shot generative conversational query rewriting.  In SIGIR 2020.
  * [337] Yu, S., Liu, Z., Xiong, C., and Liu, Z. (2021a).  Few-shot conversational dense retrieval.  In SIGIR 2021.
  * [338] Yu, T., Zhang, R., Er, H. Y., Li, S., Xue, E., Pang, B., Lin, X. V., Tan, Y. C., Shi, T., Li, Z., et al. (2019b).  Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases.  arXiv preprint arXiv:1909.05378.
  * [339] Yu, T., Zhang, R., Polozov, O., Meek, C., and Awadallah, A. H. (2021b).  SCoRe: Pre-training for context representation in conversational semantic parsing.  In Submitted to International Conference on Learning Representations.  Under review.
  * Yu et al., [2018] Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018).  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  arXiv preprint arXiv:1809.08887.
  * [341] Yu, T., Zhang, R., Yasunaga, M., Tan, Y. C., Lin, X. V., Li, S., Er, H., Li, I., Pang, B., Chen, T., et al. (2019c).  Sparc: Cross-domain semantic parsing in context.  arXiv preprint arXiv:1906.02285.
  * Zamani and Craswell, [2020] Zamani, H. and Craswell, N. (2020).  Macaw: An extensible conversational information seeking platform.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 2193–2196. Association for Computing Machinery.
  * [343] Zamani, H., Dumais, S., Craswell, N., Bennett, P., and Lueck, G. (2020a).  Generating clarifying questions for information retrieval.  In The Web Conference 2020 (formerly WWW conference).
  * [344] Zamani, H., Lueck, G., Chen, E., Quispe, R., Luu, F., and Craswell, N. (2020b).  Mimics: A large-scale data collection for search clarification.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3189–3196.
  * [345] Zamani, H., Mitra, B., Chen, E., Lueck, G., Diaz, F., Bennett, P. N., Craswell, N., and Dumais, S. T. (2020c).  Analyzing and learning from user interactions for search clarification.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1181–1190.
  * [346] Zhang, H., Liu, Z., Xiong, C., and Liu, Z. (2019a).  Grounded conversation generation as guided traverses in commonsense knowledge graphs.  arXiv preprint arXiv:1911.02707.
  * [347] Zhang, H., Song, X., Xiong, C., Rosset, C., Bennett, P. N., Craswell, N., and Tiwary, S. (2019b).  Generic intent representation in web search.  In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 65–74.
  * [348] Zhang, J., Chang, J. P., Danescu-Niculescu-Mizil, C., Dixon, L., Hua, Y., Thain, N., and Taraborelli, D. (2018a).  Conversations gone awry: Detecting early signs of conversational failure.  arXiv preprint arXiv:1805.05345.
  * Zhang and Balog, [2020] Zhang, S. and Balog, K. (2020).  Evaluating conversational recommender systems via user simulation.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1512–1520.
  * [350] Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. (2018b).  Personalizing dialogue agents: I have a dog, do you have pets too?  arXiv preprint arXiv:1801.07243.
  * Zhang, [2019] Zhang, Y. (2019).  Evaluating the factual correctness for abstractive summarization.
  * [352] Zhang, Z., Li, X., Gao, J., and Chen, E. (2019c).  Budgeted policy learning for task-oriented dialogue systems.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3742–3751.
  * Zhao et al., [2021] Zhao, M., Yan, S., Liu, B., Zhong, X., Hao, Q., Chen, H., Niu, D., Long, B., and Guo, W. (2021).  Qbsum: A large-scale query-based document summarization dataset from real-world applications.  Computer Speech & Language, 66:101166.
  * Zhou et al., [2018] Zhou, H., Young, T., Huang, M., Zhao, H., Xu, J., and Zhu, X. (2018).  Commonsense knowledge aware conversation generation with graph attention.  In Proceedings of IJCAI, pages 4623–4629.
  * Zhou et al., [2020] Zhou, L., Gao, J., Li, D., and Shum, H.-Y. (2020).  The design and implementation of xiaoice, an empathetic social chatbot.  Computational Linguistics, 46(1):53–93.
  * Zhou et al., [2017] Zhou, Q., Yang, N., Wei, F., and Zhou, M. (2017).  Selective encoding for abstractive sentence summarization.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1095–1104.
  * Zhu et al., [2018] Zhu, C., Zeng, M., and Huang, X. (2018).  Sdnet: Contextualized attention-based deep network for conversational question answering.  arXiv preprint arXiv:1812.03593.
  * Zhu et al., [2020] Zhu, Q., Zhang, Z., Fang, Y., Li, X., Takanobu, R., Li, J., Peng, B., Gao, J., Zhu, X., and Huang, M. (2020).  Convlab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 142–149.