# MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents

Song Feng11footnotemark: 1
IBM Research AI
<EMAIL_ADDRESS>Siva Sankalp Patel11footnotemark: 1
IBM Research AI
<EMAIL_ADDRESS>Hui Wan
IBM Research AI
<EMAIL_ADDRESS>Sachindra Joshi
IBM Research AI
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

We propose MultiDoc2Dial, a new task and dataset on modeling goal-oriented
dialogues grounded in multiple documents. Most previous works treat document-
grounded dialogue modeling as a machine reading comprehension task based on a
single given document or passage. In this work, we aim to address more
realistic scenarios where a goal-oriented information-seeking conversation
involves multiple topics, and hence is grounded on different documents. To
facilitate such a task, we introduce a new dataset that contains dialogues
grounded in multiple documents from four different domains. We also explore
modeling the dialogue-based and document-based context in the dataset. We
present strong baseline approaches and various experimental results, aiming to
support further research efforts on such a task.

##  1 Introduction

With the recent advancements in NLP, there has been a surge of research
interests and efforts in developing conversational systems for various
domains. An important task in the field is conversational question answering
and document-grounded dialogue modeling. Prior work typically formulates the
task as a machine reading comprehension task assuming the associated document
or text snippet is given, such as QuAC (Choi et al., 2018), ShARC (Saeidi et
al., 2018), CoQA (Reddy et al., 2019), OR-QuAC (Qu et al., 2020a) and Doc2Dial
(Feng et al., 2020b). However, such task setup neglects the common real-life
scenarios where a goal-oriented conversation could correspond to several sub-
goals that are addressed in different documents. In this work, we propose a
new task and dataset, MultiDoc2Dial, on modeling goal-oriented dialogues that
are grounded in multiple documents.

We illustrate the proposed task in Figure 1. It includes a goal-oriented
dialogue with four segments on the left and three relevant documents on the
right. Each dialogue segment indicates that all turns within it are grounded
in a same document, e.g., turns from A3 to A7 in Seg-2 are all grounded in
Doc-2. The blue dashed lines connect a dialogue turn with its corresponding
relevant span in a document. The red dotted lines with arrows indicate that
the dialogue flow shifts among the grounding documents through the
conversation, i.e., Doc-1 $\to$ Doc-2 $\to$ Doc-1 $\to$ Doc-3. This example
highlights certain challenges for identifying the relevant grounding content
among different documents dynamically in a conversation. For instance, agent
response A2 mentions ‘insured’ as an important condition based on a span in
Doc-1. However, there are no more details about ‘insured’ in Doc-1. To further
discuss about ‘insured‘, the conversation naturally switches to refer to
another document Doc-2. Another challenge is to handle deep dialogue context.
For instance, to provide response to U4 or U6 in Seg-2, the agent needs to
understand the context of ‘disability benefit’ mentioned in Seg-1. There are
also cases such as Seg-4 where its turn U10 starts a new question related to
Doc-3 but seems independent of the previous segments. The task goal in the
example is naturally simple, but it still reveals the realistic expectation of
document-grounded dialogue modeling that is still yet to be met.

Figure 1:  A sample goal-oriented dialogue (on the left) that is grounded in
three relevant documents (on the right).

To the best of our knowledge, there is no existing task or dataset that
addresses the scenarios where the grounding documents of goal-oriented
dialogues are unknown and dynamic. To facilitate the study in this direction,
we introduce a new dataset that contains conversations that are grounded in
multiple documents. To construct the dialogue flows that involve multiple
documents, we derive a new approach from the data collection pipeline proposed
in Feng et al. (2020a). The newly composed dialogue flows have multiple
dialogue segments where two adjacent segments are grounded in different
documents. We harvest original utterances whenever possible and collect new
utterances when necessary. The dataset includes about 4800 dialogues with an
average of 14 turns that are grounded in 488 documents from four different
domains.

Inspired by recent advances in open retrieval question answering (QA) tasks
(Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020; Lewis et al.,
2020; Khattab et al., 2020a), we develop baseline models based on the
retriever-reader architecture (Karpukhin et al., 2020; Lewis et al., 2020).
Compared to the existing open retrieval QA (Lee et al., 2019; Min et al.,
2020) and open retrieval conversational QA tasks (Qu et al., 2020b; Li et al.,
2021), our dataset contains more complex and diverse dialogue scenarios based
on diversified documents from multiple domains. To work towards modeling the
interconnected contexts from dialogues and documents, we utilize document and
dialog-based structure information. For the former, we segment a document into
passages while maintaining its hierarchical contextual information. For the
latter, in addition to combining current turn and dialogue history (Qu et al.,
2020b, 2021), we also experiment with different ways to encode the current
turn separately based on the intuition that the latest turn, with a change in
topic, could be semantically distant from the dialogue history. We also
explore different retriever settings in our experiments.

We propose two tasks for modeling dialogues that are grounded in multiple
documents. One is to generate the grounding document span; and the other is to
generate agent response given current turn, dialogue history and a set of
documents. We evaluate the performances of the retriever and generator in
baseline models trained on MultiDoc2Dial dataset.

Data&Task | OB | GO | GA | FD | MD
---|---|---|---|---|---
ShARC | ✗ | ✓ | ✗ | ✗ | ✗
DoQA | ✗ | ✓ | ✗ | ✓ | ✗
Doc2Dial | ✗ | ✓ | ✓ | ✓ | ✗
QuAC | ✗ | ✗ | ✗ | ✓ | ✗
OR-QuAC | ✓ | ✗ | ✗ | ✓ | ✗
MultiDoc2Dial | ✓ | ✓ | ✓ | ✓ | ✓
Table 1: Comparison among selected datasets and tasks on document-grounded
dialogue and conversational QA on different aspects: open-book (OB), goal-
oriented (GO), grounding available (GA), full-document (FD), multiple-document
(MD).

We summarize our contributions as follows:

  * •

We propose a novel task and dataset, called MultiDoc2Dial, on modeling goal-
oriented dialogues that are grounded in multiple documents from various
domains. We aim to challenge recent advances in dialogue modeling with more
realistic scenarios that is hardly addressed in prior work.

  * •

We present strong baseline approaches and evaluations for two tasks based on
MultiDoc2Dial dataset in hope to support future significant research effort in
the direction. Our data and code are available at
https://doc2dial.github.io/multidoc2dial/.

##  2 Multi-Document-Grounded Dialogue

###  2.1 Data

We present MultiDoc2Dial, a new dataset that contains 4796 conversations with
an average of 14 turns grounded in 488 documents from four domains. This
dataset is constructed based on Doc2Dial dataset V1.0.1
111https://doc2dial.github.io/data.html. MultiDoc2Dial shares the same set of
annotations as Doc2Dial. For document data, it includes HTML mark-ups such as
list, title and document section information as shown in Figure 1. For
dialogue data, each dialogue turn is annotated with role, dialogue act, human-
generated utterance and the grounding span with document information. Each
dialogue contains one or multiple segment where each indicates that all turns
within one segment are grounded in a same document. For instance, the dialogue
in Figure 1 has four segments that are grounded in three documents.

We exclude the ‘irrelevant’ scenarios in Doc2Dial where the user question is
unanswerable and leave it for future work. We also filter out certain
dialogues when we identify more than four noisy turns per dialogue. There is a
total of 61078 dialogue turns in MultiDoc2Dial dataset, which consists of 38%
user questions, 12% agent follow-up questions and the rest as responding
turns. Table 2 shows the statistics of the dataset by domain, including the
number of dialogues with two segments (two-seg), more than two segments (>two-
seg), and no segmentations (single).

To create the data, we derive a new data construction approach from the
pipelined framework by Feng et al. (2020a). We first create dialogue flows
that correspond to multiple documents and then re-collect the utterances for
certain turns based on dialogue scenes in the given flow via crowdsourcing. In
addition, we aim to reuse previous turns from doc2dial dataset wherever
possible and collect new turns when necessary to compose the new dialogues.

####  2.1.1 Dialogue Flow

To construct dialogue flows grounded in multiple documents, we need to split
the existing dialogues into segments and recompose them. The main idea is to
identify the position where the previous topic can possibly end and then find
a segment with a new topic that is grounded in a different document, for which
we utilize both document-based and dialogue-based structure knowledge.

##### Dialogue Segmentation

To segment dialogues, we identify all the candidate splitting positions based
on dialogue act and turn index. Intuitively, we aim to maintain semantic
coherence of a dialogue segment (Mele et al., 2020). Thus, we only split after
an agent turn with dialogue act as ‘responding with an answer’ while the next
turn is not ‘asking a follow-up question’. We randomly select a number of
splitting positions per existing dialogue and obtain 2 to 4 segments per
dialogue.

##### Document Transition

To simulate the document-level topic shift between dialogue segments, we
identify different types of grounding document transition in dialogue,
including (1) the following grounding document is explicitly closely related
to the preceding grounding document, such as Seg-1 and Seg-2 in Figure 1; and
(2) the two documents are not necessarily closely related, such as Seg-3 and
Seg-4. For the former case, we exploit document-based structure knowledge of a
domain to determine the semantic proximity of document pairs, including (1)
document-level hierarchical structure indicated by the website URLs, such as
Doc-2 and Doc-3 shares one parent topic; and (2) hyperlinks between pages,
such as, hyperlink of ‘insured’ in Doc-1 to Doc-2. For the latter case, we
just randomly select document pairs from the same domain if they do not belong
to the former case.

##### Re-composition

Last, we combine multiple dialogue segments to form a new dialogue flow based
on the following rules: (1) a dialogue segment can only appear in one new
dialogue flow; (2) the grounding documents of two adjacent dialogue segments
must be different; (3) we keep the new dialogue flows between 6 and 20 turns
by filtering out shorter dialogues or discarding later turns of longer
dialogues.

domain |  #doc |  #dial |  two-seg |  >two-seg |  single
---|---|---|---|---|---
ssa |  109 |  1191 |  701 |  188 |  302
va |  138 |  1337 |  648 |  491 |  198
dmv |  149 |  1328 |  781 |  257 |  290
student |  92 |  940 |  508 |  274 |  158
total |  488 |  4796 |  2638 |  1210 |  948
Table 2: MultiDoc2Dial data statistics.

####  2.1.2 Data Collection

After we re-compose dialogue flows with multiple segments, we need to re-write
certain dialogue turns since some of the original turns could be under-
specified when taken out of the previous context, especially when they are re-
positioned at the beginning of a dialogue segment. For instance, if we use
Seg-3 in a new dialogue context, then we expect U8 to be enhanced with
necessary background such as “I am qualified for disability benefit. My wife
is currently unemployed. I want to know what benefit she gets from me” based
on Doc-1.

To collect the rewriting of a given dialogue turn, we provide context
information including up to four preceding turns, the succeeding turn, the
associated document title information and the grounding span in a document
section. We ask crowdsourced contributors to rewrite an utterance fit for the
given context, by adding necessary background information and removing the
irrelevant or contradicting content. For quality control, we also insert
various template-based placeholders for the crowd to modify accordingly. The
task would be rejected if they fail to modify the placeholders. We collect
over 6000 turns for improving the multi-segmented dialogues. The task was
performed by 30 qualified contributors from appen.com. More information about
the crowdsourcing task can be found in Appendix A.

| Train | Val | Test
---|---|---|---
#dialogues | 3474 | 661 | 661
#queries | 21453 | 4201 | 4094
avg query length | 104.6 | 104.2 | 96.5
avg response length | 22.8 | 21.6 | 22.3
#passages (token) | 4283
#passages (struct) | 4110
avg length (token) | 100
avg length (struct) | 106.6
Table 3: Data statistics of document passages and dialogue data based on
splits. The average length is based on the number of tokens.

###  2.2 Tasks

We propose two tasks for the evaluations on MultiDoc2Dial dataset.

####  2.2.1 Task I: Grounding Span Prediction

This task aims to predict the grounding document span for the next agent
response. The input includes (1) current user turn, (2) dialogue history, and
(3) the entire set of documents from all domains. The target output is a
grounding text span from one document that is relevant to the next agent
response. To train a dialogue system to be able to provide Fine-grained
grounding information can be an important step for improving the
interpretability and trustworthiness of neural-model-based conversational
systems (Carvalho et al., 2019).

####  2.2.2 Task II: Agent Response Generation

In this task, we aim to generate the next agent response, which includes
asking follow-up questions or providing an answer to a user question. Again,
the input includes (1) current user turn, (2) dialogue history and (3) the
entire set of documents from all domains. The target output is the next agent
response in natural language. This task is considered a more difficult task
than Task I since agent utterance varies in style and not directly extracted
from document content. A related task is to simulate user utterances, which
could be a more challenging task, since they are even more diversified in
style and content. We leave this task for future work.

##  3 Model

We propose to formulate the tasks on predicting agent turn as an end-to-end
generation task inspired by the recent development of retriever-reader
architecture (Karpukhin et al., 2020; Lewis et al., 2020). We consider
Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) as the base model.
It includes a retriever component and a generative reader component. The
retriever aims to retrieve the most relevant document passages given dialogue
query; the generator, a pre-trained seq2seq model in our case, takes the
combined dialogue query and top-n document passages as input and generates the
target output.

Since we need to deal with the contextual information of dialogue turns and
document passages, we further investigate how to utilize dialogue-based and
document-based structure information for the retriever component.

###  3.1 Document-based Structure

Previous approaches for open retrieval question answering typically split long
documents into smaller text passages by sliding window. In this work, we
investigate two ways of segmenting document content: (1) we also split a
document based on a sliding window size of N tokens; (2) we utilize document
structural information indicated by markup tags in HTML files. Inspired by
document tree structures used in Feng et al. (2020a); Wan et al. (2021), we
segment the document based on original paragraphs indicated by mark-up tags
such as <p> or <ul> and then attach the hierarchical titles to each paragraph
as a passage, e.g., adding ‘The Basics about Disability Benefits / Benefits
for Your Children / Qualification‘ to the last paragraph in [Doc-1] in Figure
1.

###  3.2 Dialogue-based Structure

For dialogues with multiple topic-based segments, a turn can be more distant
from previous turns lexically and semantically when the topic shifts at the
turn Arguello and Rosé (2006). Thus, we also consider incorporating the
retrieval results only by the current turn in addition to the retrieval
results by the combination of current turn and history (Qu et al., 2020b,
2021). To obtain the representation for current turn, we experiment with two
ways for BERT-based question encoder in RAG model. One is based on the common
[CLS] token embeddings; one is based on pooled token embeddings (Choi et al.,
2021).

##  4 Experiments

| F1 | EM | BL | @1 | @5 | @10 | F1 | EM | BL | @1 | @5 | @10
---|---|---|---|---|---|---|---|---|---|---|---|---
| test set | validation set
Dtoken-bm25 | 36.3 | 19.0 | 24.2 | 20.5 | 41.6 | 50.3 | 37.3 | 19.5 | 25.4 | 19.6 | 41.0 | 48.5
Dstruct-bm25 | 30.5 | 14.5 | 21.4 | 18.0 | 42.5 | 53.0 | 31.1 | 15.5 | 22.1 | 19.6 | 42.0 | 50.8
Dtoken-nq | 40.0 | 22.3 | 28.1 | 27.7 | 54.3 | 64.5 | 38.7 | 20.8 | 27.0 | 27.9 | 51.3 | 60.7
Dstruct-nq | 39.8 | 22.3 | 28.7 | 28.6 | 54.0 | 64.2 | 39.4 | 22.1 | 27.7 | 28.5 | 52.9 | 62.1
Dtoken-ft | 43.6 | 26.4 | 35.1 | 36.4 | 68.1 | 77.9 | 41.2 | 24.7 | 33.3 | 35.3 | 63.7 | 73.5
Dstruct-ft | 43.5 | 26.1 | 34.5 | 39.1 | 69.4 | 78.9 | 41.8 | 24.8 | 32.4 | 38.0 | 66.2 | 75.3
Dtoken-rr-cls-ft | 42.1 | 25.0 | 33.5 | 35.9 | 67.0 | 76.9 | 40.5 | 24.1 | 32.4 | 35.4 | 62.9 | 72.4
Dstruct-rr-cls-ft | 43.5 | 26.2 | 34.0 | 37.3 | 67.9 | 78.0 | 41.7 | 24.9 | 31.5 | 37.2 | 64.7 | 74.1
Dtoken-rr-pl-ft | 43.5 | 26.6 | 35.4 | 36.1 | 67.3 | 77.6 | 41.7 | 24.9 | 33.7 | 34.8 | 62.7 | 72.3
Dstruct-rr-pl-ft | 43.7 | 26.3 | 34.4 | 39.3 | 69.1 | 78.7 | 41.9 | 24.9 | 32.4 | 38.4 | 65.9 | 75.2
Table 4: Evaluation results of Task I on grounding span generation task. | Doc-Seg | @1 | @5 | @10
---|---|---|---|---
BM25 | token | 19.5 | 42.7 | 51.4
struct | 19.6 | 41.9 | 50.8
DPR-nq | token | 13.5 | 30.2 | 38.9
struct | 15.5 | 35.3 | 45.6
DPR-ft | token | 36.5 | 57.4 | 64.6
struct | 49.0 | 72.3 | 80.0
Table 5: Retrieval results on validation set.

We evaluate the proposed approaches on two tasks for predicting next agent
turn, i.e., grounding generation and response generation. Given current turn,
dialogue history and all available documents in the dataset, we aim to
evaluate generated text along with intermediate retrieval results. We split
the data into train/validation/test sets as shown in Table 3. The ratio
between train and validation/test set is close to $5:1$. Half of the dialogues
in validation/test set are grounded in “unseen” documents in train set. All
experiments were run with 1 V100 GPUs with half precision (FP16) training.
More details about experimental settings and hyper-parameters are reported in
B.1 and B.2.

###  4.1 Baseline Approaches

Our baseline approaches are based on RAG models Lewis et al. (2020). For the
retriever, we use DPR biencoder pre-trained on Natural Question dataset
222https://github.com/facebookresearch/DPR#new-march-2021-retrieval-model. It
contains a question encoder for encoding the dialogue query and a context
encoder for encoding document passages. We also fine-tune the DPR bi-encoder
using the train and validation set. For the generator, we use BART-large pre-
trained on CNN dataset. To train the retriever and generator end-to-end, we
use RAG-Token model, which allows the generator to select content from
multiple documents. We found RAG-Token to perform better than RAG-Sequence
intuitively and experimentally for our task because MultiDoc2Dial dataset
contains many longer agent responses with an average of 22 tokens that might
span over multiple passages. We experiment with different retrievers including
BM25 and multiple DPR variances.

###  4.2 Implementations

##### Fine-tuning DPR

To fine-tune DPR, we select positive and negative examples using the train
set. For positive examples, we use the grounding annotations, which include
the reference document passage information. For negative examples, we use
grounding as query to select top one retrieval results as hard negative
example; we use dialogue query up to 128 tokens as query, and the top 15 to 25
from retrieval results as the 10 regular negative examples based on best-
ranked results by BM25. We use gradient checkpointing to support a large batch
size, which we set as 128. This also allows for more in-batch negatives as
suggested in (Karpukhin et al., 2020; Lewis et al., 2020).

##### Document Index

To create document index, we segment documents into passages in two different
ways as described in Section 3.1. One is to split a document every one hundred
tokens, same as the default setting of RAG implementation by Huggingface. We
note the token-segmented documents as Dtoken. The other approach is to split a
document based on document sections. We note structured-segmented documents as
Dstruct. We use Maximum Inner Product Search (MIPS) to find the top-k
documents with Faiss. For indexing method, we use
IndexFlatIP“333https://github.com/facebookresearch/faiss/wiki/Faiss-indexes
method for indexing.

##### Dialogue Query Embedding

We combine current turn and history with [SEP] in between as one dialogue
query. The query is truncated if longer than maximum source length. To obtain
the representation embedding for a dialogue turn, we consider the common [CLS]
token embeddings and pooled token embeddings. For the latter, we utilize
token_type_ids for determining either current turn or history. Then we take
average pooling to turn the token embeddings to a fixed-length sequence
vector.

| F1 | EM | BL | @1 | @5 | @10 | F1 | EM | BL | @1 | @5 | @10
---|---|---|---|---|---|---|---|---|---|---|---|---
| test set | validation set
Dtoken-bm25 | 28.8 | 2.4 | 12.8 | 20.5 | 41.6 | 50.3 | 28.4 | 2.1 | 13.3 | 19.6 | 41.0 | 48.5
Dstruct-bm25 | 27.9 | 2.0 | 12.5 | 18.0 | 42.5 | 53.0 | 27.6 | 2.0 | 12.8 | 19.6 | 42.0 | 50.8
Dtoken-nq | 32.5 | 3.2 | 16.9 | 25.9 | 51.0 | 61.6 | 30.9 | 2.8 | 15.7 | 25.8 | 48.2 | 57.7
Dstruct-nq | 33.0 | 3.6 | 17.6 | 27.3 | 52.6 | 62.7 | 31.5 | 3.2 | 16.6 | 27.4 | 51.1 | 60.2
Dtoken-ft | 35.0 | 3.7 | 20.4 | 36.8 | 68.3 | 77.8 | 33.2 | 3.4 | 18.8 | 35.2 | 63.4 | 72.9
Dstruct-ft | 36.0 | 4.1 | 21.9 | 39.7 | 69.3 | 79.0 | 33.7 | 3.5 | 19.5 | 37.5 | 67.0 | 75.8
Dtoken-rr-cls-ft | 34.9 | 3.5 | 20.0 | 35.7 | 66.8 | 76.6 | 33.2 | 3.4 | 18.4 | 34.7 | 62.2 | 71.8
Dstruct-rr-cls-ft | 36.0 | 4.1 | 21.7 | 38.1 | 67.8 | 77.2 | 34.1 | 3.9 | 19.8 | 36.8 | 65.1 | 74.4
Dtoken-rr-pl-ft | 35.1 | 3.8 | 20.7 | 33.4 | 64.6 | 75.1 | 32.9 | 3.5 | 18.6 | 32.3 | 60.6 | 71.1
Dstruct-rr-pl-ft | 35.9 | 4.0 | 21.7 | 39.3 | 69.0 | 78.8 | 33.7 | 3.7 | 19.4 | 36.8 | 66.4 | 75.2
Table 6: Evaluation results of Task II on agent response generation.

###  4.3 Retriever Settings

We experiment with the following variations for the retriever components.

  * •

Dtoken / Dstruct-nq: uses original pre-trained bi-encoder from DPR. The
corresponding document index is based on token/structure-segmented passages.
We consider these setups as baselines.

  * •

Dtoken-ft / Dstruct-ft: uses fine-tuned DPR bi-encoder. The document index is
based on token/structure-segmented passages.

  * •

Dtoken-rr-cls-ft / Dstruct-rr-cls-ft: uses fine-tuned DPR bi-encoders. We
combine the retrieval results of entire dialogue query and only the current
turn and select the top-k unique passages. The representation of current turn
is based on [CLS] token embeddings. The document index is based on
token/structure-segmented passages.

  * •

Dtoken-rr-pl-ft / Dstruct-rr-pl-ft: uses fine-tuned DPR bi-encoders. We
combine the retrieval results of entire dialogue query and only the current
turn and select the top-k unique passages. The representation of current turn
is based on pooled token embeddings. The document index is based on
token/structure-segmented passages.

In addition, we also experiment with BM25 (Trotman et al., 2014), noted as
Dtoken-bm25/ Dstruct-bm25), where BM25 is used for retrieving top-k passages
following the experiment set up in Lewis et al. (2020).

###  4.4 Quantitative Analysis

####  4.4.1 Evaluation Metrics

We evaluate the passage retrieval results and the generated text for both
tasks. For retrieval, we compute recall ($@k$), which measures the fraction of
times the correct document is found in the top-$k$ predictions. We evaluate
text generation output based on token-level F1 score (F1), Exact Match (EM)
(Rajpurkar et al., 2016) and SacreBLEU score (BL) (Post, 2018).

####  4.4.2 Passage Retrieval Results

We first evaluate the performance of BM25, DPR and fine-tuned DPR on a passage
retrieval task. The query is the combination of current turn and dialogue
history from latest to earliest turn up to 128 tokens. Table 5 presents the
retrieval results on the validation set. BM25 performs better than DPR-nq but
worse than DPR-ft. However, it almost shows no difference for Dtoken and
Dstruct. DPR-ft shows significant improvement over DPR-nq. Both DPR-nq and
DPR-ft seem to benefit from the document-based structure as they show better
performance on Dstruct than Dtoken.

####  4.4.3 Generation Results

Table 4 and 6 present the evaluation results on test and validation set for
the two tasks respectively. All numbers on in tables are the mean of three
runs with different random seeds. We omit the standard deviation numbers as
they suggest low variance in our experiments. Even though BM25 outperforms
DPR-nq in Table 5, BM25 performs much worse than DPR-nq for the generation
tasks as shown in Table 4 and 6. RAG models with different DPR-based
retrievers generally perform better with Dstruct than Dtoken on the generation
tasks. This is consistent with the DPR-based retrieval results in Table 5. RAG
models with DPR-ft show improvement for both Dstruct and Dtoken over the ones
with DPR-nq, which confirms the importance of positive and negative examples
even in small quantity (Karpukhin et al., 2020; Khattab et al., 2020b). We
also see the retrieval performance gap between Dtoken and Dstruct is reduced
after training the fine-tuned question encoder in RAG. Overall, the retrieval
performances for the two tasks seem comparable but generation metric scores
for Task II are much lower Task I as the agent responses are free-formed
natural language.

We also experiment with a simple way to re-rank retrieved passages of the
entire query based on the retrieved results only based on current turn. We
experiment with two types of embeddings for current turn as described earlier.
As shown in Table 4 and 6, the re-ranking is not every effective. The
difference between reranking with two different kinds of encodings is
insignificant. In addition, we also evaluate the baselines on unseen domains,
where we train the models using the data from three source domains and test on
one target unseen domain. For more experiments results on the domain
adaptation setup, please see Appendix B.3.

####  4.4.4 Qualitative Analysis

To understand the challenges in dialogue grounded in multiple documents and
evaluate the data quality, we randomly select some dialog queries from the
validation set and examine the queries along with their corresponding
retrieved passages by our model. We observe certain ambiguities in the
dialogue-based and document-based contexts, which we summarize as follows.

  * •

Ambiguity in dialogue queries: when a user question is under-specific or
inquisitive about a higher level topic, it is likely to be quite relevant to
multiple document passages. For instance, query U8 in [Seg-3] could be linked
to different types of benefits described in multiple documents in ssa.gov.

  * •

Ambiguity in document content: when certain passages with very similar topics
in different documents, they could be duplicate or different in context. For
instance, same question could be addressed in an FAQ page and in another
article regarding a different specific criterion.

##### Human evaluations on ambiguity in questions

We ask human experts to identify whether a dialogue inquiry turn is ambiguous
based on its dialogue history and the associated documents. We consider a
dialogue query as ‘ambiguous’ if it is likely to be relevant to a broad range
of domain knowledge; otherwise as ‘unambiguous’. Firstly, we ask the the
annotators to annotate the queries based on their understanding of dialogue
context and the domain knowledge. Secondly, we reveal the reference document
passage and another most relevant document passage retrieved by our models and
ask them to annotate the queries again. We randomly select 100 dialogue
inquiry turns and assign them to two experts with 20% overlap. The Cohen’s
kappa agreement score is 0.85. In the first setting, 15% of the turns are
labeled as ‘ambiguous’. In the second setting, after revealing the relevant
passages, 20% turns are considered ‘ambiguous’.

Such ambiguities are generally inherent to open retrieval settings (Zhu et
al., 2021; Min et al., 2020). In practice, it would require the conversational
agents to ask follow-up questions for clarification based on dialogue history
and retrieved passages for providing a more fair and informative answer. The
current version of MultiDoc2Dial dataset can be further enhanced by adding
agent turns for asking clarification questions based on different levels such
as highest title (topic) level, sub-title (sub-topic) level or even finer
level, which we leave for near future work.

##  5 Related Work

###  5.1 Document-grounded Dialogue and Conversational Question Answering

Our work is closely related to the recent work on document-grounded dialogue
and conversational machine reading comprehension tasks, such as Doc2Dial (Feng
et al., 2020b), ShARC (Saeidi et al., 2018) and DoQA (Campos et al., 2020).
The primary goal of these papers is to provide an answer, or a dialogue
response based on a single given document or text snippet. In contrast to the
closed-book setting of these works, our task is in an open-book setting, which
aims to address more realistic scenarios in goal-oriented dialogues where the
associated document content are unknown and likely more than one documents.

Our work is built on Doc2Dial, which is a goal-oriented dialogue modeling task
based on a single document. Our dataset shares the same set documents and
annotation scheme as Doc2Dial. However, we aim to further the challenge by
dealing with cases when the document-level topic shifts through a dialogue.
Thus, we proposed a new data construction approach, data tasks, and baseline
approaches in an end-to-end setting.

###  5.2 Open Domain Question Answering

Our proposed data and tasks are also related to open domain question answering
(Chen et al., 2017; Lee et al., 2019; Kwiatkowski et al., 2019; Min et al.,
2020; Qu et al., 2020b; Mao et al., 2020; Zhu et al., 2021; Izacard and Grave,
2021; Li et al., 2021; Xiong et al., 2021; Yu et al., 2021). In particular,
our work is closely related to the recently proposed open-retrieval
conversational question answering (OR-CQA) setting for QuAC dataset, i.e., OR-
QuAC (Qu et al., 2020b). To the best of our knowledge, the search queries for
tasks are mostly created based on Wikipedia articles. In OR-QuAC (Qu et al.,
2020b), all turns in one conversation are grounded in passages from the
Wikipedia page of the given entity, i.e., there are no multiple documents
involved in a dialogue. In contrast, our task proposes to model dialogues
grounded in multiple documents. The documents are of diverse writing styles
from four real user-facing websites. In addition to the difference in document
data, MultiDoc2Dial provides more types of dialogue query based on a richer
set of dialogue acts. Table 1 provides a comparison of several most related
datasets and tasks in different aspects including whether the setup is open-
book or not, the dialogues are goal-oriented or not, the grounding is
annotated or not, the associated text is full document or not and each
dialogue corresponds to multiple documents or not. Our work is the only one
that covers all the characteristics.

###  5.3 Discourse Segmentation

This work is also largely related to discourse segmentation tasks (Arguello
and Rosé, 2006; Zhang and Zhou, 2019; Mele et al., 2020; Gao et al., 2020;
Xing and Carenini, 2021), which aims to identify the change of topic in a
dialogue. This is a very important task towards modeling of goal-oriented
dialogues in general. Some papers, such as Arguello and Rosé (2006); Hsueh et
al. (2006); Xing and Carenini (2021), focus on the task of modeling and
predicting segmentation; some papers such as Gao et al. (2020) use explicit
segmentation as the input of downstream dialogue modeling tasks on a machine
reading comprehension dataset, ShaRC. Our task is closely related to the
latter, albeit in an open book setting and with an end-to-end modeling
approach that encodes the dialogue segmentation information implicitly. We
leave the more explicit dialogue segmentation modeling for future work.

##  6 Conclusion and Future Work

We introduced MultiDoc2Dial, a new task and dataset that deals with goal-
oriented dialogues that have multiple sub-goals corresponding to different
documents. We proposed two tasks for predicting next agent turn that we
formulate as generation tasks. We presented strong baseline approaches based
on retriever-reader architecture and experimented with different variances of
neural retrievers. For future work, we aim to address the ambiguity in the
open-book dialogue modeling.

##  7 Ethical Consideration

One primary motivation of the paper is to provide data instances that simulate
how real human users converse with agents to seek information. Such data is
essential for training neural models to build conversational systems that
could assist various end users to access information in real-life domains such
as social benefit websites. However, such a dataset is largely unavailable for
research and development. Since we create the dataset via crowdsourcing, one
potential ethical concern is that it could be potentially biased or distant
from the real user queries. To address such concerns, we try to identify
qualified contributors with different backgrounds and train them via several
rounds of tasks. In addition, we provide various examples in the instruction
and feedback to contributors during the crowdsourcing task.

## References

  * Arguello and Rosé (2006) Jaime Arguello and Carolyn Rosé. 2006.  Topic-segmentation of dialogue.  In _Proceedings of the Analyzing Conversations in Text and Speech_ , pages 42–49, New York City, New York. Association for Computational Linguistics.
  * Campos et al. (2020) Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, and Eneko Agirre. 2020.  DoQA - accessing domain-specific FAQs via conversational QA.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7302–7314, Online. Association for Computational Linguistics.
  * Carvalho et al. (2019) Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. 2019.  Machine learning interpretability: A survey on methods and metrics.  _Electronics_ , 8(8):832.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Choi et al. (2021) Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. 2021.  Evaluation of bert and albert sentence embedding performance on downstream nlp tasks.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 5482–5487. IEEE.
  * Feng et al. (2020a) Song Feng, Kshitij Fadnis, Q. Vera Liao, and Luis A. Lastras. 2020a.  Doc2dial: A framework for dialogue composition grounded in documents.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(09):13604–13605.
  * Feng et al. (2020b) Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020b.  doc2dial: A goal-oriented document-grounded dialogue dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8118–8128, Online. Association for Computational Linguistics.
  * Gao et al. (2020) Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven C.H. Hoi, Caiming Xiong, Irwin King, and Michael Lyu. 2020.  Discern: Discourse-aware entailment reasoning network for conversational machine reading.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2439–2449, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Hsueh et al. (2006) Pei-Yun Hsueh, Johanna D. Moore, and Steve Renals. 2006.  Automatic segmentation of multiparty dialogue.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , Trento, Italy. Association for Computational Linguistics.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Distilling knowledge from reader to retriever for question answering.  In _International Conference on Learning Representations_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab et al. (2020a) Omar Khattab, Christopher Potts, and Matei Zaharia. 2020a.  Relevance-guided supervision for openqa with colbert.  _arXiv preprint arXiv:2007.00814_.
  * Khattab et al. (2020b) Omar Khattab, Christopher Potts, and Matei Zaharia. 2020b.  Relevance-guided supervision for openqa with colbert.  _CoRR_ , abs/2007.00814.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _arXiv preprint arXiv:2005.11401_.
  * Li et al. (2021) Yongqi Li, Wenjie Li, and Liqiang Nie. 2021.  A graph-guided multi-round retrieval method for conversational open-domain question answering.  _arXiv preprint arXiv:2104.08443_.
  * Mao et al. (2020) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020.  Generation-augmented retrieval for open-domain question answering.
  * Mele et al. (2020) Ida Mele, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, and Ophir Frieder. 2020.  Topic propagation in conversational search.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2057–2060.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Qu et al. (2021) Chen Qu, Liu Yang, Cen Chen, W. Bruce Croft, Kalpesh Krishna, and Mohit Iyyer. 2021\.  Weakly-supervised open-retrieval conversational question answering.
  * Qu et al. (2020a) Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020a.  _Open-Retrieval Conversational Question Answering_, page 539–548. Association for Computing Machinery, New York, NY, USA.
  * Qu et al. (2020b) Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit Iyyer. 2020b.  Open-retrieval conversational question answering.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 539–548.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Saeidi et al. (2018) Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018.  Interpretation of natural language rules in conversational machine reading.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2087–2097, Brussels, Belgium. Association for Computational Linguistics.
  * Trotman et al. (2014) Andrew Trotman, Antti Puurula, and Blake Burgess. 2014.  Improvements to bm25 and language models examined.  In _Proceedings of the 2014 Australasian Document Computing Symposium_ , pages 58–65.
  * Wan et al. (2021) Hui Wan, Song Feng, Chulaka Gunasekara, Siva Sankalp Patel, Sachindra Joshi, and Luis Lastras. 2021.  Does structure matter? encoding documents for machine reading comprehension.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4626–4634, Online. Association for Computational Linguistics.
  * Xing and Carenini (2021) Linzi Xing and Giuseppe Carenini. 2021.  Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 167–177, Singapore and Online. Association for Computational Linguistics.
  * Xiong et al. (2021) Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021.  Answering complex open-domain questions with multi-hop dense retrieval.  In _International Conference on Learning Representations_.
  * Yu et al. (2021) Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, and Zhiyuan Liu. 2021.  Few-shot conversational dense retrieval.  In _International ACM SIGIR Conference_.
  * Zhang and Zhou (2019) Leilan Zhang and Qiang Zhou. 2019.  Topic segmentation for dialogue stream.  In _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , pages 1036–1043.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_.

##  Appendix A Data Construction

For crowdsourcing, we filter out less qualified contributors by adding
template-based placeholders in the writing task for detecting bad
performances, which seems effective. Most contributors seem to be able to
improve either user query or agent response, sometimes both together based on
document context. However, we do find the writings from original Doc2Dial
dataset appear a bit more natural with more personalized information. We
suspect that it could be easier to add such information in the writing if the
conversation is built from scratch based on a single document while the
contributors write the dialogue history themselves. We provide feedback
accordingly to the crowd to address the issue. We observe that the crowd
contributors hardly reject any task. For quality control, we also manually
review and re-collect data during the data collection process. For the
instruction, interface, rules and examples for data collection via
crowdsourcing, please see Figure 2(a) and 2(b) for reference.

##  Appendix B Experiments

The implementation is in PyTorch. For fine-tuning RAG (Lewis et al., 2020), we
follow the
example444https://github.com/huggingface/transformers/blob/master/examples/research˙projects/rag/finetune˙rag.py
from HuggingFace and fine-tune it on our dataset for 16 epochs. For fine-
tuning DPR (Karpukhin et al., 2020), we train DPR-nq on our dataset using
facebookresearch/DPR555https://github.com/facebookresearch/DPR. Then, we
integrate fine-tuned bi-encoder in RAG model facebook/rag-token-
nq666https://huggingface.co/facebook/rag-token-nq. For pre-trained DPR, we use
the bi-encoder model trained on NQ dataset only from Facebook DPR
checkpoint777https://github.com/facebookresearch/DPR#new-march-2021-retrieval-
model. We train the models for 10 epochs and evaluate using the last
checkpoint.

###  B.1 Hyperparameters for fine-tuning DPR

We fine-tune DPR for 50 epochs with a batch size of 128. We use a learning
rate of 2e-05 using Adam, linear scheduling with warmup and dropout rate of
0.1. We set the max encoder sequence length to 128 consistent with the RAG
model. We also use one additional BM25 negative passage per question in
addition to in-batch negatives. We use gradient checkpointing to support a
large batch size as 128. The performance of checkpoint at epoch 50 and 80 are
very close. We use checkpoint at 50 epoch across experiments.

###  B.2 Hyperparameters for fine-tuning RAG

Hyper-parameters used for fine-tuning as as follows.

    train_batch_size=8
    eval_batch_size=2
    max_combined_length=300
    max_source_length=128
    max_target_length=50
    val_max_target_length=50
    test_max_target_length=50
    label_smoothing=0.1
    dropout=0.1
    attention_dropout =0.1
    weight_decay=0.001
    adam_epsilon=1e-08
    max_grad_norm=0.1
    lr_scheduler=polynomial
    learning_rate=3e-05
    warmup_steps=500
    gradient_accumulation_steps=1

|  Retriever |  Reader
---|---|---
DPR |  BERT-base |  -
2X100M |  -
RAG |  BERT-base |  BART-large
2X100M |  330M
Table 7: The number of parameters of models.

###  B.3 Experiment Results

##### Fine-tuned DPR

For fine-tuning DPR, we experiment with different ways of obtaining hard
negative examples. One is using the grounding of a dialogue query (grounding)
as query, the other is using combined dialogue utterances (query) as query.
The results turn out comparable as shown in Table 9.

##### Domain Adaptation Setup

We also experiment with domain adaption setup, where the train and validate
splits are based on all data from the three domains and the test split is
based on one unseen domain. The domain information is considered when
retrieving relevant passage. The ratio of the number of examples in train and
validation is $5:1$. Table 8 presents the retrieval results of different
source and target unseen domains, which is comparable to Table 5. However, the
EM and F1 scores in Table 10 are much lower comparing to the setup without any
unseen domain in Table 4 and 6, which confirms that the domain adaptation
setup is indeed challenging.

Unseen | Doc-Seg | R@1 | R@5 | R@10
---|---|---|---|---
ssa | token | 50.0 | 69.6 | 73.8
struct | 65.7 | 86.3 | 90.7
va | token | 48.0 | 68.0 | 73.8
struct | 64.0 | 85.2 | 90.3
dmv | token | 51.3 | 69.7 | 75.1
struct | 67.6 | 86.2 | 91.2
studentaid | token | 49.7 | 69.4 | 75.0
struct | 66.0 | 85.8 | 91.2
Table 8: Retrieval results on validation set for the domain adaption setup. Neg. | Doc-Seg | @1 | @5 | @10 | @50
---|---|---|---|---|---
| 40 epochs
query | token | 36.5 | 58.2 | 65.0 | 77.3
struct | 48.6 | 71.1 | 78.9 | 91.7
grounding | token | 37.0 | 58.8 | 65.7 | 76.5
struct | 47.7 | 70.2 | 77.7 | 91.4
| 50 epochs
query | token | 36.5 | 57.4 | 64.6 | 77.1
struct | 49.0 | 72.3 | 80.0 | 92.5
grounding | token | 35.9 | 57.6 | 65.6 | 76.8
struct | 47.8 | 71.0 | 78.8 | 92.0
Table 9: Retrieval results on validation set with respect to different DPR
settings. Task I: grounding generation
---
|  | F1 | EM | BLEU | R@1 | R@5 | R@10
ssa | Dtoken-ft | 29.0 | 8.9 | 20.8 | 15.3 | 39.8 | 52.5
Dstruct-ft | 27.2 | 7.7 | 17.6 | 15.9 | 45.9 | 61.3
Dtoken-rr-pl-ft | 29.0 | 9.0 | 20.7 | 15.3 | 39.9 | 52.5
Dstruct-rr-pl-ft | 27.3 | 7.7 | 18.0 | 15.9 | 46.0 | 61.4
va | Dtoken-ft | 31.4 | 10.9 | 23.0 | 32.1 | 57.2 | 66.7
Dstruct-ft | 31.2 | 11.8 | 23.5 | 34.1 | 61.1 | 70.5
Dtoken-rr-pl-ft | 31.2 | 11.0 | 23.1 | 32.1 | 57.2 | 66.7
Dstruct-rr-pl-ft | 31.0 | 11.7 | 23.5 | 34.1 | 61.1 | 70.5
dmv | Dtoken-ft | 29.4 | 9.6 | 19.6 | 31.7 | 58.3 | 68.8
Dstruct-ft | 28.6 | 9.8 | 20.7 | 34.9 | 61.5 | 72.2
Dtoken-rr-pl-ft | 29.3 | 9.4 | 19.5 | 31.7 | 58.3 | 68.8
Dstruct-rr-pl-ft | 28.7 | 9.9 | 20.8 | 34.9 | 61.5 | 72.2
studentaid | Dtoken-ft | 29.6 | 9.6 | 21.2 | 34.6 | 62.1 | 71.8
Dstruct-ft | 28.9 | 9.6 | 20.8 | 38.3 | 66.6 | 76.2
Dtoken-rr-pl-ft | 29.3 | 9.4 | 21.2 | 34.6 | 62.1 | 71.8
Dstruct-rr-pl-ft | 29.0 | 9.6 | 20.8 | 38.3 | 66.6 | 76.2
Table 10: Evaluation results of Task I on test set of unseen domains. Task II:
response generation
---
|  | F1 | EM | BLEU | R@1 | R@5 | R@10
ssa | Dtoken-ft | 28.1 | 1.1 | 13.3 | 15.3 | 39.8 | 52.5
Dstruct-ft | 28.0 | 1.1 | 13.2 | 15.9 | 45.9 | 61.3
Dtoken-rr-pl-ft | 28.2 | 1.1 | 13.4 | 15.3 | 39.9 | 52.5
Dstruct-rr-pl-ft | 28.0 | 1.1 | 13.3 | 15.9 | 46.0 | 61.4
va | Dtoken-ft | 30.4 | 2.4 | 17.2 | 32.1 | 57.2 | 66.7
Dstruct-ft | 31.2 | 2.4 | 18.6 | 34.1 | 61.1 | 70.5
Dtoken-rr-pl-ft | 30.7 | 2.5 | 17.4 | 32.1 | 57.2 | 66.7
Dstruct-rr-pl-ft | 31.2 | 2.3 | 18.6 | 34.1 | 61.1 | 70.5
dmv | Dtoken-ft | 28.9 | 1.4 | 13.8 | 31.7 | 58.3 | 68.8
Dstruct-ft | 29.5 | 1.7 | 14.9 | 34.9 | 61.5 | 72.2
Dtoken-rr-pl-ft | 28.8 | 1.3 | 13.8 | 31.7 | 58.3 | 68.8
Dstruct-rr-pl-ft | 29.6 | 1.6 | 15.0 | 34.9 | 61.5 | 72.2
studentaid | Dtoken-ft | 29.0 | 0.9 | 13.6 | 34.6 | 62.1 | 71.8
Dstruct-ft | 28.8 | 1.0 | 14.0 | 38.3 | 66.6 | 76.2
Dtoken-rr-pl-ft | 29.1 | 1.0 | 13.8 | 34.6 | 62.1 | 71.8
Dstruct-rr-pl-ft | 28.8 | 0.9 | 13.9 | 38.3 | 66.6 | 76.2
Table 11: Evaluation results of Task II on test set of unseen domains.

(a) The screenshot of task description and data collection interface of the
crowdsourcing task.

(b) The screenshot of part of the rules and examples of the crowdsourcing
task.


# Logical Parsing from Natural Language Based on a Neural Translation Model

Liang Li1, Pengyu Li1, Yifan Liu1, Tao Wan2 and Zengchang Qin1  2Biomedical
Imaging and Informatics Lab, School of Biological Science and Medical
Engineering
1Intelligent Computing and Machine Learning Lab, School of ASEE
Beihang University, Beijing 100191, China
{liangli919, lipengyu, yifan_liu, taowan<EMAIL_ADDRESS>
###### Abstract

Semantic parsing has emerged as a significant and powerful paradigm for
natural language interface and question answering systems. Traditional methods
of building a semantic parser rely on high-quality lexicons, hand-crafted
grammars and linguistic features which are limited by applied domain or
representation. In this paper, we propose a general approach to learn from
denotations based on Seq2Seq model augmented with attention mechanism. We
encode input sequence into vectors and use dynamic programming to infer
candidate logical forms. We utilize the fact that similar utterances should
have similar logical forms to help reduce the searching space. Under our
learning policy, the Seq2Seq model can learn mappings gradually with noises.
Curriculum learning is adopted to make the learning smoother. We test our
method on the arithmetic domain which shows our model can successfully infer
the correct logical forms and learn the word meanings, compositionality and
operation orders simultaneously.

###### Index Terms:

Seq2Seq; Semantic parsing; Weak supervision

##  I Introduction

The problem of learning a semantic parser has been receiving significant
attention. Semantic parsers map natural language into a logical form that can
be executed on a knowledge base and return an answer (denotation). The early
works use logical forms as supervision [1, 2, 3, 4], given a set of input
sentences and their corresponding logical forms, learning a statistical
semantic parser by weighting a set of rules mapping lexical items and
syntactic patterns to their logical forms. Given an input, these rules are
applied recursively to derive the most probable logical form. However, the
tremendous labor needed for annotating logical forms has turned the trend to
weak supervision $-$ using denotation of logical forms as the training target.
It has been successfully applied in different fields including question-
answering [5, 6, 7, 8] and robot navigation [9]. All these works need hand-
crafted grammars that are crucial in semantic parsing but pose an obstacle for
generalization. Wang _et al._ [10] build semantic parsers in 7 different
domains and hand engineer a separate grammar for each domain.

The rise of Seq2Seq model [11] provides an alternative method to tackle the
mapping problem and no more manual grammars are needed. The ability to deal
with sequences with changeable length as input and/or output has translated
this model into applications including machine translation [11, 12], syntactic
parsing [13], and question answering [14]. All of these work do not need hand-
crafted grammars and are so-called end-to-end learning. But they do not
resolve the problem of supervision by denotation, which makes one step further
and needs logic reasoning and operation. Our model adopts the encoder-decoder
framework and tries to use denotation as the target of supervised learning. We
take the advantage of the Seq2Seq model’s ability of tackling input/output
with different length and grammar-free form to learn the mappings from natural
language to logical forms. Our main focus is to infer logical forms from
denotations in a generalizable way. We wish to add minimal extra constraints
or manual features, so that it can be applied to other domains. For now, we
can not infer correct logical forms all the time but we prove that the Seq2Seq
model is capable of learning with noises in training data and the curriculum
learning form mitigates this effect.

A problem of weak supervision is the search of the consistent logical forms
when only denotation is available. The number of logical forms grows
exponentially as their size increases and inferring from denotations
inevitably induces spurious logical forms $-$ those that do not represent the
original sentence semantics but get the correct answer accidently [15]. To
control the searching space, previous works relied on restricted sets of rules
which limits expressivity and are possible to rule out the correct logical
form. Instead, we utilize dynamic programming to construct the candidate
logical form set with an extra base case set to filter all the candidates, and
use the ongoing training Seq2Seq model to determine the best one. In this way,
we can maintain the expressivity and train our model iteratively by feeding
the suggested best logical form back to our model. Consequently, a right pick
results in a positive learning, which allows our model to put a higher
probability on the correct logical form over others and leads to the desired
mapping over training.

We evaluate our model on arithmetic domain through a toy example. And the
model is capable of translating row sentences into mathematical equations in
structured tree form and returning the answer directly. We provide a base case
set which serves as a pivot for the model to learn.Our model learns the
arithmetic calculation in a curriculum way, where simpler sentences with fewer
words are inputted at initial state. Our model assumes no prior linguistic
knowledge and learns the meaning of all the words, compositionality, and order
of operations from just the natural language $-$ denotation pairs.

###  I-A Related work

We adopt the general encoder-decoder framework based on neural networks
augmented with attention mechanism [16], which allows the model to learn soft
alignment between utterances and logical forms. Our work is related to [17]
and [13], both of which use the Seq2Seq model to map natural language to
logical forms in tree structure without hand-engineered features. But our work
makes one step further by using denotation of logical form as the learning
target and regard logical forms as latent variables.

How to reduce the searching space is a chief challenge in weak supervision. A
common approach is to constrain the set of possible logical form compositions,
which can significantly reduce the searching space but also constrain the
expressivity [18]. Lao _et al._ [19] use random walks to generate logical
forms and use denotation to cut down the searching space during learning.
Liang _et al._ [20] adopt a similar method to narrow down the options and
allow more complex semantics to be composed. Different from generating logical
forms forwardly as mentioned above, an alternative method is to use denotation
to infer logical forms using dynamic programming [21, 15]. In this way, it is
more likely to recover the full set and find the desired one. Inspired by
their work, we also employ this method and store the denotation $-$ logical
form pairs in advance to accelerate the lookup efficiency.

Our work is similar to [22] in the sense that we both focus on the arithmetic
domain and learn from denotations directly. But their work relies on hand-
crafted grammars to construct logical forms and hand engineered features to
filter out incorrect logical forms. Instead, the Seq2Seq model we use is
grammar-free and the features we select to screen logical forms is more
general. Our main idea is that similar utterance should have similar logical
forms, so that we can filter out incorrect logical forms by using similarity
measurement. We build up a small base case set to assist this idea and the
similarity function we adopt is simply bag-of-words, which is replaceable when
extending to other fields. Furthermore, to sort the candidates, we do not have
a particular scoring function to weight extracted features, instead, we use
the ongoing training Seq2Seq model to evaluate their loss.

There are some other related work, Neural Programmer [23] augmented with a
small set of arithmetic and logic operations is able to perform complex
reasoning and has shown success in question answering [24]. Neural Turing
Machines [25] can infer simple algorithms such as copying and sorting with
external memory.

TABLE I: The base case set with several pairs of $<u,s>$, helping to reduce searching space Utterance | Logical form | Denotation
---|---|---
one plus two $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘1’ ‘+’ ‘2’ ’]’ ’End’ $\rangle$ | 3.0
three minus four times five $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘3’ ‘-’ ‘(’ ‘4’ ‘*’ ‘5’ ‘)’ ‘]’ ‘End’ $\rangle$ | -17.0
one times two divide three $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘(’ ‘(’ ‘1’ ‘*’ ‘2’ ‘)’ ‘/’ ‘3’ ‘)’ ‘End’ $\rangle$ | 0.667
two divide four plus five $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘(’ ‘2’ ‘/’ ‘4’ ‘)’ ‘+’ ‘5’ ‘]’ ‘End’ $\rangle$ | 5.5
five divide one times two plus three $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘(’ ‘(’ ‘5’ ‘/’ ‘1’ ‘)’ ‘*’ ‘2’ ‘)’ ‘+’ ‘3’ ‘]’ ‘End’ $\rangle$ | 13.0
four minus two times three plus one $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘[’ ‘4’ ‘-’ ‘(’ ‘2’ ’*’ ‘3’ ‘)’ ‘]’ ‘+’ ‘1’ ‘]’ ‘End’ $\rangle$ | -1.0
three divide four minus five plus two $\langle$ eos $\rangle$ |  $\langle$ ‘Go’ ‘[’ ‘[’ ‘(’ ‘3’ ‘/’ ‘4’ ‘)’ ‘-’ ‘5’ ‘]’ ‘+’ ‘2’ ‘]’ ’End’ $\rangle$ | -2.25

##  II Background:sequence-to-sequence model and attention mechanism

Before introducing our model, we describe briefly the Seq2Seq model and
attention mechanism.

###  II-A Sequence-to-sequence model

The Seq2Seq model takes a source sequence $X=(x_{1},x_{2},...,x_{T})$ as input
and outputs a translated sequence $Y=(y_{1},y_{2},...,y_{T^{\prime}})$. The
model maximizes the generation probability of $Y$ conditioned on $X$:
$p(y_{1},...,y_{T^{\prime}}|x_{1},x_{2},...,x_{T})$. Specifically, the Seq2Seq
is in an encoder-decoder structure. In this framework, an encoder reads the
input sequence word by word into a vector $c$ through recurrent neural network
(RNN).

$h_{t}=f(x_{t},h_{t-1})$ (1)

and

$c=q({h_{1},...,h_{T}}),$

where $h_{t}$ is the hidden state at time $t$, $c$ is commonly taken directly
from the last hidden state of encoder $q({h_{1},...,h_{T}})=h_{T}$, and $f$ is
a non-linear transformation which can be either a long-short term memory unit
(LSTM) [26] or a gated recurrent unit (GRU) [27]. In this paper, LSTM is
adopted and is parameterized as

$\left({\begin{array}[]{*{20}{c}}{{i_{t}}}\\\ {{f_{t}}}\\\ {{o_{t}}}\\\
{{{\tilde{c}}_{t}}}\end{array}}\right)=\left({\begin{array}[]{*{20}{c}}\sigma\\\
\sigma\\\ \sigma\\\
{\tanh}\end{array}}\right)T\left({\begin{array}[]{*{20}{c}}{{x_{t}}}\\\
{{h_{t-1}}}\end{array}}\right)$ (2a)

${c_{t}}={f_{t}}\circ{c_{t-1}}+{i_{t}}\circ{{\tilde{c}}_{t}}$ (2b)
${h_{t}}={o_{t}}\circ\tanh({c_{t}})$ (2c)

where $\circ$ is an element-wise multiplication, $T$ is an affine
transformation, $\sigma$ is the logistic sigmoid that restricts its input to
[0,1], $i_{t}$, $f_{t}$ and $o_{t}$ are the input, forget, and output gates of
the LSTM, and $c_{t}$ is the memory cell activation vector. The forget and
input gates enable the LSTM to regulate the extent to which it forgets its
previous memory and the input, while the output gate controls the degree to
which the memory affects the hidden state. The encoder employs
bidirectionality, encoding the sentences in both the forward and backward
directions, an approach adopted in machine translation [16, 12]. In this way,
the hidden annotations
${h_{t}}=(\overrightarrow{h}_{t}^{T};\overleftarrow{h}_{t}^{T})$ concatenate
forward $\overrightarrow{h}_{t}^{T}$ and backward annotations
$\overleftarrow{h}_{t}^{T}$ together, each determined using Equation 2c.

The decoder is trained to estimate generation probability of next word $y_{t}$
given all the previous predicted words and the context vector $c$. The
objective function of Seq2Seq can be written as

$p(y_{1},...,y_{T^{\prime}}|x_{1},...,x_{T})=\prod_{t=1}^{T^{\prime}}p(y_{t}|c,y_{1},...,y_{t-1}).$
(3)

With an RNN, each conditional probability is modeled as

$p(y_{t}|{y_{1},...,y_{t-1}},c)=g(y_{t-1},s_{t},c),$ (4)

where $g$ is a non-linear function that outputs the probability of $y_{t}$,
$y_{t-1}$ is the predicted word at time $t-1$ in the response sequence, and
$s_{t}$ is the hidden state of the decoder RNN at time $t$, which can be
computed as

$s_{t}=f(y_{t-1},s_{t-1},c).$ (5)

###  II-B Attention mechanism

The traditional Seq2Seq model predicts each word from the same context vector
$c$, which deprives the source sequence information and makes the alignment
imprecisely. To address this problem, attention mechanism is introduced to
allow decoder focusing on the source sequence instead of a compressed vector
upon prediction [16]. In Seq2Seq with attention mechanism, each $y_{i}$ in $Y$
corresponds to a context vector $c_{i}$ instead of $c$. The conditional
probability in Equation 4 becomes

$p(y_{i}|y_{1},...,y_{i-1},x)=g(y_{i-1},s_{i},c_{i}),$ (6)

where the hidden state $s_{i}$ is computed by

$s_{i}=f(y_{i-1},s_{i-1},c_{i}).$ (7)

The context vector $c_{i}$ is a weighted average of all hidden states
$\\{{h_{t}}\\}_{t=1}^{T}$ of the encoder, defined as

$c_{i}=\sum_{j=1}^{T}\alpha_{ij}h_{j},$ (8)

where the weight $\alpha_{ij}$ is given by

$\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k=1}^{T}\exp(e_{ik})},$ (9)

where $e_{ij}$ is an alignment model which scores how well the inputs around
position $j$ and the output at position $i$ match,

$e_{ij}=a(s_{i-1},h_{j}),$ (10)

where $a$ is a feed forward neural network, trained jointly with other
components of the system.

Figure 1: Learning algorithms applied to one example. The utterance is ”Five
plus three times two” with denotation 11. In (a), we show the inferred logical
forms from the denotation using dynamic programming. In (b), the candidate
logical forms are filtered by base case set. In (c), we use the ongoing
Seq2Seq model to rank the loss of each candidate logical form and return the
least one for training. (d) shows Seq2Seq model with attention mechanism with
3-layer LSTM on both encoder and decoder side.

##  III Learning from denotations

We use the triple $\langle u,s,d\rangle$ to denote the linguistic objects,
where $u$ is an utterance, $s$ is a logical form and $d$ is the denotation of
$s$. We use $\lfloor u\rfloor$ to represent the translation of utterance into
its logical form, and we use $[\kern-1.49994pt[s]\kern-1.49994pt]$ for the
denotation of logical form $s$. Each training data is composed by the pair
$\langle u,d\rangle$ without explicitly telling its correct logical form $s$.

With denotation as target label of learning, the Seq2Seq model is trained to
put a high probability on $\lfloor u\rfloor$’s that are consistent-logical
forms that execute to the correct denotation $d$. When the space of logical
forms is large, searching for the correct logical form could be cumbersome.
Additionally, different from the previous study which incorporates prior
knowledge such as word embedding [20], object categories [9], our model in
this mathematical expression learning example has no such knowledge and has to
learn the meanings of input utterance.

Here we formally describe the methods to reduce the searching space and how to
infer the correct logical forms from denotations.

###  III-A Dynamic programming on denotations

Our first step is to generate all logical forms that have the correct
denotations. Formally, given a denotation $d$, we wish to generate a candidate
logical form set that satisfy the denotation demand
$\Omega=\\{s|[\kern-1.49994pt[s]\kern-1.49994pt]=d\\}.$ Previous work use beam
search to generate candidates but it is hard to recover the full set $\Omega$
due to pruning. Noticing that one denotation may correspond to multiple
logical forms, which leads to the increase of the number of distinct
denotations is much slower than the number of logical forms. We use dynamic
programming on denotations to recover the full set, following the work of [15,
21]. A necessary condition for dynamic programming to work is denotationally
invariant semantic function $g$, such that the denotation of the resulting
logical form $g(s_{1},s_{2})$ only depends on the denotations of $s_{1}$ and
$s_{2}$. In the arithmetic domain, the result of an equation can be computed
recursively and independently which certainly satisfies this requirement.

Our primary purpose is to collapse logical forms with the same denotation
together, so that given a denotation $d$ the candidate set $\Omega$ (Figure 1.
(a)) can be returned directly. In order to speed up the lookup efficiency, we
store the pair $(d,\Omega)$ in advance.

###  III-B Filter candidate logical forms

In order to filter out incorrect logical forms, we utilize the fact that
similar utterances should have similar logical forms to reduce the searching
space. We build up a base case set $B$ (Table I) that stores several $\langle
u_{b},s_{b}\rangle$ pairs with varying utterance length. Specifically, given
an input pair $\langle{u_{i},d_{i}}\rangle$, we iterate the base case set to
find the one which shares the most similarity with input utterance $u_{i}$.

${\tilde{u}_{b}}=\mathop{\arg\min}\limits_{{u_{b}}\in
B}e(\phi({u_{b}}),\phi({u_{i}})),$ (11)

where $\phi$ extracts features from utterance $u_{b}$ and $u_{i}$, here we use
bag-of-words as feature extraction function, and we simply counts the shared
features as the feature similarity measurement function $e$.

After finding the base case utterance ${\tilde{u}_{b}}$ which is closest to
the input utterance $u_{i}$, we use the corresponding base case logical form
${\tilde{s}_{b}}$ to filter the candidate set $\Omega$.

$\Gamma=\\{s|s=\mathop{\arg\max}\limits_{{s_{j}}\in\Omega}\frac{{e(\phi({{\tilde{u}}_{b}}),\phi({u_{i}}))}}{{e(\phi({{\tilde{s}}_{b}}),\phi({s_{j}}))}}\\}.$
(12)

We use a new set $\Gamma$ (Figure 1. (b)) to store these updated candidate
logical forms that have similar features with the base case
$\langle\tilde{u}_{b},\tilde{s}_{b}\rangle$. To further determine the most
probable one from set $\Gamma$, we use the Seq2Seq model to examine the loss
of each candidate and select the one with least loss to return for training.

${\tilde{s}_{i}}=\mathop{\arg\max}\limits_{s\in\Gamma}{p_{Seq2Seq}}(s)$ (13)

The selected logical form $\tilde{s}_{i}$ is paired with its utterance to form
a training example $\langle{u_{i},\tilde{s}_{i}}\rangle$ which feeds back to
the Seq2Seq model for training (Figure 1. (c) - (d)).

##  IV Experimental studies

###  IV-A Dataset

For the experiments, we randomly generate 8000 utterances with varying length
from 3 to 7, split into a training set of 6000 training examples and 2000 test
examples. Each utterance consists of integers from one to five and four
operators ‘plus’, ‘minus’, ‘times’, ‘divide’. Every utterance represents a
legal arithmetic expression. Even the scope is quite limited, the searching
space for an equation with length equal to seven is still numerous:
$5^{4}*4^{3}=4*10^{4}$, which is caused by the rich compositionality of
arithmetic equations. Besides, this nature gives rise to one denotation can
correspond to much more logical forms compared with other domains, resulting
in increasing noises for inference.

We select two ways to represent logical forms, both of them are represented by
Arabic numerals and executable operators, but one is inserted with brackets to
denote the calculation order, linearized from tree structure, the other
assumes knowing calculation order without brackets for denotation. The base
case set consists of seven samples, the brackets are omitted directly when
considering logical forms with calculation knowledge (Table I).

The involvement of brackets largely increases the meaning of our logical form,
for the reason that it actually represents tree structure with logic
reasoning. Besides, not only has the model to learn soft alignment for
brackets which are not introduced in the utterance explicitly, but it has to
learn the brackets matching relationship.

###  IV-B Settings

For the convenience of data preprocessing and vectorization, each input
sentence is appended an ending mark $\langle{eos}\rangle$, noting the end of
input. Also, we manually set a maximum length for the input and output
sentence, where the blank position will be automatically filled by ‘PAD’ mark.

We construct 3 layers of LSTM on both encoder and decoder side with 20 hidden
units for each layer. An embedding and a softmax layer is inserted as the
first and last layer. Dropout is used for regularizing the model with a
constant rate 0.3. Dropout operators are used between different LSTM layers
and for the hidden layers before the softmax classifier. This technique can
significantly reduce overfitting, especially on datasets of small size.
Dimensions of hidden vector and word embedding are set to 20.

We use the RMSProp algorithm to update parameters with learning rate 0.001 and
smoothing constant 0.9. Parameters are randomly initialized from a uniform
distribution $\mathcal{U}(-0.05,0.05)$. We run the model for 200 epochs.

###  IV-C Results and analysis

We report the results with the Seq2Seq model on two variants, i.e., with
brackets noting calculation order and without brackets as logical forms. To
compare the performance of weak supervision, we also test the performance of
traditional training with gold standard logical form. The result is reported
on the accuracy of denotation $-$ the portion of input sentences are
translated into the correct denotation. Table II presents the comparison.

TABLE II: Overall accuracy on denotation Method | Seq2Seq
---|---
without brackets | with brackets
Train with logical form | 100.0% | 92.4%
Train with denotation | 72.7% | 69.2%

Overall, the accuracy with logical forms as supervision is much higher than
with denotations, for the reason that training by gold logical forms does not
bring any noises. On the other hand, the spurious logical forms affect our
model’s performance unavoidably. For instance, an utterance “Five plus three
times four” can be mistakenly translated into $[5+(4*3)]$, which has the
correct denotation. This is reasonable because we adopt bag-of-words to
measure similarity and this method does not take the order into consideration.
Besides, not all of the training logical forms returned by our inference
process are correct, but at least they have the correct denotation. So it is
noticeable that even only approximately 45% logical forms (Figure 2) returned
for training are correct, the accuracy on denotation of our model is much
higher than this limit.

To find the best logical form for training, the final pick decision is made by
the ongoing training Seq2Seq model, this training-by-prediction policy makes
the model difficult to converge (Figure 2). But we can see this policy is
effective to correct its previous prediction, which we call the ability of
self-correction. This proves Seq2Seq model can learn word meanings and
compositionality with noises.

Figure 2: The percentage of correct logical form returned for training as the
number of epoch increases.

In addition, the performance of training on logical forms with brackets is
inferior to the model without brackets, which makes sense that longer sequence
add difficulty for learning and brackets introduce more complicated mapping
relationships. However, by utilizing soft alignment, the model can learn tree
structure successfully with only denotation as supervision.

##  V Conclusion

In this paper, we present an encoder-decoder neural network model for mapping
natural language to their meaning representations with only denotation as
supervision. We use dynamic programming to infer logical forms from
denotations, and utilize similarity measurement to reduce the searching space.
Also, curriculum learning strategy is adopted to smooth and accelerate the
learning process. Under the policy training-by-predictions, our model has the
ability of self-correction. We apply our model to the arithmetic domain and
experimental results show that this model can learn word meanings and
compositionality without resources to domain- or representation-specific
features. One major problem remained in our work is that the model may confuse
the order of predictions, which is caused by the inherent weakness of bag-of-
words similarity measurement. This could be enhanced by some sequence-based
similarity measurement in future work.

Although the example we test is rather simple, the expansibility to other
fields and application scenarios is promising due to the few hand-engineered
features and its capability of learning structured form. It would be
interesting to learn a question-answering model with only question-answer
pairs, or apply it to robot navigation task. We expect to extend our model to
these fields and continue to enrich it.

## References

  * [1] S. Miller, D. Stallard, R. Bobrow, and R. Schwartz, “A fully statistical approach to natural language interfaces,” in _Proceedings of the 34th annual meeting on Association for Computational Linguistics_ , 2002, pp. 55–61.
  * [2] R. Ge and R. J. Mooney, “A statistical semantic parser that integrates syntax and semantics,” in _Conference on Computational Natural Language Learning_ , 2005, pp. 9–16.
  * [3] W. Lu, H. T. Ng, W. S. Lee, and L. S. Zettlemoyer, “A generative model for parsing natural language to meaning representations,” in _Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 October 2008, Honolulu, Hawaii, Usa, A Meeting of Sigdat, A Special Interest Group of the ACL_ , 2008, pp. 783–792.
  * [4] L. Zettlemoyer, “Learning to map sentences to logical form,” _Eprint Arxiv_ , pp. 658–666, 2010.
  * [5] J. Clarke, D. Goldwasser, M. W. Chang, and D. Roth, “Driving semantic parsing from the world’s response,” in _Fourteenth Conference on Computational Natural Language Learning_ , 2010, pp. 18–27.
  * [6] P. Liang, M. I. Jordan, and D. Klein, “Learning dependency-based compositional semantics,” in _Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 2011, pp. 590–599.
  * [7] J. Krishnamurthy and T. M. Mitchell, “Weakly supervised training of semantic parsers,” in _Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , 2012, pp. 754–765.
  * [8] S. Reddy, M. Lapata, and M. Steedman, “Large-scale semantic parsing without question-answer pairs,” _Transactions of the Association for Computational Linguistics_ , vol. 2, pp. 377–392, 2014.
  * [9] Y. Artzi and L. Zettlemoyer, “Weakly supervised learning of semantic parsers for mapping instructions to actions,” _Transactions of the Association for Computational Linguistics_ , vol. 1, pp. 49–62, 2013.
  * [10] Y. Wang, J. Berant, and P. Liang, “Building a semantic parser overnight,” in _Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , 2015, pp. 1332–1342.
  * [11] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” _Advances in Neural Information Processing Systems_ , vol. 4, pp. 3104–3112, 2014.
  * [12] K. Cho, B. V. Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _Computer Science_ , 2014.
  * [13] O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton, “Grammar as a foreign language,” _Eprint Arxiv_ , 2014.
  * [14] K. M. Hermann, T. Kočiský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” _Computer Science_ , 2015.
  * [15] P. Pasupat and P. Liang, “Inferring logical forms from denotations,” _arXiv preprint arXiv:1606.06900_ , 2016.
  * [16] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _Computer Science_ , 2014.
  * [17] L. Dong and M. Lapata, “Language to logical form with neural attention,” in _Meeting of the Association for Computational Linguistics_ , 2016, pp. 33–43.
  * [18] P. Pasupat and P. Liang, “Compositional semantic parsing on semi-structured tables,” _arXiv preprint arXiv:1508.00305_ , 2015.
  * [19] N. Lao, T. Mitchell, and W. W. Cohen, “Random walk inference and learning in a large scale knowledge base,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 2011, pp. 529–539.
  * [20] C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao, “Neural symbolic machines: Learning semantic parsers on freebase with weak supervision,” 2016\.
  * [21] T. Lau, S. A. Wolfman, P. Domingos, and D. S. Weld, “Programming by demonstration using version space algebra,” _Machine Learning_ , vol. 53, no. 1, pp. 111–156, 2003.
  * [22] P. Liang and C. Potts, “Bringing machine learning and compositional semantics together,” _Annual Review of Linguistics_ , vol. 1, no. 1, pp. 355–376, 2015\.
  * [23] A. Neelakantan, Q. V. Le, and I. Sutskever, “Neural programmer: Inducing latent programs with gradient descent,” _Computer Science_ , 2015.
  * [24] A. Neelakantan, Q. V. Le, M. Abadi, A. Mccallum, and D. Amodei, “Learning a natural language interface with neural programmer,” 2016.
  * [25] A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” _Computer Science_ , 2014.
  * [26] S. Hochreiter and J. Schmidhuber, “Long short-term memory.” _Neural Computation_ , vol. 9, no. 8, p. 1735, 1997.
  * [27] K. Cho, B. V. Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” _Computer Science_ , 2014.