air on tv ?

**ContextNQG**: When did fox begin airing ?

**CorefNQG**: When did american idol begin airing ?

**Input 3**: ... the a38 dual-carriageway runs from east to west across the north of the city .

**Human**: What is the a38 called inside the city ?

**ContextNQG**: What is another name for the city ?

**CorefNQG**: What is the city designated as ?

\[fig:example\]

Human Study
-----------

We hired four native speakers of English to rate the systems’ outputs. Detailed guidelines for the raters are listed in the supplementary materials. The evaluation can also be seen as a measure of the quality of the generated dataset (Section \[sec:dataset\]). We randomly sampled 11 passages/paragraphs from the test set; there are in total around 70 question-answer pairs for evaluation.

We consider three metrics — “grammaticality”, “making sense” and “answerability”. The evaluators are asked to first rate the grammatical correctness of the generated question (before being shown the associated input sentence or any other textual context). Next, we ask them to rate the degree to which the question “makes sense” given the input sentence (i.e., without considering the correctness of the answer span). Finally, evaluators rate the “answerability” of the question given 