give a score on how relevant the generated question is to the given fact. The relevance score helps us gauge whether the question should have been generated or not irrespective of its grammaticality.[^6]

Each question was evaluated by three people scoring grammaticality and relevance on a 5 point Likert scale. The inter-rater agreement (Krippendorff’s co-efficient) among human evaluations was 0.72. The instructions given to the Mturk raters are provided in the Appendix Figure \[fig:mturk\]. The results of the evaluation are shown in Table \[res-table\]. Syn-QG generates a larger number of questions than H&S and performs strongly on grammaticality ratings. Syn-QG is also able to generate highly relevant questions without the use of a ranker. Also, rule-based approaches seem to be much better at generating relevant questions than neural ones.

QG-STEC also used variety and question types as their evaluation criteria and rewarded systems to generate questions meeting a range of specific question types. Syn-QG’s questions cover each of those question types.

Since many times, despite the ability to paraphrase (Table \[vocab-dist-table\]), back-translated outputs tend to change the meaning of the original sentence, we also measured back-translation’s impact on the above QG metrics. We considered questions generated from 50 facts of 