Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred. They can learn from large unlabeled conversation datasets, build a deep understanding of conversational context, and generate a wide variety of responses. This flexibility comes at the cost of control. Undesirable responses in the training data will be reproduced by the model at inference time, and longer generations often don’t make sense. Instead of generating responses one word at a time, we train a classifier to choose from a predefined list of full responses. The classifier is trained on (conversation context, response class) pairs, where each response class is a noisily labeled group of interchangeable responses. At inference, we generate the exemplar response associated with the predicted response class. Experts can edit and improve these exemplar responses over time without retraining the classifier or invalidating old training data.
 Human evaluation of 775 unseen doctor/patient conversations shows that this tradeoff improves responses. Only 12% of our discriminative approach’s responses are worse than the doctor’s response in the same conversational context, compared to 18% for the generative model. A discriminative model trained without any manual labeling of response classes achieves equal performance to the generative model. Task oriented dialogue systems, exemplified by Budzianowski et al. (2018) , tend to solve narrow tasks like restaurant and hotel reservations and require access to a large knowledge base. After each user utterance, these systems run multiple modules which parse the user utterance, to try to fill (slot, value) pairs, and pick an action. This setup is too cumbersome for primary care medical conversations, our setting, because (a) building the external knowledge base would require the enumeration of the very large symptom, diagnosis and remedy spaces and (b) each module requires separate training data in large volumes. The seq2seq group, which we call generative models (GM) 1 require neither labeling nor structured representations of the dialogue state, but manage to learn strong representations of the conversational context with similar content to a knowledge base, according to Petroni et al. (2019) . They have a key drawback, however: there are no mechanisms to ensure high quality responses. Wallace et al. (2019) show that GPT2 (Radford et al., 2019) can be attacked with four word sequences to "spew racist output". Many production chatbots check each word in a generated utterance against a blacklist of curse words, but this fails to solve subtler failure modes. Even in a cooperative setting, typos, inaccuracies, and other frequent mistakes in the training data will be reproduced by the model at inference time. See et al. (2019) find that GM "often repeat or contradict previous statements" and frequently produce generic, boring utterances like "I don't know". Our discriminative approach attempts to remedy these shortcomings by restricting generations to a manageable set of high quality exemplar responses. We ensure that exemplars are all factual, sensible and grammatical by allowing experts to edit them before or after training. For example, if we wanted to switch from recommending users sleep 6-8 hours per night to recommending 7-9 hours, we could simply update the message associated with the output class and the discriminative model would immediately generate the new advice in the same conversational context, without retraining. Experts can also remove response classes with short, generic exemplars before training to redirect responses towards more productive content. For example the class associated with that makes sense, could be removed with the intention of increasing the likelihood of generating that makes sense. How bad is the pain on a 1-10 scale? . We address a key difficulty in this setup -creating non-overlapping response groups that cover a wide range of situations -with weak supervision. A pretrained similarity model merges nearly identical responses into clusters, and a human merges the most frequently occurring of these clusters into larger response classes. To summarize, we propose a system that can generate reasonable responses across multiple domains while restricting generations to a fixed set of high quality responses that are easy to control. We expect our approach to be most useful in task-oriented settings with a wider range of topics, like patient diagnostics and customer service interactions. The paper is organized as follows: Section 2 discusses related conversational agents and their methods. Section 3 documents our approach, with special attention to the procedure for creating a manageable number of response classes that manage to cover a wide range of conversational contexts. Section 4 explains our main results and the results of experiments which compare the quality of responses suggested by different classification architectures and response class generation procedures. Clustering Statistics Preprocessing and filtering yielded 60,000 frequent responses. Candidate Generation (step 2) yielded 1 million pairs for evaluation. Automated clustering (step 4) yielded 40,000 response clusters with high within-group similarity but many overlapping groups; the largest cluster is 10 distinct responses and 87% of clusters contain only one response. In the manual step 5, one labeler created 187 groups from the 3,000 most frequently occurring clusters in 3 hours. This leaves roughly 90% of responses unlabeled. We hypothesize that our automated clustering procedure leaves many interchangeable clusters unmerged because the sentence encoders were trained to encode a sentence's meaning rather than conversational impact. For example, no encoder produces ("You're welcome. Hoping for the best.", "Take care, my pleasure.") as a candidate pair. One advantageous property of our approach is that the manual merging step need not be fully completed, unlike Wan & Chen (2018) Given the low correlation of automated metrics such as BLEU score to human judgment of response quality reported in Liu et al. (2016) , a group of medical doctors evaluated the quality of generated responses on the test data. For a given conversational context, evaluators compared the doctor response observed in the data to a model's suggested response in a model-blind setting. Evaluators reported whether a model's response is either (a) equivalent to the true response, (b) different but higher quality, (c) different but equal quality, or (d) different but lower quality. For example, "Hoping for the best, take care." and "Take care!!!!" would be marked equivalent. The results are shown in Tables 2 and 5 . Accuracy for comparing classifiers. Tables 3 and 4 , which compare different classifiers on the same dataset, measure accuracy on unseen labeled data. Generative Baseline For a generative baseline, we use the AWD-LSTM language model 4 before classification finetuning. For this reason, the context representation used by the classification model head and the language model (generative) head tend to be very similar. We use greedy decoding because beam search was too slow to meet our latency requirements. In this work, we propose a classification model that leverages advances in pretraining techniques to generate useful responses in a wide variety of contexts while restricting generations to a fixed, easy to update set of high quality responses. This allows full control of generations, but also helps the average suggested response quality, compared to a generative model with a nearly identical backbone. The key error source for both models, asking the same question twice could be more easily fixed for the discriminative model, by using the second most likely class if the most likely class has already been generated in a conversation. For the generative model, this could be addressed with conditional training, following See et al. (2019) . The key difficulty in this approach, and opportunity for future work, is the creation of response classes. We might be able to obviate the need for a manual merging step, for example, by allowing creation (through merging) of clusters with pairs of constituents that were not generated as candidates or by training a separate classifier to predict which response class an unseen response belongs to. Finally, we intend to test whether the control for flexibility tradeoff provides similar quality improvements in other conversational domains.
Avoid generating responses one word at a time by using weak supervision to training a classifier  to pick a full response. A way to generate responses for medical dialog using a classifier to select from expert-curated responses based on the conversation context.