\begin{table}[h]
\centering
\begin{tabular}{@{}lcc@{}}
\toprule
Model              & SGD  & Assistant \\ \midrule
BERT-Large         & 0.88 & 0.91      \\\midrule
Megatron-GPT - 345M & 0.87 & 0.88      \\
Megatron-GPT - 1.3B & 0.91 & 0.92      \\ 
Megatron-GPT - 5B   & 0.95 & 0.94      \\ \bottomrule
\end{tabular}
\caption{Classification Accuracy on test sets of the SGD and Assistant datasets}
\label{tab:indomain}
\end{table}
We observe that using such \textit{canonical forms} as labels for the intent classification task allows the model to generalize better to domains that are adjacent, but not seen at train time (\textit{e.g.}, \textit{Flight Reservations} $\rightarrow$ \textit{Bus Bookings}). We also find that it is beneficial to do a two-stage P-tuning for domain adaption, \textit{i.e.}, once we have a p-tuned large language model on a wide set of domains, we can continue p-tuning this model on a small set of labelled samples from the target domain to allow the model to generalize better. We find that this few-shot approach works very well and this has promising implications for developers for dialogue systems; with minimal effort it would be feasible to adapt an existing model pre-trained on multiple domains to a new domain. In summary, our contributions are:

\subsection{Datasets}
\label{sec:datasets}
We consider two widely known datasets in the dialogue community, the Schema Guided Dialogue (SGD) dataset \citep{sgd-dataset} and the Virtual Assistant dataset \citep{assistant-dataset}.\\\\
\textbf{Schema Guided Dialogue} - This dataset covers 16 domains and has over 16k annotated conversations. The domains span a variety of user actions, including setting calendars and alarms, travel booking (car rentals, flights, buses and trains), music, weather, movies, and more. The dataset also contains \textit{multi-domain} dialogues where the utterances switch between domains. For the purpose of our experiments, we consider only the \textit{single-domain} dialogues with 37 intents across all utterances. \\\\
\textbf{Virtual Assistant Dataset} - This dataset covers 21 domains with 64 intents across all utterances. As the name suggests, the domains relate to user queries over a wide range of topics, including operating smart-home devices, media consumption, weather and travel. It has over 25k annotated user utterances that identify intents and slot values.

\begin{itemize}
    \item Using Fasttext Embeddings \citep{bojanowski2016enriching}: We take the mean of all the embedding vectors of the generated canonical form and consider the vector obtained to be the representation of the whole sequence. We compute similar vectors for all the canonical form labels and consider the canonical form label that has the maximum cosine similarity with the generated one as the model's prediction. 
    \item Using Sentence Transformers \citep{reimers-2019-sentence-bert}: We use the \textit{miniLM-QA} \citep{minilm} transformer model that has been pretrained on multiple datasets on the text entailment/semantic search task, \textit{i.e.}, given a query and a set of keys (documents/labels), it ranks the keys in order of relevance. We give as input to the model the generated canonical form (query) and the list of canonical form labels (keys). The model then returns the closest canonical form label to the generated canonical form which we consider as the prediction.  
\end{itemize}

\begin{itemize}
    \item \textbf{BERT-based finetuned model} (Intent Classification): We finetune BERT models on the datasets described in section \ref{sec:datasets}. While some of the Megatron-GPT models we use are larger than the BERT model in terms of number of parameters, it should be noted that the LM parameters are frozen during the training stage of p-tuning and only the weights of the LSTM (~14M parameters) are updated.
\end{itemize}

\subsubsection{In-Domain}
This setting corresponds to the traditional dataset splits where the train and test sets come from similar distributions. We p-tune the Megatron-GPT models on the train set and evaluate them on the test set for intent classification.

\begin{itemize}
    \item \textbf{Schema Guided Dialogue} (SGD): We hold out utterances corresponding to \textit{bus bookings} and \textit{hotel reservations} to form our test set. The train set includes utterances from adjacent domains: flight booking and restaurant reservations. This should be a relatively easy setting for the language model to generalize to.
    \item \textbf{Virtual Assistant}: To make things more challenging, we hold out utterances corresponding to \textit{operating IOT devices} and \textit{media consumption commands} (\textit{e.g.}, commands that are variants of "play" - play movie, play audiobook). The train set does not have utterances from similar domains and this setting is more challenging for the model.
\end{itemize}

\begin{itemize}
    \item \textbf{Zero-shot}: P-tune the model on the train set and evaluate zero-shot on the unseen domain test set.
    \item \textbf{Few-shot}: After p-tuning on the train set, we do a second stage p-tuning on a set of \textit{k} samples from the target domain. Unless otherwise noted, \textit{k} here is 5, 10, 50 or 100 samples. 
\end{itemize}

\subsubsection{In-domain}
We find that both the p-tuned GPT model as well as the BERT baseline perform very well on the standard in-domain split where both the train and test set come from the same distribution (Table \ref{tab:indomain}). The classification accuracy of Megatron-GPT increases as we increase the model size. The trend of results remains consistent for both the SGD and Assistant datasets. 
{"value": "0.88", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT-Large", "model settings": {"xx": "yy"}} {"value": "0.91", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant"], "test data/set": ["Assistant", "Virtual Assistant"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT-Large", "model settings": {"xx": "yy"}} {"value": "0.87", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 345M", "Megatron-GPT"], "model settings": {"model size": "345M"}} {"value": "0.88", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant"], "test data/set": ["Assistant", "Virtual Assistant"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 345M", "Megatron-GPT"], "model settings": {"model size": "345M"}} {"value": "0.91", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 1.3B", "Megatron-GPT"], "model settings": {"model size": "1.3B"}} {"value": "0.92", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant"], "test data/set": ["Assistant", "Virtual Assistant"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 1.3B", "Megatron-GPT"], "model settings": {"model size": "1.3B"}} {"value": "0.95", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 5B", "Megatron-GPT"], "model settings": {"model size": "5B"}} {"value": "0.94", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant"], "test data/set": ["Assistant", "Virtual Assistant"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["Megatron-GPT - 5B", "Megatron-GPT"], "model settings": {"model size": "5B"}}