\begin{table}[h]
\centering
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Mode }         & \multicolumn{3}{c}{\textbf{Accuracy}} \\ \midrule
              & \textbf{345M }    & \textbf{1.3B    }&\textbf{ 5B }     \\ \midrule
ZS - Original & 0.08     & 0.13    & 0.21    \\ 
ZS - Modified & 0.755    & 0.762   & 0.787   \\ \bottomrule
\end{tabular}
\caption{Zero shot (ZS) performance on utterances from \textit{Bus Bookings}. \textbf{Original} refers to having the canonical form for flight bookings as \textit{search tickets for flight one way} which led to incorrect generalizations. \textbf{Modified} refers to having the improved canonical form for flight bookings as \textit{search for flights one way}.  }
\label{tab:ood-sgd-original}
\end{table}
\subsection{Datasets}
\label{sec:datasets}
We consider two widely known datasets in the dialogue community, the Schema Guided Dialogue (SGD) dataset \citep{sgd-dataset} and the Virtual Assistant dataset \citep{assistant-dataset}.\\\\
\textbf{Schema Guided Dialogue} - This dataset covers 16 domains and has over 16k annotated conversations. The domains span a variety of user actions, including setting calendars and alarms, travel booking (car rentals, flights, buses and trains), music, weather, movies, and more. The dataset also contains \textit{multi-domain} dialogues where the utterances switch between domains. For the purpose of our experiments, we consider only the \textit{single-domain} dialogues with 37 intents across all utterances. \\\\
\textbf{Virtual Assistant Dataset} - This dataset covers 21 domains with 64 intents across all utterances. As the name suggests, the domains relate to user queries over a wide range of topics, including operating smart-home devices, media consumption, weather and travel. It has over 25k annotated user utterances that identify intents and slot values.

\begin{itemize}
    \item Using Fasttext Embeddings \citep{bojanowski2016enriching}: We take the mean of all the embedding vectors of the generated canonical form and consider the vector obtained to be the representation of the whole sequence. We compute similar vectors for all the canonical form labels and consider the canonical form label that has the maximum cosine similarity with the generated one as the model's prediction. 
    \item Using Sentence Transformers \citep{reimers-2019-sentence-bert}: We use the \textit{miniLM-QA} \citep{minilm} transformer model that has been pretrained on multiple datasets on the text entailment/semantic search task, \textit{i.e.}, given a query and a set of keys (documents/labels), it ranks the keys in order of relevance. We give as input to the model the generated canonical form (query) and the list of canonical form labels (keys). The model then returns the closest canonical form label to the generated canonical form which we consider as the prediction.  
\end{itemize}

\begin{itemize}
    \item \textbf{BERT-based finetuned model} (Intent Classification): We finetune BERT models on the datasets described in section \ref{sec:datasets}. While some of the Megatron-GPT models we use are larger than the BERT model in terms of number of parameters, it should be noted that the LM parameters are frozen during the training stage of p-tuning and only the weights of the LSTM (~14M parameters) are updated.
\end{itemize}

\begin{itemize}
    \item \textbf{Schema Guided Dialogue} (SGD): We hold out utterances corresponding to \textit{bus bookings} and \textit{hotel reservations} to form our test set. The train set includes utterances from adjacent domains: flight booking and restaurant reservations. This should be a relatively easy setting for the language model to generalize to.
    \item \textbf{Virtual Assistant}: To make things more challenging, we hold out utterances corresponding to \textit{operating IOT devices} and \textit{media consumption commands} (\textit{e.g.}, commands that are variants of "play" - play movie, play audiobook). The train set does not have utterances from similar domains and this setting is more challenging for the model.
\end{itemize}

\begin{itemize}
    \item \textbf{Zero-shot}: P-tune the model on the train set and evaluate zero-shot on the unseen domain test set.
    \item \textbf{Few-shot}: After p-tuning on the train set, we do a second stage p-tuning on a set of \textit{k} samples from the target domain. Unless otherwise noted, \textit{k} here is 5, 10, 50 or 100 samples. 
\end{itemize}

\subsubsection{Out-of-Domain}
The out-of-domain setting is where the advantage of using a LLM becomes apparent. It is not feasible to expect a finetuned BERT model to generalize to an unseen domain not present in the train set. Such models continue to predict that the intent belongs to one of the intent labels they see during training. The p-tuned Megatron-GPT models, on the other hand, show impressive zero-shot and few-shot generalization capabilities on the SGD dataset (Table \ref{tab:ood-sgd}). For instance, having seen intents such as \textit{"buy flight roundtrip tickets"} when presented with utterances for \textit{Flight Reservations} in training, we can expect the model to reasonably generalize to utterances from \textit{Bus Reservations} with utterances like "Get me a return trip on the bus" with the model's prediction for the intent being \textit{"buy bus roundtrip tickets"}.

\subsection{How important is framing the right canonical form?}
The phrasing of canonical forms has a significant impact on zero-shot cross domain generalization. In our initial experiments, we observed that the language models, especially the smaller ones, sometimes rely on spurious correlations to predict the intent. For instance, if the intent \textit{SearchFlightOneWay} is mapped to the canonical form \textit{search tickets for flight one way}, the model correlates the word \textit{ticket} in both the user utterance and canonical form to identify the intent. When we use this model to predict the intent of user utterances related to \textit{bus bookings} in a zero-shot manner, the model predicts that that the intent is related to a \textit{flight booking} as most utterances in the \textit{bus domain} contain the word \textit{ticket}. 

Rephrasing the canonical form for the intent \textit{SearchFlightOneWay} to \textit{search for flights one way} helps the model to avoid making the spurious correlation and the performance in the zero-shot setting (Table \ref{tab:ood-sgd-original}) is significantly improved.
{"value": "345M", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "1.3B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "5B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "0.08", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Original"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.13", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Original"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.21", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Original"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.755", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Modified"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.762", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Modified"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.787", "type": "Result", "training data/set": ["SGD", "Schema Guided Dialogue", "flight booking restaurant reservations"], "test data/set": "SGD Bus Bookings", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "ZS - Modified"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}}