\begin{table}[!htbp]
\resizebox{\columnwidth}{!}{
\begin{tabular}{@{}lllllll@{}}
\toprule
Mode                    & \multicolumn{3}{l}{IOT devices} & \multicolumn{3}{l}{Media Consumption} \\ \midrule
                        & 345M      & 1.3B     & 5B       & 345M        & 1.3B       & 5B         \\ \midrule
Zero Shot               & 0.096     & 0.011    & 0.022    & 0.037       & 0.008      & 0.012      \\
FS - 10 samples & 0.62      & 0.71     & 0.75     & 0.58            & 0.62           & 0.68            \\
FS - 50 samples & 0.69     & 0.83     & 0.87     & 0.67            & 0.86           & 0.89          \\ \bottomrule     
\end{tabular}
}
\caption{Zero-shot and Few Shot (FS) performance on the held out domains of the Assistant dataset. The columns indicate the size of the Megatron-GPT model.}
\label{tab:ood-asst}
\end{table}
\begin{itemize}
\item We cast the problem of intent classification into a generative approach and rewrite intent labels in a more descriptive format (\textit{canonical forms}).
    \item When using such canonical forms, generative approaches with Large Language Models(LLMs) show promising results when compared with traditional methods for intent classification.
\item Generative models generalize very well to unseen domains in zero-shot and few-shot settings when compared with BERT-style approaches. 
\item We demonstrate the sample efficiency of p-tuning LLMs where we can achieve close to full dataset performance with a fraction of the data.
\end{itemize}

\subsection{Datasets}
\label{sec:datasets}
We consider two widely known datasets in the dialogue community, the Schema Guided Dialogue (SGD) dataset \citep{sgd-dataset} and the Virtual Assistant dataset \citep{assistant-dataset}.\\\\
\textbf{Schema Guided Dialogue} - This dataset covers 16 domains and has over 16k annotated conversations. The domains span a variety of user actions, including setting calendars and alarms, travel booking (car rentals, flights, buses and trains), music, weather, movies, and more. The dataset also contains \textit{multi-domain} dialogues where the utterances switch between domains. For the purpose of our experiments, we consider only the \textit{single-domain} dialogues with 37 intents across all utterances. \\\\
\textbf{Virtual Assistant Dataset} - This dataset covers 21 domains with 64 intents across all utterances. As the name suggests, the domains relate to user queries over a wide range of topics, including operating smart-home devices, media consumption, weather and travel. It has over 25k annotated user utterances that identify intents and slot values.

\begin{itemize}
    \item Using Fasttext Embeddings \citep{bojanowski2016enriching}: We take the mean of all the embedding vectors of the generated canonical form and consider the vector obtained to be the representation of the whole sequence. We compute similar vectors for all the canonical form labels and consider the canonical form label that has the maximum cosine similarity with the generated one as the model's prediction. 
    \item Using Sentence Transformers \citep{reimers-2019-sentence-bert}: We use the \textit{miniLM-QA} \citep{minilm} transformer model that has been pretrained on multiple datasets on the text entailment/semantic search task, \textit{i.e.}, given a query and a set of keys (documents/labels), it ranks the keys in order of relevance. We give as input to the model the generated canonical form (query) and the list of canonical form labels (keys). The model then returns the closest canonical form label to the generated canonical form which we consider as the prediction.  
\end{itemize}

\subsubsection{Out-of-Domain}
In this setting, we aim to explore the generalization capability of LLMs. We hold out certain domains from the train set and use utterances from the held out domains as our test. This helps us understand how well these LLMs can generalize to unseen domains. The held out sets that we consider are:

\begin{itemize}
    \item \textbf{Schema Guided Dialogue} (SGD): We hold out utterances corresponding to \textit{bus bookings} and \textit{hotel reservations} to form our test set. The train set includes utterances from adjacent domains: flight booking and restaurant reservations. This should be a relatively easy setting for the language model to generalize to.
    \item \textbf{Virtual Assistant}: To make things more challenging, we hold out utterances corresponding to \textit{operating IOT devices} and \textit{media consumption commands} (\textit{e.g.}, commands that are variants of "play" - play movie, play audiobook). The train set does not have utterances from similar domains and this setting is more challenging for the model.
\end{itemize}

\begin{itemize}
    \item \textbf{Zero-shot}: P-tune the model on the train set and evaluate zero-shot on the unseen domain test set.
    \item \textbf{Few-shot}: After p-tuning on the train set, we do a second stage p-tuning on a set of \textit{k} samples from the target domain. Unless otherwise noted, \textit{k} here is 5, 10, 50 or 100 samples. 
\end{itemize}

\subsubsection{Out-of-Domain}
The out-of-domain setting is where the advantage of using a LLM becomes apparent. It is not feasible to expect a finetuned BERT model to generalize to an unseen domain not present in the train set. Such models continue to predict that the intent belongs to one of the intent labels they see during training. The p-tuned Megatron-GPT models, on the other hand, show impressive zero-shot and few-shot generalization capabilities on the SGD dataset (Table \ref{tab:ood-sgd}). For instance, having seen intents such as \textit{"buy flight roundtrip tickets"} when presented with utterances for \textit{Flight Reservations} in training, we can expect the model to reasonably generalize to utterances from \textit{Bus Reservations} with utterances like "Get me a return trip on the bus" with the model's prediction for the intent being \textit{"buy bus roundtrip tickets"}.

In the Assistant dataset, the p-tuned models face the same issue as the BERT models: they struggle to generalize to completely unseen domains and the performance is close to random (Table \ref{tab:ood-asst}). Unlike in SGD, the held-out domains do not have sufficiently similar domains in training from which to generalize. However, the few-shot setting holds promise as the performance of the models improves with few samples. Since the held out domains have far more intents compared to the held out domains from the SGD dataset, we employ stratified sampling to ensure that the few-shot examples are representative of all intents in the domain. 
{"value": "345M", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "1.3B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "5B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "345M", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "1.3B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "5B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["Assistant", "Virtual Assistant"]} {"value": "0.096", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.011", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.022", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.037", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.008", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.012", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "Zero Shot", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.62", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.71", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.75", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.58", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.62", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.68", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 10 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.69", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.83", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.87", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant IOT devices", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "0.67", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.86", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.89", "type": "Result", "training data/set": ["Assistant", "Virtual Assistant", "train"], "test data/set": "Assistant Media Consumption", "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"Mode": "FS - 50 samples", "out-of-domain": "true"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}}