\begin{table}[h]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{@{}l|c|ccc@{}}
\toprule
\textbf{\#Samples/Intent} & \textbf{Train Size} & \multicolumn{3}{c}{\textbf{Accuracy}} \\ \midrule
                          && 345M        & 1.3B        & 5B        \\\midrule
10                         & 370 & 0.77       & 0.81       & 0.827      \\
20                        & 740 & 0.82        & 0.83        & 0.844      \\
30                        & 1110 & 0.84        & 0.85        & 0.87     \\\bottomrule
\end{tabular}
}
\caption{Accuracy on the SGD test set when using only k samples per intent. The columns indicate the size of the Megatron-GPT model used.}
\label{tab:sgd-fs-indomain}
\end{table}
\begin{itemize}
    \item \textbf{Schema Guided Dialogue} (SGD): We hold out utterances corresponding to \textit{bus bookings} and \textit{hotel reservations} to form our test set. The train set includes utterances from adjacent domains: flight booking and restaurant reservations. This should be a relatively easy setting for the language model to generalize to.
    \item \textbf{Virtual Assistant}: To make things more challenging, we hold out utterances corresponding to \textit{operating IOT devices} and \textit{media consumption commands} (\textit{e.g.}, commands that are variants of "play" - play movie, play audiobook). The train set does not have utterances from similar domains and this setting is more challenging for the model.
\end{itemize}

\begin{itemize}
    \item \textbf{Zero-shot}: P-tune the model on the train set and evaluate zero-shot on the unseen domain test set.
    \item \textbf{Few-shot}: After p-tuning on the train set, we do a second stage p-tuning on a set of \textit{k} samples from the target domain. Unless otherwise noted, \textit{k} here is 5, 10, 50 or 100 samples. 
\end{itemize}

\subsubsection{In-domain}
We find that both the p-tuned GPT model as well as the BERT baseline perform very well on the standard in-domain split where both the train and test set come from the same distribution (Table \ref{tab:indomain}). The classification accuracy of Megatron-GPT increases as we increase the model size. The trend of results remains consistent for both the SGD and Assistant datasets. 

\subsubsection{Out-of-Domain}
The out-of-domain setting is where the advantage of using a LLM becomes apparent. It is not feasible to expect a finetuned BERT model to generalize to an unseen domain not present in the train set. Such models continue to predict that the intent belongs to one of the intent labels they see during training. The p-tuned Megatron-GPT models, on the other hand, show impressive zero-shot and few-shot generalization capabilities on the SGD dataset (Table \ref{tab:ood-sgd}). For instance, having seen intents such as \textit{"buy flight roundtrip tickets"} when presented with utterances for \textit{Flight Reservations} in training, we can expect the model to reasonably generalize to utterances from \textit{Bus Reservations} with utterances like "Get me a return trip on the bus" with the model's prediction for the intent being \textit{"buy bus roundtrip tickets"}.

Rephrasing the canonical form for the intent \textit{SearchFlightOneWay} to \textit{search for flights one way} helps the model to avoid making the spurious correlation and the performance in the zero-shot setting (Table \ref{tab:ood-sgd-original}) is significantly improved.

However, the few-shot setting (Table \ref{tab:ood-sgd-original-fs}) alleviates this problem of sensitivity of the model to the canonical form structure. When we provide the model with a few samples from the the target domain, it learns to associate that the important words to distinguish between the domains are \textit{flight} and \textit{bus} and not \textit{ticket}.

\begin{itemize}
    \item \textbf{Similarity in structure}: Use similar verbs for similar actions/domains, \textit{e.g.}, \textbf{book} a flight, \textbf{book} bus tickets, \textbf{search} for hotels, \textbf{search} for restaurant reservations.
    \item \textbf{ Compositional}: Using similar structures for canonical forms in similar domains naturally lends to compositionality. This makes it easier for the model to generalize in the zero-shot/few-shot setting while still allowing the developers to easily map the generations to a supported service on the backend.
    \item \textbf{Looks like natural language}: Since LLMs are pretrained on very large corpora of natural language, the benefit of pre-training is realized when the canonical forms resemble natural language rather than complex semantic forms. Making discrete intents look more like typical verb phrases brings out the expressive nature of language models. 
\end{itemize}

\subsection{Do we need the entire training set for p-tuning?}
We look for the fewest labelled samples for p-tuning needed to get an accuracy close to accessing the entire train set. We randomly sample \textit{k} samples per intent ($k \in {5, 10, 20, 30}$) to form the train set the model is p-tuned on, and evaluate on the same test set as above. The train and test sets are from the in-domain setting for both SGD (Table \ref{tab:sgd-fs-indomain}) and Assistant (Table \ref{tab:asst-fs-indomain}) datasets.
{"value": "345M", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "1.3B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "5B", "type": "Hyper-parameter/Architecture", "model": "Megatron-GPT", "parameter/architecture name": ["Model Size", "number of parameters"], "dataset": ["SGD", "Schema Guided Dialogue"]} {"value": "10", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples per intent", "# samples / intent"], "dataset features": {"xx": "yy"}} {"value": "370", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples", "size"], "dataset features": {"xx": "yy"}} {"value": "0.77", "type": "Result", "training data/set": "SGD 370 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "10", "train size": "370"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.81", "type": "Result", "training data/set": "SGD 370 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "10", "train size": "370"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.827", "type": "Result", "training data/set": "SGD 370 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "10", "train size": "370"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "20", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples per intent", "# samples / intent"], "dataset features": {"xx": "yy"}} {"value": "740", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples", "size"], "dataset features": {"xx": "yy"}} {"value": "0.82", "type": "Result", "training data/set": "SGD 740 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "20", "train size": "740"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.83", "type": "Result", "training data/set": "SGD 740 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "20", "train size": "740"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.844", "type": "Result", "training data/set": "SGD 740 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "20", "train size": "740"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}} {"value": "30", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples per intent", "# samples / intent"], "dataset features": {"xx": "yy"}} {"value": "1110", "type": "Data Stat.", "dataset": ["SGD", "Schema Guided Dialogue"], "sub-set/group name": "Train", "attribute name": ["number of samples", "size"], "dataset features": {"xx": "yy"}} {"value": "0.84", "type": "Result", "training data/set": "SGD 1110 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "30", "train size": "1110"}, "model": "Megatron-GPT", "model settings": {"Model Size": "345M"}} {"value": "0.85", "type": "Result", "training data/set": "SGD 1110 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "30", "train size": "1110"}, "model": "Megatron-GPT", "model settings": {"Model Size": "1.3B"}} {"value": "0.87", "type": "Result", "training data/set": "SGD 1110 samples", "test data/set": ["SGD", "Schema Guided Dialogue"], "task": "Intent Classification", "metric": "Accuracy", "experimental settings": {"number of samples per intent": "30", "train size": "1110"}, "model": "Megatron-GPT", "model settings": {"Model Size": "5B"}}