# Augment Before Copy-Paste: Data and Memory Efficiency-Oriented
Instance Segmentation Framework for Sport-scenes

1 Chih-Chung Hsu 2 Chia-Ming Lee 3Ming-Shyen Wu
Institute of Data Science, National Cheng Kung University
No.1, University Rd. Tainan City, Taiwan
<EMAIL_ADDRESS><EMAIL_ADDRESS>3<EMAIL_ADDRESS>
###### Abstract

Instance segmentation is a fundamental task in computer vision with broad
applications across various industries. In recent years, with the
proliferation of deep learning and artificial intelligence applications, how
to train effective models with limited data has become a pressing issue for
both academia and industry. In the Visual Inductive Priors challenge
(VIPriors2023), participants must train a model capable of precisely locating
individuals on a basketball court, all while working with limited data and
without the use of transfer learning or pre-trained models. We propose Memory
effIciency inStance Segmentation framework based on visual inductive prior
flow propagation that effectively incorporates inherent prior information from
the dataset into both the data preprocessing and data augmentation stages, as
well as the inference phase. Our team (ACVLAB) experiments demonstrate that
our model achieves promising performance (0.509 AP@0.50:0.95) even under
limited data and memory constraints.

##  1 Introduction

Instance segmentation, a cornerstone of computer vision within deep learning,
boasts diverse applications like pedestrian detection and multi-object
tracking. With the rise of deep learning, industries are integrating it into
various aspects of their businesses to gain a competitive edge. However,
implementing deep learning at these granular levels faces challenges such as
insufficient annotated data and limited computational resources. These
constraints can lead to models performing below expectations. Therefore,
effectively leveraging limited data to train models has become one of today’s
research hotspots. The ’VIPriors: Visual Inductive Priors for Data-Efficient
Deep Learning’ workshop [14][1] introduces a challenge, urging participants to
build generalizable models by fusing dataset-specific prior knowledge in
resource-scarce contexts. Notably, the use of pre-trained models is
disallowed.

In the instance segmentation track, the goal is to predict players,
basketballs, referees and coaches in a basketball court. Task-specific data
augmentation based on instance traits yields substantial performance gains,
validated in the challenge. Nevertheless, conventional state-of-the-
art[13][17] prioritize performance without due regard for computational
resources. We propose a ”augmentation before copy-paste” pipeline with RGB
distortion and geometry transformation before copy-paste augmentation,
exploiting object-derived semantic representation effectively.

Furthermore, we remove redundant image area to find out a region of interests.
Utilizing prior knowledge from basketball court backgrounds—often defined by
rectangular boundary lines, we extract court areas for training and inference
on smaller images to minimize the computational consumption without feature
distortion.

Figure 1: The overall of proposed instance segmentation framework. The visual
inductive prior is fully utilized at each stage to make effective
optimizations. This approach not only reduces computational resource
consumption but also maintains solid model performance. We begin by employing
the Canny-Hough operator to adaptively combine image-level prior to detect the
basketball court’s position. Subsequently, we leverage class-level prior for
identity identification. We then utilize this information for style
transformation of various objects, integrating image-level prior knowledge
through copy-paste augmentation. Finally, model inference solely based on the
detected basketball court’s location.

In recent years, deep learning-based instance segmentation methods have
garnered widespread recognition, exemplified by approaches such as Cascade-
MaskRCNN[2], Maskformer[6], queryinst[10], and CBNet[15], among others. Over
the past couple of years, [13][17]. have demonstrated remarkable achievements
within the VIPriors Workshop through the CBNet-based model architectures.

CBNet is distinguished by its capability to synergistically couple multiple
backbone networks and detectors, enabling the effective fusion of both low-
level and high-level semantic representations. This architecture,
characterized by its scalability and ease of training, imparts a diverse range
of model inductive attributes. It accomplishes this while maintaining
noteworthy detection precision and generalization ability without compromising
inference speed.

Data augmentation constitutes a critical facet in training deep learning
models, especially when confronted with limited data. By subjecting original
images to operations such as hue variations, geometric transformations, and
erasure transformations, the diversity of features is augmented, enhancing the
model’s capacity for generalization across unseen domains.

Copy-paste augmentation[11] emerges as an effective augmentation strategy for
instance segmentation task. It leverages prior knowledge of objects to enhance
a model’s generalization and robustness to out-of-domain objects. In VIPriors
instance segmentation challenge , [13][17] introduced task-specific copy-paste
augmentation. This procedure leverages object-derived visual prior knowledge
to optimize the data pipeline, ensuring that the features of pasted objects
align more closely with plausible scenarios. For instance, constraints can be
applied to the coordinates of pasted images based on the probable player
positions, effectively filtering out implausible results.

The experimental results confirm that our model can be effectively trained on
a single GPU with 24 GB of memory while maintaining promising performance.

##  2 Methodology

In this section, we elaborate on all the components presented in Figure 1,
including the basketball court detection algorithm, illustration about
augmentation pipeline, and inference on region of interests.

###  2.1 Basketball Court Detection and Cropping

Images with large sizes will prolong training and inference times, and may
lead to memory insufficient. Resizing to a fixed size is common strategy to
handle it. But it may lead to distorting features or losing image texture
details. Conversely, the cropping approach can retain more information from
source images. This method relies more heavily on prior knowledge within the
image to determine the exact cropping boundaries. We introduce a basketball
court detection and cropping algorithm, which is based on prior with canny-
hough straight line detection operator[3][7] to detect the location of
basketball court and reduce image size.

Algorithm 1 Basketball Court Detection Algorithm

1:Data: All image data denoted as $\textbf{I}_{\text{original}}$, All Cropped
image data denoted as $\textbf{I}_{\text{cropped}}$

2:Denote $\phi(\cdot)$ as canny operator, and $\tau(\cdot)$ as hough operator

3:for each image $\textbf{I}_{i}$ in $\textbf{I}_{\text{original}}$ do

4: Initialize $\textbf{I}_{\text{ih}}$, $\textbf{I}_{\text{iw}}$ = the height
and width of image $\textbf{I}_{i}$

5: $\text{min}_{\text{h}}=\frac{1}{9}\textbf{I}_{\text{ih}}$,
$\text{max}_{\text{h}}=\frac{8}{9}\textbf{I}_{\text{ih}}$,

6: $\text{min}_{\text{w}}=\frac{1}{15}\textbf{I}_{\text{iw}}$,
$\text{max}_{\text{w}}=\frac{14}{15}\textbf{I}_{\text{iw}}$

7: Detect all lines $L$ in images $\tau(\phi(\textbf{I}_{i}))$

8: Compute the maximum convex hull $\delta$ in $L$

9: Crop $\textbf{I}_{i}$ based on the coordinate $(x,y,w,h)=$

10: (min($\text{min}_{\text{w}}$, $\delta_{x}$), max($\text{min}_{\text{h}}$,
$\delta_{y}$) - 50,

11: max($\text{min}_{\text{w}}$, $\delta_{w}$), min($\text{max}_{\text{h}}$,
$\delta_{h}$))

12:end for

Figure 2: The illustrations for the cropping algorithm. The left figure is the
original image. The right one is cropped, with red lines detected by the Canny
edge detector and Hough transform. The blue line shows a boundary based on
image size, while the green lines indicate dynamic boundary from the detected
lines.

###  2.2 Identity Identification

To further optimize the data augmentation pipeline, it is essential to make
more effective use of prior knowledge. We have observed that referees and
coaches generally stand around the perimeter of the basketball court for a
better view of the players’ movements, while the players themselves are active
within the interior of the court. This information can be effectively utilized
to estimate the likely identity of objects through basketball court area
detection. Specifically, we consider 20% of the detected area as a decision
boundary and identify the objects based on the bottom coordinates of their
detection boxes.

Figure 3: The left figure displays a region identified based on the maximum
convex hull, which is determined using the endpoints of all lines detected by
the Canny-Hough operator. The subclass attributes of the object are determined
by its bounding box coordinates. The object marked by a dotted line represents
the result of location-based copy-paste augmentation.

###  2.3 Identity-based Style Transformation

Previous research in data augmentation presents two issues:(1) whole image-
level augmentation may unnecessarily increase the complexity of the feature
space, resulting in limited performance gains;(2) the data augmentation
procedures for different classes or scenarios are incomplete. Objects on the
basketball court are diverse. The ’human’ class may have different sub-class,
including ’player, referee, coach,’ each with highly diverse internal feature
attributes; the ’ball’ class may also exist in various lighting and occlusion
conditions, as well as variations in the game ball itself. Using a few classes
to simply distinguish object attributes can make it difficult for basic data
augmentation procedures to effectively expand the source domain. Further
leveraging prior knowledge to incrementally decompose high-level classes can
improve the model’s predictive capabilities for unseen targets to a certain
extent.

Specifically, we can distinguish the sub-classes of objects through the
content explained in Section 3-2 and then apply different enhancement
strategies to them. Players may wear clothing of different colors, high
saturation, and strong contrast to increase their distinctiveness, or they may
have varying skin tones or genders. For the ’player’ sub-class, we employ RGB
curve distortion for object-level data augmentation. As for other categories
such as referees, coaches, and balls, where the available prior knowledge is
relatively limited, we resort to using salt-and-pepper noise and brightness
variations to increase the model’s robustness against varying lighting
conditions.

Figure 4: The demo of identity-based style transfer applied to basketball
players. Significant variations in appearance are evident after the hue or RGB
transformation. In the left example, there is a noticeable change in skin
tone, while in the right example, the player’s jersey changes dramatically,
almost as if he has switched to a different team.

###  2.4 Location-based Copy-paste Augmentation

In previous research on copy-paste augmentation, the coordinates of the
objects are constrained within a range determined by the image’s height and
width, as denoted by the blue bounding box in Figure 2. Such restrictive
boundaries might result in objects being augmented in unreasonable locations.
In our study, we vary the possible boundary regions based on prior knowledge.
As described in Sec. 3-2 and 3-3, our proposed algorithm can effectively
determine the areas where various types of objects are likely to appear. We
then perform copy-paste augmentation based on these locations.

###  2.5 Inference on Region of Interests

To better utilize memory usage and reduce computational consumption, we
believe that just inferencing specific region via the vision prior to crop out
redundant area like Sec.3-2, 3-4, is an efficient and effective way to achieve
this goal. We reduce memory usage and inference time during model inference by
resizing images while preserving the region of interest.

###  2.6 Model Architecture

Our model is architecturally founded on the HybridTaskCascade [8] detector.
Subsequent to this foundation, we utilize the CB-SwinTransformer-Base [15][16]
as our backbone to extract image features. After this, we integrate the CB-
feature pyramid network, which employs group normalization [18] as model’s
neck to better capture from low to high-level feature representations. As for
model’s head parts, we use the region proposal network with its default
setting. This is further followed by the inclusion of the
HybridTaskCascadeRoIHead. Within this RoI head, there are two pivotal
components: the bounding-bbox head, which retains its default setting, and the
mask-head. The mask-head is replaced by mask-scoring head [12] to improve
model performance on instance’s texture and boundary details.

##  3 Experiments

###  3.1 Training Details

All our experiments are conducted on a single GPU (NVIDIA TITAN RTX) and are
based on the MMDetection toolbox[4].

Our dataset is provided by Synergy Sports. On VIPriors instance segmentation
challenge, there are 184, 62, 64 images in the training, validation, testing
set.

We train our model with totally 36 epoches. Within this framework, we adopt
the AdamW optimizer, setting the learning rate at 0.0001 and the weight decay
at 0.05. The batch size is set to 1 due to the limitation of GPU memory. We
duplicate training and validation images 10 times, then implement proposed
augmentation to train instance segmentation model.

After undergoing the cropping pipeline, the image sizes of the training set,
validation set, and test set are reduced by 33.98 %, 33.17 %, and 40.72 % of
the original sizes, respectively. We conduct statistics for each basketball
court category, as shown in Figure 5.

On the other hand, in the online augmentation, each training images have a 0.5
probability of undergoing a horizontal flip, and is then randomly resized to
either (1400, 800) or (1400, 1200). Subsequently, 70% of the image area is
randomly cropped from it, and normalization operator is used on each images.
Finally, GridMask[5] augmentation is performed on each images.

Models |<EMAIL_ADDRESS>| AP@0.50:0.95 |  | AP@0.50:0.95
---
(small)
| AP@0.50:0.95
---
(medium)
| AP@0.50:0.95
---
(large)
Vanillia instance segmentation model | 0.789 | 0.403 | 0.401 | 0.470 | 0.631
With simple copy-paste augmentation | 0.863 | 0.444 | 0.462 | 0.561 | 0.667
With proposed augmentation pipeline | 0.870 | 0.481 | 0.515 | 0.579 | 0.700
With post-processing | 0.896 | 0.509 | 0.533 | 0.584 | 0.731
Table 1: Results of Ablation Study for VIPriors instance segmentation
challenge 2023 with or without the proposed augmentation pipeline.

###  3.2 Post-Processing

During the training process, we faced constraints related to limited GPU
memory. Consequently, we abstained from resizing the images to larger
dimensions in both training and testing phases to extract more granular
feature information. This compromise adversely impacted the performance of our
model. To mitigate this drawback, we employed the Stochastic Weight
Averaging[19] strategy to obtain the average model weights over subsequent
training epochs. Further, we applied variable-intensity GridMask and executed
additional data augmentation techniques on the original dataset to generate
diverse training samples. Subsequently, we leveraged model ensemble and
modelsoup[9] to enhance the overall performance of our model.

###  3.3 Ablation Study and Performance Comparison

Our experimental results are presented in Table 1 and 2. The outcomes
demonstrate that our proposed method, based on vision inductive prior, can
effectively surpass the performance established by conventional approaches.

The model shows strong performance on the<EMAIL_ADDRESS>metric, implying that it can
effectively detect the majority of instances in the testing set. However, it
underperforms in terms of fine-grained segmentation. As a result, the overall
performance at the AP@0.50:0.95 metric is slightly below the state-of-the-art
model [17]. Finally, we achieved a final result of 0.509 on the AP@0.50:0.95
metric.

Additionally, our model not only requires significantly less memory usage
compared to [17] , using only 34.6% of its memory, but also maintains
competitive performance.

Figure 5: The cropped area statistic barchart. The x-axis is corresponding to
basketball courts; the y-axis is the cropped area ratio against whole raw
image. From left to right, the three colors correspond to the training,
validation, and testing set.

Methods |  | AP@
---
0.50:0.95
| AP@
---
0.50
| Memory
---
(G)
| Inference times
---
(s)
Yunusov et al. [13] | 0.477 | 0.747 | 27.1 | 6.47
Yan et al. [17] | 0.531 | 0.837 | 65.6 | 6.98
Ours | 0.509 | 0.896 | 22.7 | 3.95

Table 2: Comparison of performance and computational resource requirements. We
compare proposed method with the SOTA from VIP2021, 2022. *The architectures
of these methods are CBNet-based, but the training process or hyperparameters
may vary.

##  4 Conclusion

In this paper, we propose an efficient instance segmentation framework that
integrates visual inductive priors into various stages, including data
preprocessing, data augmentation, and model inference stage. The experiments
demonstrate that such an approach can significantly enhance model performance
even in resource-constrained environments, without the need for any pre-
trained weight or the use of transfer learning.

Notably, increasing the image size during the training and inference time can
toward improve model performance if there is sufficient available memory.

## References

  * [1] Robert-Jan Bruintjes, Attila Lengyel, Marcos Baptista Rios, Osman Semih Kayhan, Davide Zambrano, Nergis Tomen, and Jan van Gemert.  Vipriors 3: Visual inductive priors for data-efficient deep learning challenges.  arXiv preprint arXiv:2305.19688, 2022.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
  * [3] John Canny.  A computational approach to edge detection.  IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  * [4] Kai et al. Chen.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [5] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Gridmask data augmentation.  arXiv:2001.04086, 2020.
  * [6] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In Advances in Neural Information Processing Systems, 2021.
  * [7] Richard O Duda and Peter E Hart.  Use of the hough transformation to detect lines and curves in pictures.  Communications of the ACM, 15(1):11–15, 1972.
  * [8] Kai Chen et al.  Hybrid task cascade for instance segmentation.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4969–4978, 2019.
  * [9] Mitchell Wortsman et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  International Conference on Machine Learning, 2022.
  * [10] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Instances as queries.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6910–6919, October 2021.
  * [11] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
  * [12] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang.  Mask scoring r-cnn.  Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
  * [13] Yunusov Jahongir, Rakhmatov Shohruh, Namozov Abdulaziz, Gaybulayev Abdulaziz, and Kim Tae-Hyong.  Instance segmentation challenge track technical report, vipriors workshop at iccv 2021: Task-specific copy-paste data augmentation method for instance segmentation.  arXiv preprint arXiv:2110.00470, 2021.
  * [14] Attila Lengyel, Robert-Jan Bruintjes, Marcos Baptista Rios, Osman Semih Kayhan, Davide Zambrano, Nergis Tomen, and Jan van Gemert.  Vipriors 2: Visual inductive priors for data-efficient deep learning challenges.  arXiv preprint arXiv:2201.08625, 2021.
  * [15] Tingting Liang, Xiaojie Chu, Yudong Liu, Yongtao Wang, Zhi Tang, Wei Chu, Jingdong Chen, and Haibin Ling.  Cbnet: A composite backbone network architecture for object detection.  Proceedings of the AAAI Conference on Artificial Intelligence, 2022\.
  * [16] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  Proceedings of the IEEE/CVF international conference on computer vision, 2021.
  * [17] Bo Yan, Xingran Zhao, Yadong Li, and Hongbin Wang.  Task-specific data augmentation and inference processing for vipriors instance segmentation challenge.  arXiv preprint arXiv:2211.11282, 2022.
  * [18] Kaiming He Yuxin Wu.  Group normalization.  European Conference on Computer Vision, 2018.
  * [19] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf.  Swa object detection.  arXiv preprint arXiv:2012.12645, 2020.


#  GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot
Filling

Shikib Mehri  Maxine Eskenazi
Language Technologies Institute, Carnegie Mellon University
<EMAIL_ADDRESS>
###### Abstract

In transfer learning, it is imperative to achieve strong alignment between a
pre-trained model and a downstream task. Prior work has done this by proposing
task-specific pre-training objectives, which sacrifices the inherent
scalability of the transfer learning paradigm. We instead achieve strong
alignment by simultaneously modifying both the pre-trained model and the
formulation of the downstream task, which is more efficient and preserves the
scalability of transfer learning. We present GenSF (Generative Slot Filling),
which leverages a generative pre-trained open-domain dialog model for slot
filling. GenSF (1) adapts the pre-trained model by incorporating inductive
biases about the task and (2) adapts the downstream task by reformulating slot
filling to better leverage the pre-trained model’s capabilities. GenSF
achieves state-of-the-art results on two slot filling datasets with strong
gains in few-shot and zero-shot settings. We achieve a 9 $\mathbf{F_{1}}$
score improvement in zero-shot slot filling. This highlights the value of
strong alignment between the pre-trained model and the downstream task.

##  1 Introduction

The advent of pre-trained language models (Devlin et al., 2019; Radford et
al., 2019) has transformed natural language processing. The dominant paradigm
has shifted away from designing task-specific architectures towards transfer
learning. Fine-tuning pre-trained models on downstream datasets achieves
strong performance on a variety of natural language understanding tasks (Wang
et al., 2018). Generally, prior to fine-tuning, the pre-trained models are
adapted to the specifics of the downstream task through minor architectural
modifications (e.g., adding a classification layer) (Chen et al., 2019; Mehri
et al., 2020). By avoiding major task-specific changes to the models, it is
assumed that the underlying pre-trained models possess a degree of generality
that allows transfer to a variety of tasks. We posit that this assumption is
flawed. Consequently this paper demonstrates the importance of incorporating
inductive biases that achieve stronger alignment between the pre-trained model
and the downstream task.

Figure 1: To achieve a stronger alignment, both the downstream task and the
pre-trained models must be adapted. The downstream task can be adapted with
knowledge of the properties and capabilities of the pre-trained models.
Likewise, the pre-trained model can be adapted with knowledge of the
downstream task/data.

Recent work has validated the idea that stronger alignment between pre-
training and the downstream task results in improved performance. Rather than
fine-tuning off-the-shelf models, it is more effective to first understand the
downstream task and adapt the model’s architecture, pre-training and inference
algorithm accordingly. Adapting pre-trained models in this manner is
equivalent to incorporating inductive biases about the downstream task. For
example, pre-training on open-domain dialog data results improves performance
on downstream dialog tasks (Henderson et al., 2019; Mehri et al., 2020).
Designing task-specific pre-training objectives has yielded strong results in
extractive question answering (Glass et al., 2019), paraphrase and translation
(Lewis et al., 2020) and slot filling (Henderson and Vulić, 2020). This body
of work attains stronger alignment by significantly modifying the pre-trained
model through task-specific pre-training. This necessitates a new pre-trained
model for every downstream task, and therefore relinquishes the inherent
scalability of the transfer learning paradigm. Instead, we achieve stronger
alignment by simultaneously adapting both the pre-trained model and the
downstream task, such that both contain inductive biases about one another.

The downstream task can be adapted to achieve stronger alignment with the
capabilities of the pre-trained model. To effectively leverage pre-trained
models, it is important to first understand the properties and capabilities of
the model derived from the model architecture, the pre-training data and task.
Then the downstream task can be adapted to be better aligned with the model.
Adapting the task to the model is equivalent to incorporating inductive biases
about the pre-trained model into the downstream task. For example, given a
pre-trained model that was trained with a ranking objective, it is likely to
be more effective if the downstream fine-tuning and inference algorithms are
modified to rank rather than to classify. By simultaneously adapting both the
downstream task and the pre-trained model, we intend to achieve stronger
alignment without sacrificing the inherent scalability of the transfer
learning paradigm (i.e., avoiding task-specific pre-trained models).

We address the task of slot filling, a natural language understanding task
with the goal of identifying values for pre-defined attributes (slots) in a
natural language utterance. We leverage a DialoGPT (Zhang et al., 2020), a
generative language model, pre-trained on open-domain dialog data. To achieve
strong alignment between the slot filling task and DialoGPT, we (1)
reformulate slot filling as a natural language response generation task, and
(2) augment the DialoGPT architecture with a copy-mechanism, constrained
decoding and a post-processing heuristic. The resulting model, GenSF
(Generative Slot Filling), is shown to achieve state-of-the-art results on two
slot filling datasets. GenSF achieves the strongest performance gains in few-
shot and zero-shot settings, highlighting the importance of stronger alignment
in the absence of abundant data. Our code is open-sourced and can be found at
https://github.com/shikib/generative_slot_filling.

##  2 Related Work

Slot filling is the task of identifying values for pre-defined attributes, or
slots, in a natural language utterance (Tur and De Mori, 2011). Slot filling
is a vital natural language understanding component of task-oriented dialog
systems (Young, 2002, 2010). A variety of architectures have been explored for
the task of slot filling, including CNNs (Vu, 2016), deep LSTMs (Yao et al.,
2014), RNNs with external memory (Peng et al., 2015), encoder labeler LSTMs
(Kurata et al., 2016) and joint pointer and attention seq2seq networks (Zhao
and Feng, 2018). With the introduction of large-scale pre-trained language
models (Devlin et al., 2019; Radford et al., 2019), strong slot filling
results have been achieved with simple architectures (Chen et al., 2019).

Several approaches have been proposed for zero-shot slot filling. Bapna et al.
(2017) leverage slot names and descriptions to align slots across domains.
Shah et al. (2019) leverage examples for zero-shot slot filling. Liu et al.
(2020) achieve strong results in zero-shot slot filling with a coarse-to-fine
approach in combination with template regularization. We use the Coach+TR
model (Liu et al., 2020) as a baseline in our zero-shot experiments.

Working on the hypothesis that pre-trained language models, such as BERT
(Devlin et al., 2019), do not effectively capture the intricacies of dialog,
recent work has attempted to mitigate this issue. Coope et al. (2020) use
ConveRT (Henderson et al., 2019), a lightweight model pre-trained on dialog
data, in combination with CNN and conditional random field (CRF) to outperform
BERT. Mehri et al. (2020) achieves similar results with ConvBERT, a model that
further pre-trains BERT on open-domain dialog data. Recently, Henderson and
Vulić (2020) introduces a ‘pairwise cloze’ pre-training objective that uses
open-domain dialog data to specifically pre-train for the task of slot
filling. The resulting ConVEx model achieves significant improvements,
particularly in few-shot settings. A common theme in recent work is achieving
better alignment between the pre-trained models and the downstream task,
either by pre-training on data that is closer to the domain of the downstream
task (i.e., dialog data) (Henderson et al., 2019; Mehri et al., 2020) or by
designing custom pre-training objectives that better model the downstream task
(Henderson and Vulić, 2020). Our proposed approach shares the goal of
achieving better alignment, but we simultaneously adapt both the pre-trained
model and the downstream task, with the goal of leveraging a generative pre-
trained dialog model, DialoGPT, for slot filling.

##  3 Methods

In order to effectively leverage a pre-trained generative dialog model,
DialoGPT (Zhang et al., 2020), for the task of slot-filling, we introduce the
GenSF model which achieves stronger alignment between the downstream task and
the pre-trained model, by simultaneously adapting the task to the model and
the model to the task. This paper first describes how the slot filling task is
reformulated as a natural language response generation task to be better
aligned with the DialoGPT model. Next, it describes several modifications to
the DialoGPT architecture and inference algorithm that act as inductive biases
for the slot filling task.

###  3.1 Slot Filling as Response Generation

Given an utterance $u=\\{w_{1},w_{2},\dots w_{n}\\}$, a set of possible slot
keys $s=\\{s_{1},s_{2},\dots s_{k}\\}$, and a list of slots requested by the
system $r=\\{r_{1},r_{2},\dots r_{m}\\}$ (where $r_{i}\in s$ and $m\geq 0$),
the task of slot filling is to assign a value to a subset of the slot keys.
Concretely, for a given slot key $s_{i}$, the output will either be NULL or a
contiguous span of words from the utterance: $s_{i}=\\{w_{i},\dots
w_{i+j}\\}$.

In response generation, given a dialog context consisting of a sequence of
utterances: $c=\\{x_{1},x_{2},\dots x_{n}\\}$ wherein each utterance $x_{i}$
is a sequence of words, the task is to generate a valid response
$y=\\{w_{1},w_{2},\dots w_{m}\\}$.

Many tasks can be represented as an input to output mapping (Raffel et al.,
2019; Hosseini-Asl et al., 2020; Peng et al., 2020), making sequence-to-
sequence a universal formulation. Trivially, slot filling can be represented
as a sequence-to-sequence task by setting the context to be the concatenation
of the utterance and the requested slots: $c=\\{u,r\\}$ and the target
response to be the slot mappings $y=\\{(s_{1},w_{i:j}),(s_{2},{\tt
NULL}),\dots(s_{k},(w_{j:n})\\}$. However, this does not leverage the natural
language capabilities of pre-trained dialog models. While this trivial
formulation may suffice with sufficient training, it will under-perform in
few-shot and zero-shot settings. To this end, this paper presents a
reformulation of slot filling that better aligns with the natural language
capabilities of DialoGPT.

We hypothesize that to some degree, large-scale dialog pre-training can result
in a model implicitly learning to fill slots. For example, given the slot key
‘time’, such a model should understand what time is and should be able to
generate a valid time (e.g., ‘4:15 pm’). An effective task formulation can
leverage these implicitly learned slot filling capabilities. An off-the-shelf
pre-trained model is likely to only be capable of filling generic slots (e.g.,
time, date, price, etc.). But by reformulating slot filling in a manner that
is better aligned with the pre-training task, it should be easier for the
model to adapt to novel slot keys.

Concretely, given a slot filling input ($u$, $r$) and a particular slot key
$s_{i}$, we construct a natural language dialog context using a template-based
approach: $c=$ ‘What is the $\\{f(r)\\}$? [eos] $\\{u\\}$ [eos] Ok, the
$\\{f(s_{i})\\}$ is’. Here, $f$ denotes a manually constructed function that
maps slot keys to a natural language phrase (e.g., $first\\_name$: first name,
$departure\\_location$: leaving from). Given the constructed dialog context,
the model is tasked with completing the partial response (i.e., Ok, the
$\\{f(s_{i})\\}$ is) by auto-regressively generating the slot value. During
training the model would be tasked with generating either the slot value or
the phrase not provided. With this natural language reformulation, the slot
filling task is being adapted to better leverage the capabilities of the pre-
trained DialoGPT model. As this achieves better alignment between the pre-
trained model and the downstream task, it should be more effective for zero-
shot and few-slot filling. To better illustrate the conversion of the slot-
filling input (utterance $u$ and request slots $r$), several examples are
shown in Table 1.

Utterance |  Requested Slots |  Slot Key |  Natural Language Context
---|---|---|---
We will require an outside table to seat 9 people on August 23rd |  None |  date |  We will require an outside table to seat 9 people on August 23rd [EOS] Ok, the date is
Laurice Hoisl |  first_name, last_name |  first_name |  What is the first name, last name? [EOS] Laurice Hoisl [EOS] Ok, the first name is
My party will be 9 people. My name is Nancie Waltemeyer and the time is 7pm |  None |  people |  My party will be 9 people. My name is Nancie Waltemeyer and the time is 7pm [EOS] Ok, the number of people is
Table 1: Examples of slot filling inputs reformulated as natural language
dialog contexts

###  3.2 DialoGPT for Slot Filling

In order to adapt the pre-trained DialoGPT model to the slot filling task, we
augment the architecture and modify the inference algorithm. These adaptations
are motivated by the observation that if the slot value is provided, it will
always be a contiguous span of tokens from the utterance. As such, the
generative model can only produce: (1) ‘not provided‘ if the slot does not
appear in the utterance, (2) the end of sentence token, and (3) tokens from
the input utterance.

A copy-mechanism is incorporated into the DialoGPT architecture to allow the
model to explicitly generate tokens from the input utterance. Given a context
$c=\\{x_{1},x_{2},\dots x_{n}\\}$, through its self-attention layers, the
model will produce a hidden state representation for each token,
$h=\\{h_{1},h_{2},\dots,h_{n}\\}$. A probability distribution over the
vocabulary is then obtained by passing $h_{n}$ through a classification layer:

$P_{vocab}=\textrm{softmax}(Wh_{n}+b)$ (1)

To explicitly generate tokens from the input, $h_{n}$ is used to attend to
$h_{1:n}$ to produce a probability distribution over $x_{1:n}$. The process
for computing the probability for a specific word, $P_{copy}(w)$ is as
follows:

$\alpha=\textrm{softmax}(h_{n}^{T}h_{1:n})$ (2)

$P_{copy}(w)=\sum_{i:x_{i}=w}\alpha_{i}$ (3)

These two probability distributions are combined through a weighted sum. The
weight assigned to each of the distributions is predicted using $h_{n}$:

$p_{copy}=\sigma(W_{copy}h_{n}+b_{copy})$ (4)

The final probability distribution is therefore:

$P_{final}=(1-p_{copy})P_{vocab}+p_{copy}P_{copy}$ (5)

The copy-mechanism requires training, as it introduces new weights
($w_{copy}$, $b_{copy}$) and the off-the-shelf DialoGPT model does not
necessarily produce attention weights, $\alpha$, that can be used to create an
output probability distribution. As such, to attain strong zero-shot
performance we must also modify the inference algorithm to account for the
aforementioned observation. This is done using both constrained decoding and a
post-processing heuristic.

Constrained decoding is a modification of greedy decoding wherein the argmax
sampling is modified to only generate (1) words that appear in the input
utterance, (2) the end of sentence token and (3) the phrase ‘not provided’.

The slot values may consist of terms that the model has not frequently
observed during pre-training (e.g., names, times). As such, because the
DialoGPT model leverages a subword vocabulary, some subword tokens may be
dropped during generation and therefore the slot values may be generated with
typos (e.g., ‘Mocer’ vs ‘Mocher’). A simple post-processing heuristic is
applied to mitigate this problem. If the slot value produced by the model is
not present in the utterance, the Levenshtein distance to every contiguous
span of tokens in the utterance is computed. If the best edit distance is
within a certain threshold ($0.3\times\textrm{len}(y)$), the corresponding
span is returned as the slot value.

Through these modifications, the DialoGPT model is adapted to reflect the
properties of the slot filling task. The copy-mechanism, constrained decoding
and post-processing mechanism serve as an inductive bias to enable the pre-
trained model to be better adapted for the downstream slot filling task.

Fraction | Span-ConveRT | Span-BERT | ConVEx | GenSF
---|---|---|---|---
1 (8198) | 95.8 | 93.1 | 96.0 | 96.1
$\nicefrac{{1}}{{2}}$ (4099) | 94.1 | 91.4 | 94.1 | 94.3
$\nicefrac{{1}}{{4}}$ (2049) | 91.2 | 88.0 | 92.6 | 93.2
$\nicefrac{{1}}{{8}}$ (1024) | 88.5 | 85.3 | 90.6 | 91.8
$\nicefrac{{1}}{{16}}$ (512) | 81.1 | 76.6 | 86.4 | 89.7
$\nicefrac{{1}}{{32}}$ (256) | 63.8 | 53.6 | 81.8 | 82.1
$\nicefrac{{1}}{{64}}$ (128) | 57.6 | 42.2 | 76.0 | 76.1
$\nicefrac{{1}}{{128}}$ (64) | 40.5 | 30.6 | 71.7 | 72.2
Table 2: $F_{1}$ scores across all slots for the evaluation on the
restaurants-8k test data with varying proportions of the training set. Numbers
in brackets denote the training set sizes. The best scores (statistically
significant by t-test to $p<0.05$) are shown in boldface.

##  4 Experiments

Experiments are performed to empirically validate the hypothesis that
simultaneously adapting the downstream task and the pre-trained model results
in stronger alignment and improved performance. We present experiments on two
datasets and assess GenSF in full-data, few-shot and zero-shot settings. An
ablation study is performed to characterize the source of the performance
gains and demonstrate the importance of simultaneous adaptation.

###  4.1 Datasets

Experiments are carried out on restaurants-8k (Coope et al., 2020) and the
dstc8 datasets (Rastogi et al., 2020). restaurants-8k consists of 8,198
utterances from a commercial restaurant booking system and includes 5 slots
(date, time, people, first name, last name). The dstc8 datasets span four
different domains (buses, events, homes, rental cars) for a total of 5,569
utterances with slot annotations extracted by Coope et al. (2020).

In both datasets, the value for a particular slot is always a contiguous span
of the utterance. Some utterances consist of a set of slots requested by the
system prior to the user utterance. This allows an otherwise ambiguous
utterance like ‘four’ to be interpreted as either ‘four people’ or ‘four
o’clock’.

###  4.2 Experimental Setup

We use the pre-processing and evaluation scripts provided by the DialoGLUE
benchmark (Mehri et al., 2020). We follow the setup of Coope et al. (2020) and
Henderson and Vulić (2020), wherein a validation set is not used and the
experiments are therefore performed with fixed hyperparameters. Throughout all
the experiments, the medium version of DialoGPT (Zhang et al., 2020) is used.
We use the AdamW optimizer (Loshchilov and Hutter, 2017) with a learning rate
of 5e-5. On restaurants-8k, the models are trained for 10 epochs in the full-
data setting, 20 epochs in the few-shot settings and 40 epochs in the extreme
few-shot settings ($\nicefrac{{1}}{{32}}$ \- $\nicefrac{{1}}{{128}}$; or less
than $256$ training examples). On the dstc8 datasets, the models are trained
for 20 epochs in the full-data setting and 40 epochs in the few-shot setting.

The models are evaluated on the full test set, regardless of the amount of
training data, using macro-averaged $F_{1}$ score (Coope et al., 2020).

To facilitate reproducibility, the code and the trained models will be
released upon publication.

| Setting | Span-ConveRT | Span-BERT | ConVEx | GenSF
---|---|---|---|---|---
Buses_1 | Full-Data (1133) | 93.5 | 93.3 | 96.0 | 98.1
| Few-Shot (283) | 84.0 | 77.8 | 86.7 | 90.5
Events_1 | Full-Data (1498) | 92.7 | 84.3 | 91.7 | 94.7
| Few-Shot (374) | 82.2 | 78.6 | 87.2 | 91.2
Homes_1 | Full-Data (2064) | 94.8 | 96.3 | 98.3 | 96.9
| Few-Shot (516) | 95.4 | 95.1 | 94.5 | 93.7
RentalCars_1 | Full-Data (874) | 94.0 | 92.8 | 92.0 | 93.5
| Few-Shot (218) | 83.0 | 81.4 | 87.4 | 86.7
Table 3: $F_{1}$ scores across all slots for evaluation on the dstc8 single-
domain datasets in the full-data and few-shot settings. Numbers in brackets
denote training set sizes. The best scores (statistically significant by
t-test, to $p<0.05$) are shown in boldface.

###  4.3 Slot Filling Results

Throughout the experiments we compare to several models from prior work. Span-
ConveRT (Coope et al., 2020) and Span-BERT train a CNN and a CRF on top of
contextual subword embeddings produced by ConveRT (Henderson et al., 2020) and
BERT (Devlin et al., 2019), respectively. ConVEx (Henderson and Vulić, 2020)
devises a pairwise cloze pre-training objective specifically for slot-filling.
This task-specific pre-training objective is an example of significantly
adapting the pre-trained model to the downstream task. In contrast to ConVEx,
GenSF achieves strong alignment between the pre-trained model and the
downstream task by simultaneously adapting both the task and the model. As
such, GenSF does not need a task-specific pre-trained model and is inherently
more scalable. The ConVEx pre-training takes 8 hours to train on 12 GPUs,
while GenSF takes less than four hours to train on a single GTX 1080TI.

As shown in Table 2, GenSF achieves state-of-the-art results across all
experimental settings on the restaurants-8k dataset. In the full-data setting,
GenSF slightly outperforms ConVEx. Though the performance gain is small, this
result signifies that our model can leverage an abundance of data. The value
of strong alignment between the downstream task and the pre-trained model is
better exemplified in the few-shot settings. Especially in the extreme few-
shot settings (i.e., $\nicefrac{{1}}{{32}}$ \- $\nicefrac{{1}}{{128}}$ of the
training set), GenSF strongly outperforms Span-ConveRT and Span-BERT, with
greater than $30$ $F_{1}$ score improvements. The few-shot performance of both
ConVEx and GenSF in these few-shot settings underlies the value of effectively
aligning the pre-trained model and the downstream task. However, GenSF
achieves this alignment by simultaneously incorporating inductive biases about
the model into the task rather than designing a complex pre-training
objective. By incorporating inductive biases into both the task and the model,
the approach outlined in this paper does not require task-specific pre-trained
models and therefore preserves the inherent generality of the transfer
learning paradigm. Furthermore, GenSF attains moderate improvements over
ConVEx, especially in the few-shot settings, with a $3$ $F_{1}$ score
improvement in the $\nicefrac{{1}}{{16}}$th setting.

The results on the dstc8 single-domain datasets is shown in Table 3. Here, we
evaluate on both full-data and few-shot (25% of the training data) settings.
On average, GenSF achieves strong performance improvements over prior work. In
the full-data settings the best performance is observed on the buses and
events domains, where GenSF achieves a $2.1$ and $3.0$ $F_{1}$ score
improvement over ConVEx, respectively. In the few-shot settings, GenSF
achieves a $4.0$ $F_{1}$ score improvement over ConVEx on these domains and a
$6.5$ and $9.0$ point improvement over Span-ConveRT. These strong
improvements, over both Span-ConveRT and ConVEx, highlight the value of strong
alignment between the pre-trained model and the downstream task, particularly
in the few-shot experiments.

GenSF moderately underperforms on the homes and rental cars domains. On the
homes domain, GenSF outperforms Span-ConveRT and Span-BERT but scores $1.4$
points below ConVEx. Similarly, on the rental cars domain, GenSF outperforms
ConVEx and Span-BERT, but is $0.5$ points below Span-ConveRT. Though GenSF is
still competitive in these domains, these results nonetheless highlight a
weakness of our model. Our use of a generative pre-trained dialog model,
specifically DialoGPT (Zhang et al., 2020), was motivated by the hypothesis
that such models can implicitly learn to identify certain slots through
response generation pre-training. This hypothesis is empirically validated
through improved performance on restaurants-8k and the buses/events domains of
dstc8. GenSF relies on the pre-trained model having an implicit understanding
of the slots. This implicit understanding results in strong performance on
slots like ‘time’ or ‘first name’, since such terms are likely to have been
observed during pre-training. However, this is not the case for all slots and
GenSF can underperform on slots that are ambiguous, ill-defined or are
unlikely to have been observed during open-domain dialog pre-training. The
homes domain consists of the slot, ‘area’, which has several definitions and
is therefore challenging for the pre-trained model to understand and detect.
The rental cars domain contains the slots ‘pickup date’ and ‘dropoff date’.
While the DialoGPT model has learned to detect a ‘date’, the distinction
between these two slots is more nuanced and therefore may cause some amount of
confusion. As such, while GenSF is competitive in these domains and is only
outperformed by one of the three models, these domains demonstrate that there
are limitations at present to leveraging a generative pre-trained model.
However, it is possible that by further adapting the downstream task to the
pre-trained model, for example by renaming these slots (e.g., ‘area’ may be
renamed to ‘city’), the performance drops may be mitigated.

Overall, GenSF achieves impressive performance gains in both full-data and
few-shot settings, underlying the value of achieving strong alignment between
the pre-trained model and the downstream task. Furthermore, GenSF achieves
this alignment by simultaneously adapting both the task and the model and
without sacrificing the inherent scalability of the transfer learning paradigm
or necessitating task-specific pre-training. In the restaurants-8k and the
single-domain dstc8 datasets, GenSF achieves state-of-the-art results and
outperforms prior work. In few-shot settings, we achieve a $30$ $F_{1}$ score
improvement over Span-BERT and Span-ConveRT. On average, GenSF moderately
outperforms ConVEx, with $>2.0$ $F_{1}$ score improvements in the few-shot
settings on restaurant-8k, and both the full data and few-shot settings on two
of the dstc8 datasets. These experiments empirically validate (1) the
importance of aligning the pre-trained model and the downstream task by
simultaneously incorporating inductive biases into both the task and the model
and (2) that through response generation pre-training, dialog models have
implicitly learned to detect certain slots, which can be leveraged by
effectively adapting the downstream task.

###  4.4 Zero-shot Slot Filling

Slot | Metric | Coach+TR | ConVEx | GenSF
---|---|---|---|---
| P | 1.7 | 2.3 | 13.7
First Name | R | 4.1 | 20.1 | 36.1
| $F_{1}$ | 2.5 | 4.1 | 19.8
| P | 0 | 1.9 | 10.6
Last Name | R | 0 | 16.2 | 19.7
| $F_{1}$ | 0 | 3.4 | 13.8
| P | 10.2 | 2.2 | 10.7
Date | R | 34.8 | 10.1 | 15.3
| $F_{1}$ | 15.7 | 3.6 | 12.6
| P | 47.4 | 5.6 | 27.5
Time | R | 27.9 | 23.6 | 46.9
| $F_{1}$ | 35.1 | 9.1 | 34.7
| P | 0 | 3.8 | 14.5
People | R | 0 | 13.9 | 18.9
| $F_{1}$ | 0 | 6.0 | 16.4
Average | $F_{1}$ | 10.7 | 5.2 | 19.5
Table 4: Zero-shot slot filling results on restaurants-8k. All models are evaluated on the test set without any training on the dataset. Model | Full-Data | Few-Shot ($\nicefrac{{1}}{{16}}$) | Zero-Shot
---|---|---|---
GenSF | 96.1 | 89.7 | 19.5
Removing Model Adaptation
– Copy-mechanism | 95.6 | 87.8 | 19.5
– Constrained Decoding | 95.4 | 89.5 | 0.5
– Post-processing | 96.1 | 89.7 | 18.1
– All model adaptation | 95.4 | 87.8 | 0.5
Removing Task Adaptation
– Natural Language Slot Names | 95.3 | 86.6 | 12.2
– Natural Language Templates | 94.8 | 88.5 | 0.0
– All Natural Language | 95.5 | 88.9 | 0.0
Removing All Adaptation
– All Adaptation | 95.8 | 89.2 | 0.0
Table 5: Ablation experiments. We remove (1) adaptations to the model, (2)
adaptations to the downstream task and (3) all adaptations proposed in this
paper. The experiments are carried out on the full-data, few-shot
($\nicefrac{{1}}{{16}}$th of the training set) and zero-shot settings of
restaurants-8k.

For zero-shot slot filling, we must have strong alignment between the pre-
trained model and the downstream task. Since the model is not fine-tuned on
the task, it is necessary to effectively align the formulation of the
downstream task to the capabilities of the model. As such, zero-shot
experiments validate our proposed reformulation of slot filling as natural
language response generation.

For these experiments, we compare to the published results of ConVEx
(Henderson and Vulić, 2020). Furthermore, we run a Coach+TR model (Liu et al.,
2020) on the restaurant-8k dataset. Note that while ConVEx and GenSF have only
been trained on open-domain dialog, Coach+TR trains on adjacent task-oriented
domains (i.e., SNIPS), meaning that the zero-shot performance is higher on
slots that are domain agnostic.

The experiments used the restaurants-8k dataset with the GenSF model. The
copy-mechanism is removed from the model, as it adds additional weights to the
model and therefore requires training. However, the constrained decoding and
the post-processing heuristic of GenSF, allow us to enforce that the slot
values will always be a contiguous span from the input utterance. Table 4
demonstrates that GenSF significantly outperforms prior work on zero-shot slot
filling with a 14 $\mathbf{F_{1}}$ score improvement over ConVEx and a 9
$\mathbf{F_{1}}$ score improvement over Coach+TR. These results further
validate the hypothesis that pre-trained dialog models have implicitly learned
to detect slots and that this ability can be leveraged through the proposed
task reformulation.

Most noteworthy is the performance on the ‘first name’ and ‘last name’ slots.
This suggests that, to some degree, DialoGPT (Zhang et al., 2020) can
disambiguate between a first name and a last name when provided simultaneously
(e.g., ‘my name is Lakesha Mocher’). It should be noted that the macro-
averaged $F_{1}$ score used to evaluate the models considers a slot value to
be incorrect unless it exactly predicts the ground-truth slot value. In many
cases, the GenSF model produces appropriate slot values that differ from the
ground-truth, e.g., ‘wednesday’ instead of ‘next wednesday’. It is possible
that by incorporating additional inductive biases about the specific
formulation of the slot values (e.g., slots should have maximal information)
into the inference algorithm, the zero-shot performance can be further
increased.

GenSF is shown to strongly outperform prior work on zero-shot slot filling.
This impressive performance validates the proposed approach of simultaneously
adapting both the downstream task and the pre-trained model. Furthermore,
zero-shot performance also confirms the hypothesis that pre-trained response
generation models have implicitly learned to understand and detect slots,
thereby highlighting the potential of leveraging generative pre-trained models
for language understanding tasks. Future work should explore mechanisms for
reformulating other downstream tasks (e.g., intent prediction, dialog state
tracking) in order to leverage generative pre-trained models. Furthermore, it
is possible that these zero-shot results could be further improved through
two-stage pre-training (e.g., further pre-train with the ‘pairwise cloze’
task).

###  4.5 Ablation

GenSF has been shown to outperform prior work in full-data, few-shot and zero-
shot settings. To determine the source of the improvements, we perform an
ablation study. The ablation experiments remove the adaptations used in GenSF
and evaluate on restaurants-8k across full-data, few-shot
($\nicefrac{{1}}{{16}}$ of the training set) and zero-shot settings. Removing
all the ablation, is equivalent to training a DialoGPT model from scratch on
the task, similar to the approach proposed by Madotto (2020).

As shown in Table 5, the various adaptations are vital to the strong
performance of GenSF. Of the model adaptations, only the copy-mechanism is
necessary in the full-data setting, since the model effectively learns to copy
tokens from the input utterance and therefore does not need constrained
decoding and post-processing. However, constrained decoding is necessary for
the zero-shot settings, as the zero-shot model does not leverage a copy-
mechanism. Task adaptation, especially the use of natural language templates,
is shown to be important across all of the experimental settings. This
highlights the importance of formulating the downstream task in a manner that
can effectively leverage the capabilities of the pre-trained models.

The results of the ablation study further validate this paper’s primary
hypothesis. Pre-trained models work better for downstream tasks, when the task
and the model are effectively aligned. As shown in the results of the ablation
study, removing this adaptation results in a performance decrease.

##  5 Conclusion

This paper simultaneously adapts both the task and the pre-trained model in
order to achieve strong alignment between a generative pre-trained dialog
model and the downstream slot filling task. The resulting GenSF model achieves
state-of-the-art results on two slot filling datasets, with particularly
strong gains in few-shot and zero-shot settings. The empirical results
underlie the importance of incorporating inductive bias into both the task and
the pre-trained model. While this paper demonstrates the value of simultaneous
adaptation for the task of slot filling, a similar paradigm could potentially
be extended to alternate tasks. Future work should (1) explore improved
mechanism for achieving stronger alignment between the task and the model, (2)
extend the simultaneous adaptation strategy to other problems and (3) explore
the use of pre-trained generative models for language understanding tasks.

## References

  * Bapna et al. (2017) Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017.  Towards zero-shot frame semantic parsing for domain scaling.  _arXiv preprint arXiv:1707.02363_.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_.
  * Coope et al. (2020) Samuel Coope, Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson. 2020.  Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 107–121, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Glass et al. (2019) Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, GP Bhargav, Dinesh Garg, and Avirup Sil. 2019.  Span selection pre-training for question answering.  _arXiv preprint arXiv:1909.04120_.
  * Henderson et al. (2019) Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2019.  Convert: Efficient and accurate conversational representations from transformers.  _arXiv preprint arXiv:1911.03688_.
  * Henderson et al. (2020) Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2020.  ConveRT: Efficient and accurate conversational representations from transformers.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2161–2174, Online. Association for Computational Linguistics.
  * Henderson and Vulić (2020) Matthew Henderson and Ivan Vulić. 2020.  Convex: Data-efficient and few-shot slot labeling.  _arXiv preprint arXiv:2010.11791_.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  _arXiv preprint arXiv:2005.00796_.
  * Kurata et al. (2016) Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016.  Leveraging sentence-level information with encoder lstm for semantic slot filling.  _arXiv preprint arXiv:1601.01530_.
  * Lewis et al. (2020) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020.  Pre-training via paraphrasing.  _arXiv preprint arXiv:2006.15020_.
  * Liu et al. (2020) Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020.  Coach: A coarse-to-fine approach for cross-domain slot filling.  _arXiv preprint arXiv:2004.11727_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Madotto (2020) Andrea Madotto. 2020.  Language models as few-shot learner for task-oriented dialogue systems.  _arXiv preprint arXiv:2008.06239_.
  * Mehri et al. (2020) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _arXiv preprint arXiv:2009.13570_.
  * Peng et al. (2020) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020.  Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model.  _arXiv preprint arXiv:2005.05298_.
  * Peng et al. (2015) Baolin Peng, Kaisheng Yao, Li Jing, and Kam-Fai Wong. 2015.  Recurrent neural networks with external memory for spoken language understanding.  In _Natural Language Processing and Chinese Computing_ , pages 25–35. Springer.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Schema-guided dialogue state tracking task at dstc8.  _arXiv preprint arXiv:2002.01359_.
  * Shah et al. (2019) Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur. 2019.  Robust zero-shot cross-domain slot filling with example values.  _arXiv preprint arXiv:1906.06870_.
  * Tur and De Mori (2011) Gokhan Tur and Renato De Mori. 2011.  _Spoken language understanding: Systems for extracting semantic information from speech_.  John Wiley & Sons.
  * Vu (2016) Ngoc Thang Vu. 2016.  Sequential convolutional neural networks for slot filling in spoken language understanding.  _arXiv preprint arXiv:1606.07783_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Yao et al. (2014) Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014\.  Spoken language understanding using long short-term memory neural networks.  In _2014 IEEE Spoken Language Technology Workshop (SLT)_ , pages 189–194. IEEE.
  * Young (2002) Steve Young. 2002.  Talking to machines (statistically speaking).  In _Seventh International Conference on Spoken Language Processing_.
  * Young (2010) Steve Young. 2010.  Still talking to machines (cognitively speaking).  In _Eleventh Annual Conference of the International Speech Communication Association_.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278, Online. Association for Computational Linguistics.
  * Zhao and Feng (2018) Lin Zhao and Zhe Feng. 2018.  Improving slot filling in spoken language understanding with joint pointer and attention.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 426–431.