# Exploring Small Language Models with Prompt-Learning Paradigm for Efficient
Domain-Specific Text Classification

Hengyu Luo1,2, Peng Liu1, Stefan Esping1
1 Ingka Group, IKEA, 2 Uppsala University Corresponding author. Email:
<EMAIL_ADDRESS>
###### Abstract

Domain-specific text classification faces the challenge of scarce labeled data
due to the high cost of manual labeling. Prompt-learning, known for its
efficiency in few-shot scenarios, is proposed as an alternative to traditional
fine-tuning methods. And besides, although large language models (LLMs) have
gained prominence, small language models (SLMs, with under 1B parameters)
offer significant customizability, adaptability, and cost-effectiveness for
domain-specific tasks, given industry constraints. In this study, we
investigate the potential of SLMs combined with prompt-learning paradigm for
domain-specific text classification, specifically within customer-agent
interactions in retail. Our evaluations show that, in few-shot settings when
prompt-based model fine-tuning is possible, T5-base, a typical SLM with 220M
parameters, achieve approximately 75% accuracy with limited labeled data (up
to 15% of full data), which shows great potentials of SLMs with prompt-
learning. Based on this, We further validate the effectiveness of active few-
shot sampling and the ensemble strategy in the prompt-learning pipeline that
contribute to a remarkable performance gain. Besides, in zero-shot settings
with a fixed model, we underscore a pivotal observation that, although the
GPT-3.5-turbo equipped with around 154B parameters garners an accuracy of
55.16%, the power of well designed prompts becomes evident when the
FLAN-T5-large, a model with a mere 0.5% of GPT-3.5-turbo’s parameters,
achieves an accuracy exceeding 31% with the optimized prompt, a leap from its
sub-18% performance with an unoptimized one. Our findings underscore the
promise of prompt-learning in classification tasks with SLMs, emphasizing the
benefits of active few-shot sampling, and ensemble strategies in few-shot
settings, and the importance of prompt engineering in zero-shot settings.

##  1 Introduction

In today’s digital age, the explosion of textual data across platforms such as
social media, online forums, and customer interactions is unmistakable.
Interpreting this enormous amount of data poses challenges for businesses and
researchers alike. At the heart of this endeavor is text classification, a
pivotal tool for gleaning insights and facilitating informed decision-making.

Industry domain-specific text classification is of paramount importance. In
domains such as customer support, discerning customer intentions accurately
can expedite response times and route inquiries more effectively. The
acquisition of labeled data, essential for model training, remains a
significant hurdle. This data acquisition is labor-intensive and often
requires domain-specific expertise. Even with advances in text classification,
tailoring these models for niche tasks often demands substantial resources and
data, prompting the search for methods with diminished reliance on labeled
datasets.

The recent advent of pre-trained language models, such as BERT [Devlin et al.,
2018], GPT [Radford et al., 2018], and RoBERTa [Liu et al., 2019], has brought
about transformative changes in NLP. These models, pre-trained on vast text
repositories, have captured intricate language patterns and representations,
enabling them to be fine-tuned for a variety of downstream tasks. However,
conventional fine-tuning approaches require a significant amount of labeled
data, which can be a constraint, especially in domain-specific applications.

Here, prompt-learning emerges as a promising alternative. This technique
capitalizes on the rich representations within pre-trained models, using
specific prompts to guide them towards desired outputs, effectively
transforming them into classifiers. This method can be particularly
advantageous in contexts with limited labeled data, as it reduces the
dependence on extensive datasets. In this landscape, there have been some
advancements in prompt-learning, both in few-shot and zero-shot settings. In
the realm of few-shot settings where model fine-tuning is allowed, recent
models such as PET-TC [Schick and Schütze, 2021], PET-Gen [Schick and Schütze,
2020], and LM-BFF [Gao et al., 2021] have demonstrated the potential of
prompt-learning. For instance, LM-BFF has introduced innovative techniques
like automated template generation and label word selection, significantly
reducing the need for manual prompt engineering [Gao et al., 2021]. These
models hold promise for domain-specific text classification, a focus of our
research, where acquiring extensive labeled data can be challenging.
Conversely, in zero-shot settings, models like LAMA [Petroni et al., 2019] and
GPT-3 [Brown et al., 2020] have explored tuning-free prompting. This approach
relies on the careful engineering of prompts for achieving desired outputs.
Our work builds upon these foundations to explore the effectiveness of prompt-
learning, especially when coupled with smaller language models, within the
specific context of retail interactions.

In addition to advancements of the prompt-learning paradigm, while large
language models (LLMs) continue to dominate discussions, there’s an emerging
appreciation for smaller language models (SLMs, with under 1 billion
parameters). These SLMs, while potentially less powerful, can be specialized
for specific tasks, making them more efficient and economical.

The convergence of these research leads us to the primary objective of this
paper: to evaluate the effectiveness of text classification based on prompt-
learning methodology, especially when paired with SLMs. Here we do a case
study of the text classification task based on the domain of retail
interactions. Given this backdrop, the core contributions and findings of this
paper, highlighted upfront for clarity, are:

  * •

In few-shot settings, we showcase the potential of SLMs combined with the
prompt-learning approach for domain-specific text classification, specifically
in retail interactions, achieving around 75% accuracy with 15% of full labeled
data using T5-base, which shows strong competitiveness to the traditional
fine-tuning approach.

  * •

In few-shot settings, we validate the active few-shot sampling and ensemble
strategies in the prompt-learning process, leading to a notable performance
boost.

  * •

In zero-shot settings, we underscore that, although the giant GPT-3.5-turbo
with approximately 154B parameters achieves an impressive 55.16% accuracy, our
optimized prompts enable significantly smaller models like FLAN-T5-large to
achieve over 30% accuracy, reaching over half the performance of GPT-3.5-turbo
with a fraction of the parameters.

With these contributions in perspective, the remainder of this paper is
structured as follows: Section 2 offers a brief introduction to the dataset we
utilize. Section 3 elucidates our prompt-learning pipeline, detailing
methodologies and experimental setups for both few-shot and zero-shot
scenarios. Section 4 presents our findings and an in-depth analysis. After
introducing related work in Section 5, we conclude with insights and potential
directions for future research in Section 6. Through our experiments, we aim
to uncover an optimized strategy for deploying prompt-learning with SLMs,
especially in contexts with limited labeled data.

##  2 Dataset

The dataset, sourced from IKEA’s customer support service in the retail
domain, captures conversations between customerws and agents over designated
months. All data within the set is text-based and has been cleaned to exclude
any personally identifiable information (PII), safeguarding the privacy of
both customers and agents. This collection encompasses 7,477 interactions
between a consumer and an agent represented by text strings.

Following the acquisition, the data underwent manual categorization. It was
annotated with 13 distinct labels reflecting various customer intents, plus an
"other" label for instances not aligning with the predefined 13 categories.
These intentions span topics such as order processing, product inquiry,
billing issues, and technical support. A demonstration sample is shown in
Table 1.

Conversation | Label
---|---
| …
---
Customer: I’d like to know the detailed instruction to assemble this desk.
Agent: Sure, let me check our website. Hold on, please.
…
“Help integrating the product”

Table 1: An example of the labeled conversation between customers and agents
in the dataset

The dataset exhibits an imbalance: certain intentions are notably more
frequent than others. The distribution of labels can be seen in table 2.

No. | Name of Label | Count of Samples
---|---|---
0 | Product / Service Availability | 951
1 | General | 113
2 | General after Purchase | 108
3 | Help Integrating the Product | 306
4 | Initiate After-sales Service | 907
5 | Issue Handling | 655
6 | Order Creation | 1259
7 | Order Fulfillment Issues | 997
8 | Order Processing | 1069
9 | Other | 102
10 | Planning & Advice | 192
11 | Prepare for Exchange & Returns | 531
12 | Product / Service Information | 283
13 | Service Fulfillment | 29
total |  | 7477
Table 2: Label distribution in the dataset

To evaluate the efficacy of prompt-learning and zero-shot text classification
techniques, the dataset was segmented into three subsets: a combined training
and development set (50% of the total, 3738 entries), a validation set (25%,
1870 entries), and a test set reserved for the final stage of the experiments
(25%, 1869 entries). Utilizing stratified sampling, we ensured that the
intention distribution in each subset mirrored that of the entire dataset. The
training and development set catered to model training and selection, the
validation set gauged model performance, while the test set, devoid of
external knowledge gained from prior experiments, served the final assessment.
All the experimental results will be based on the final test set, offering a
objective evaluation of the models’ generalization capabilities.

##  3 Methodology and Experimental Setup

###  3.1 Prompt-Learning Pipeline

The prompt-learning pipeline, mathematically described by Liu et al. [2023],
is a systematic process illustrated in Fig. 1.

Figure 1: A demonstration of prompt-learning pipeline

The basic structure of this pipeline involves three essential steps. First,
the input text (usually preprocessed for improvement of data quality) is
transformed into a prompt using a prompting function, often applying a text
template with designated slots. In our classification task, a sample input “…
Customer: I’d like to know the detailed instruction to assemble this desk.
Agent: Sure, let me check our website. …” is processed with a template "What
is the topic of this conversation? <MASK>" to produce a modified prompt.
Second, an attempt is made to identify the <MASK> token that the language
model (LM) scores the highest for, using a predefined set representing
permissible values. Using a pre-trained LM, probabilities of the filled
prompts are calculated, and the best answer is deduced through $argmax$
search. Lastly, the highest-scoring answer <MASK> is mapped to its
corresponding output label, which probably requires a specialized mapping
function (verbalizer) for tasks if multiple answers might map to the same
output. This is particularly crucial when representing outputs like labels in
classification tasks.

###  3.2 Investigation in Few-shot Prompt-based Model Fine-tuning Settings

####  3.2.1 Active Few-shot Sampling

In the context of prompt-based text classification, few-shot sampling is an
essential component of the pipeline. Rather than resorting to random sampling,
our goal is to probe the merits of selecting the most representative samples
for each label. To achieve this, BERT [Devlin et al., 2018] is employed to
embed text data from the training and development set into 512-dimensional
vectors. Subsequently, we discern samples that lie at the core of each class’s
distribution in the vector space. A simple demonstration is shown in Fig 2.
These centric samples which are likely to exemplify their respective classes,
are speculated to raise the performance of prompt-based methods. Experiments
comparing these centric samples with randomly chosen counterparts will
illuminate the differential effects on the approach’s performance.

Figure 2: A comparison between a) random sampling vs. b) active sampling.
Here, embedded data samples are converted into 2D projections for a
straightforward visual demonstration.

####  3.2.2 Prompt and Verbalizer Ensemble

We designed various prompt templates and verbalizers as a critical step in our
prompt-learning pipeline. For a detailed overview of the prompt templates and
the verbalizers we experimented with, see Appendix A.

Figure 3: A demonstration of ensemble of prompt templates Figure 4: A
demonstration of ensemble of verbalizers

Equipped with four templates and an equal number of verbalizers, we drew
inspiration from PET [Schick and Schütze, 2021] to adopt the concept of prompt
ensemble, verbalizer ensemble, and prompt and verbalizer ensemble. To
elaborate on our approach, for example, in prompt ensemble, we first assemble
all the templates (1-4) with a shared verbalizer, resulting in the creation of
four distinct models using different templates but the same verbalizer. We
then combine these four models by summing up the softmaxed logits and base our
decision on the aggregated predicted probabilities. A simple demonstration is
shown below in Fig. 3. We follow a similar principle when assemble all the
verbalizers together, demonstrated in Fig. 4. Moreover, for prompt and
verbalizer ensemble, we assemble a total of 16 models by combining the four
templates and four verbalizers, allowing us to make a collective decision
based on the combined outputs of these models.

####  3.2.3 Baseline Settings

We establish a baseline for convenience in the experimental settings and
proceed to conduct experiments by modifying the components within the baseline
pipeline. The baseline settings is shown in Table 3.

Component | Choice
---|---
Language model | T5-base111https://huggingface.co/t5-base (Seq2seq LM, 220M Parameters)
Few-shot sampling strategy | 5% data with stratified sampling, randomly per label
Choice of template | Template 1
Choice of verbalizer | Verbalizer 1
Table 3: Baseline of the investigation in few-shot prompt-based model fine-
tuning settings

###  3.3 Investigation in Zero-shot Settings

####  3.3.1 Choice of Language Model

In the exploration in zero-shot settings, where no training data is involved,
we experimented with different model architectures (masked language models,
language models, and seq2seq language models) to evaluate their impact on the
classification performance, as is shown in the table 4.

Language Model | Architecture | # of Parameters
---|---|---
Roberta-large222https://huggingface.co/roberta-large | Masked LM | 354M
GPT-2-large333https://huggingface.co/gpt2-large | Autoregressive LM | 774M
GPT3.5-turbo444https://openai.com/blog/chatgpt, with API version ’2023-03-15-preview’ | Autoregressive LM |  $\sim$ 154B
T5-large555https://huggingface.co/t5-large | Seq2seq LM | 770M
FLAN-T5-large666https://huggingface.co/google/flan-t5-large | Seq2seq LM | 770M
Table 4: Different language models selected in zero-shot settings

####  3.3.2 Detailed Prompt Template Design

In this part, we will explore the effectiveness of prompt templates with
detailed instructions, which are designed to introduce the task and provide
descriptions of the different labels to help the model better understand the
task, as is shown in template 5 in Table 5.

Template No. | Prompt Template
---|---
Template 1 | “{conversation} Classify this conversation : {mask} ”
Template 5 |  | “{conversation}
---
Given this conversation, we have 14 classes:
General: General information and issues customer has before buying at IKEA;
Product / Service Information: Information about products and services;
…
Please classify this conversation into one class out of these 14 classes:
{mask}”
Table 5: Different prompt templates tried in zero-shot settings

####  3.3.3 Baseline Settings

We establish a baseline for convenience in the experimental settings and
proceed to conduct experiments by modifying the components within the baseline
pipeline. The baseline settings is shown in Table 6.

Component | Choice
---|---
Choice of verbalizer777Except experiments with GPT-3.5-turbo, as one of base models of ChatGPT. Since the model is packed in the OpenAI ChatGPT API and mapping the label with the verbalizer is not feasible, we add the prompt, “Return the index of the label, please.” and parse the response afterwards, instead of using a verbalizer. | Verbalizer 1
Table 6: Baseline of the investigation in zero-shot settings

###  3.4 Evaluation Metrics

We evaluate the effectiveness of the prompt-learning pipeline for text
classification using accuracy and macro F1-score. Accuracy provides a direct
measure of model performance by comparing the number of correct predictions
with the total number of samples. On the other hand, macro F1-score takes into
account the performance of the model for each class individually, making it a
reliable metric especially for imbalanced datasets. Detailed definitions and
formulas for these metrics can be found in the Appendix B.

###  3.5 OpenPrompt: An Open-Source Framework for Prompt-Learning

For our research, we employed
OpenPrompt888https://github.com/thunlp/OpenPrompt[Ding et al., 2021], a
dedicated open-source platform tailored for prompt-learning studies.
OpenPrompt offers an adaptable environment for the configuration and execution
of diverse prompt-learning components, ranging from prompt templates and
verbalizers to pre-trained language models. Significantly, it ensures smooth
integration with the Hugging Face Model Hub, streamlining the use of a variety
of pre-trained language models.

Throughout this paper, OpenPrompt served as our primary tool for organizing
and conducting prompt-learning experiments. The platform’s capabilities proved
invaluable in efficiently crafting and evaluating numerous prompt templates
and verbalizers. Furthermore, OpenPrompt’s management features were crucial in
comparing and tracking our experimental outcomes, assisting us in pinpointing
the optimal prompt-learning approaches for domain-specific text
classification.

##  4 Experimental Results and Analysis

###  4.1 Investigation in Few-shot Prompt-based Model Fine-tuning Settings

####  4.1.1 General Comparison with Traditional Fine-tuning

We conduct experiments on the performance of few-shot prompt-based model fine-
tuning compared to traditional fine-tuning approaches. The results are
presented in Table 7.

Training Strategy | Accuracy | Macro F1-score
---|---|---
100% Data Traditional Fine-tuning | 80.63 | 72.84
5% Data Traditional Fine-tuning | 32.64 | 16.27
5% Data Prompt-based Fine-tuning (Baseline) | 59.02 | 45.49
Table 7: Comparison of results between fine-tuning with or without prompt

The observations from Table 7 demonstrate that the few-shot prompt-based model
fine-tuning parallels the performance of traditional fine-tuning approaches
even when using the entire training and development dataset999Here, 80% data
for training, 20% data for development. This equivalence is particularly
pronounced when working with limited labeled data. For instance, using only 5%
of labeled data, traditional fine-tuning witnesses a dramatic decline in
accuracy, plummeting from 80.63% to 32.64%. In contrast, prompt-based fine-
tuning retains higher accuracy and macro F1-score with the same data fraction.

Our extended experiments, detailed in Table 8, indicate that increasing the
proportion of labeled data sampled enhances the performance of the few-shot
prompt-based fine-tuning. Specifically, accuracy surges to 73.35% and macro
F1-score reaches 62.80% as the sampled data grows from 1% to 15%.

Proportion of Data Sampled | Accuracy | Macro F1-score
---|---|---
1% | 37.29 | 29.99
3% | 53.07 | 40.41
5% (Baseline) | 59.02 | 45.49
10% | 67.84 | 55.80
15% | 73.35 | 62.80
Table 8: Results of few-shot prompt-based model fine-tuning with different
proportion of data sampled

These results highlight the potential of the prompt-learning pipeline with
SLMs for sparsely labeled data scenarios. The probable reason might be the
susceptibility of these models to overfitting when fine-tuned on smaller
datasets.

Traditional fine-tuning methods often require substantial amounts of data to
adapt pretrained models to specific downstream tasks. This is because there’s
a significant "gap" between the objectives of pre-training and the specific
tasks these models are fine-tuned for. This gap necessitates more data for
fine-tuning, making models susceptible to overfitting, especially in scenarios
with limited samples. On the other hand, prompt-learning seeks to bridge this
gap. Instead of heavily adapting the model’s parameters, prompt-learning
utilizes the vast knowledge already embedded in pretrained models,
transforming downstream tasks to fit the pre-training objectives. By specific
using prompts, this method translates tasks into a natural language format
that the model is familiar with, effectively mining the pretrained model’s
potential.

Furthermore, while traditional fine-tuning often introduces new parameters for
specific tasks (like adding a classification layer), prompt-learning maintains
the same objectives as the pre-training phase and doesn’t require additional
parameters. This approach reduces the risk of overfitting, especially in few-
shot scenarios, making it a more efficient strategy when data is scarce.

Based on the promising results shown above, by modifying various components
within this pipeline, we expect that optimal utilization of limited labeled
data is achievable, leading to augmented outcomes.

####  4.1.2 Active Few-shot Sampling

We conducted the experiments by varying the few-shot sampling strategy from
the baseline, to do random sampling or active sampling from the training and
development set, and the experimental results are presented below in Table 9.

Few-shot Sampling Strategy | Accuracy | Macro F1-score
---|---|---
3% stratified sampling, randomly per label | 53.07 | 40.41
3% stratified sampling, actively per label | 60.03 | 49.75
5% stratified sampling, randomly per label (Baseline) | 59.02 | 45.49
5% stratified sampling, actively per label | 62.01 | 52.77
Table 9: Results of few-shot prompt-based fine-tuning with active or random
few-shot sampling strategies

From Table 9, it’s evident that active sampling notably boosts accuracy and
macro F1-score outperforming random sampling (7% performance gain with 3% data
sampled, and 3% performance gain with 5% data sampled). This indicates that
judicious selection of dataset samples significantly uplifts performance over
random few-shot sampling.

Specifically, active learning, by design, prioritizes the most "informative"
examples in the training set. The idea is to maximize the information gain
from each labeled instance, thus more efficiently updating the model’s
parameters. This strategy can be particularly impactful in scenarios with
limited labeled data.

####  4.1.3 Prompt and Verbalizer Ensemble

We conducted the experiments by varying the prompts and verbalizers from the
baseline, and the experimental results are presented below in Table 10.

Template No. Acc. / Macro F1-score Verbalizer No. | 1 | 2 | 3 | 4 | .*
---|---|---|---|---|---
1 | 59.02 / 45.49 | 60.56 / 53.66 | 61.95 / 50.89 | 60.67 / 48.20 | 64.74 / 54.63
2 | 62.49 / 48.42 | 60.88 / 53.42 | 65.91 / 52.69 | 62.49 / 55.14 | 68.96 / 58.40
3 | 60.62 / 45.58 | 62.38 / 55.89 | 64.04 / 49.19 | 60.78 / 52.51 | 67.04 / 55.23
4 | 62.17 / 47.76 | 60.19 / 46.01 | 65.60 / 50.11 | 59.22 / 43.76 | 67.09 / 52.80
.* | 64.63 / 49.84 | 63.77 / 55.98 | 67.79 / 53.60 | 63.02 / 52.73 | 68.70 / 56.76

Table 10: Results of few-shot prompt-based model fine-tuning with different
templates and verbalizers (“.*” means all the templates / verbalizers are
assembled), baseline with template 1 and verbalizer 1

From the experimental results shown in Table 10, we can see that the choice of
prompt template and verbalizer is important in prompt-based text
classification. Different templates or verbalizers may violate the
effectiveness of the model and it shows a lack of robustness of prompt-
learning process. A potential reason behind the lack of robustness in
individual prompts and verbalizers might be their specificity. Some prompts
may align better with certain types of input data than others, resulting in
performance discrepancies. This variance underscores the importance of
diversifying the prompts and verbalizers – a kind of "ensembling" that can
provide a more holistic understanding of the data.

Accordingly, we can observe from the table 10 that by assembling the templates
and/or the verbalizers, the performance of the assembled model improves
greatly and surpasses that of any individual model alone. This highlights the
effectiveness of combining multiple templates and verbalizers to enhance the
overall performance and achieve better classification results in comparison to
using a single template or verbalizer in isolation.

###  4.2 Investigation in Zero-shot Settings

We conducted the experiments by varying lanaguage models with a different
prompt templates (template 1 and 5), and the experimental results are
presented below in Table 11. Reviewing Table 11, normal prompts, like prompt
1, generally yield relatively low performance, consistently below 20% barring
the advanced GPT-3.5-turbo model. This limits their applicability in practical
contexts.

Name of Model (# of Params) Template No. Acc. / Macro F1-score | 1 | 5
---|---|---
RoBerta-large (354M) | 4.07 / 1.81 | 11.66 / 7.91
GPT-2-large (774M) | 3.96 / 1.66 | 7.01 / 3.74
T5-large (770M) | 3.31 / 1.00 | 19.31 / 8.58
FLAN-T5-large (770M) | 17.92 / 9.97 | 31.35 / 20.57
GPT-3.5-turbo ($\sim$154B) | 31.57 / 19.57 | 55.16 / 43.38
Table 11: Results in zero-shot Settings

This led us to explore detailed prompts as a potential enhancement. The
outcomes, presented in Table 11, show significant performance elevations with
detailed templates. For instance, the T5-large model’s zero-shot accuracy
leaped from 3.31% to 19.31%, and the FLAN-T5-large model’s zero-shot Macro F1
score rose from 17.92% to 31.35%. Moreover, the GPT-3.5-turbo model’s zero-
shot accuracy surged past 50%. This indicates that explicit, task-centric
prompts can amplify performance in zero-shot scenarios. Moreover, even though
GPT-3.5-turbo demonstrates state-of-the-art performance, Flan-T5-large’s
performance has the highest percentage increase, which nearly doubled from
31.57% to 55.16% in terms of accuracy, indicating strong potential of SLMs in
zero-shot settings.

It can be seen from the results that, While template 1 might be more general,
leading to lower performance, template 5, being more detailed, gives the model
more context to work with. This demonstrates the importance of prompt design
in zero-shot learning scenarios, when training data, or even demonstration
data to put into prompts, is not available.

Besides, the enlightenment behind the impressive performance of SLMs, in
comparison to its giant counterpart GPT-3.5-turbo model, warrant a deeper
dive. Such an exploration might reveal that smart tricks of prompt engineering
can further make the performance of SLM excel, while being remarkably more
resource-efficient.

##  5 Conclusion and Future Work

In this paper, we investigated the utility and efficiency of prompt-learning
diagram with small language models (SLMs) for domain-specific text
classification, with a special emphasis on customer-agent interactions in the
retail domain. Our systematic examination reveals that prompt-learning offers
notable advantages for few-shot and zero-shot settings. Key findings include
the superiority of few-shot prompt-based model fine-tuning in contexts with
limited data, the heightened performance of prompt-learning pipeline applying
active few-shot sampling strategy and ensemble strategy of components in the
pipeline, and the observed performance boost when detailed instructions are
provided in zero-shot settings.

Our results hold significant implications for both researchers and industry
professionals. In particular, the adoption of prompt-learning on open-source
SLMs can revolutionize downstream NLP tasks in the industrial field. With the
strategies identified here, practitioners can design efficient and lightweight
classifiers even when working with minimal labeled data, showcasing the
adaptability and versatility of prompt-learning with open-source SLMs for
various NLP challenges.

While our study has shed light on many aspects of prompt-learning and SLMs,
future research opportunities abound. Potential areas include refining prompt-
learning pipeline construction, broadening the scope to other domain-specific
sectors like healthcare and finance, and devising methods to better leverage
lightweight SLMs in the era of LLMs. As the NLP realm further evolves, we
foresee prompt-learning becoming increasingly pivotal in efficiently analyzing
domain-specific textual data, especially in environments with scarce labeled
data.

## Acknowledgements

We extend our sincere appreciation to Joakim Nivre, professor of computational
linguistics at Uppsala University, for the continuous supervision from the
academic side. Additionally, we are grateful to Axel Goteman for the robust
technical assistance during the experiments on Google Cloud Platform.

This work was supported from the Digital D&A Customer Support & Returns Team
in Ingka Group, IKEA.

## References

  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ding et al. [2021] Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun.  Openprompt: An open-source framework for prompt-learning.  _arXiv preprint arXiv:2111.01998_ , 2021.
  * Gao et al. [2021] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.295.  URL https://aclanthology.org/2021.acl-long.295.
  * Grootendorst [2022] Maarten Grootendorst.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_ , 2022.
  * Liu et al. [2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35, 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Petroni et al. [2019] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1250.  URL https://aclanthology.org/D19-1250.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  _OpenAI blog_ , 2018.  URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
  * Schick and Schütze [2020] Timo Schick and Hinrich Schütze.  Few-shot text generation with pattern-exploiting training.  _arXiv preprint arXiv:2012.11926_ , 2020.
  * Schick and Schütze [2021] Timo Schick and Hinrich Schütze.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.20.  URL https://aclanthology.org/2021.eacl-main.20.

## Appendix

###  A Design of Prompt Template and Verbalizer in Few-shot Settings

The design of prompt template and verbalizer is a crucial step in the prompt-
learning pipeline. We firstly designed four different prompt templates that we
will use in our experiments as is shown in the table 12.

Template No. | Prompt Template
---|---
Template 1 | “{conversation} Classify this conversation : {mask}”
Template 2 | “{conversation} What is the topic of this conversation ? {mask}”
Template 3 | “{conversation} What is the intent of the customer ? {mask}”
Template 4 | “{conversation} We will be happy to help you with your {mask}.”
Table 12: Different prompt templates tried in few-shot prompt-based model
fine-tuning settings

And besides, the verbalizer, as a crucial component of the prompt-learning
pipeline, maps the outputs of the language model to the desired labels or
categories. We introduce four different verbalizers that we will use in our
experiments, shown below in the Table 13.

Label No. | Label words of Verbalizer 1 | Label Words of Verbalizer 2 | Label Words of Verbalizer 3 | Label Words of Verbalizer 4
---|---|---|---|---
0 | [“availability”] | [“availability”,“stock”,“order”] | [“availability”,“purchase”] | [“availability”,“stock”,“order”,“purchase”]
1 | [“general”] | [“general”,“membership”] | [“general”,“problem”] | [“membership”,“problem”,“account”]
2 | [“general”, “purchase”] | [“general”,“help”] | [“general”, “purchase”,“problem”] | [“general”,“help”,“problem”]
3 | [“help”,“integrate”,“product”] | [“assembly”,“product”] | [“help”,“integrate”,“product”,“purchase”] | [“assembly”,“product”,“purchase”]
4 | [“initiate”, “sales”] | [“aftersales”] | [“initiate”, “sales”,“problem”] | [“aftersales”,“problem”]
5 | [“issue”, “handling”] | [“issue”, “refund”] | [“issue”, “handling”,“refund”] | [“issue”, “refund”]
6 | [“order”, “creation”] | [“order”, “availability”,“delivery”] | [“order”, “creation”] | [“order”, “availability”,“delivery”]
7 | [“order”, “fulfillment”, “issues”] | [“order”, “product”, “refund”,“fulfillment”] | [“order”, “fulfillment”, “issue”,“problem”] | [“order”, “product”, “refund”,“problem”]
8 | [“order”, “processing”] | [“order”, “address”,“delivery”] | [“order”, “processing”] | [“order”, “address”,“delivery”]
9 | [“other”] | [“other”] | [“other”] | [“other”]
10 | [“planning”, “advice”] | [“planning”, “advice”,“suggestion”] | [“planning”, “advice”,“project”] | [“planning”, “advice”,“suggestion”,“project”]
11 | [“prepare”, “exchange”,“return”] | [“exchange”,“return”] | [“prepare”, “exchange”,“return”] | [“exchange”,“return”]
12 | [“product”, “service”, “information”] | [“stock”,“delivery”,“information”,“order”] | [“product”, “service”, “information”,“order”] | [“stock”,“delivery”,“information”,“order”]
13 | [“service”, “fulfillment”] | [“service”, “fulfillment”] | [“service”, “fulfillment”,“order”] | [“service”, “fulfillment”,“order”]

Table 13: Different verbalizers tried in few-shot prompt-based model fine-
tuning settings

The first verbalizer harnesses the phrasing of original intention labels set
in the dataset. The second verbalizer extracts label words from the text via
the unsupervised topic modeling tool BERTopic
101010https://maartengr.github.io/BERTopic [Grootendorst, 2022] coupled with
manual scrutiny. Further, we automatically generated the label words based on
prompt template 4, following the methodology of LM-BFF [Gao et al., 2021].
These words then added the first two verbalizers, forming the third and fourth
variants.

###  B Detailed Evaluation Metrics

####  B.1 Accuracy

Accuracy is defined as:

$Accuracy=\frac{C}{T}$ (1)

Where $C$ represents the number of correctly classified samples and $T$ is the
total number of samples in the dataset.

####  B.2 Macro F1-score

Macro F1-score is the average of the F1-scores for all the classes:

${MacroF1-score}=\frac{1}{N}\sum_{i=1}^{N}{F1-score}_{i}$ (2)

where $N$ is the number of classes. The F1-score for each class, denoted as
${F1-score}_{i}$, is the harmonic mean of precision ($P$) and recall ($R$):

${F1-score}_{i}=\frac{2\times P\times R}{P+R}$ (3)

Macro F1-score is preferred over accuracy in many scenarios because it
considers each class equally, irrespective of its size.

###  C Hyperparameter Settings

We provide a comprehensive overview of the experimental environment and
hyperparameters used with OpenPrompt in Table 14 and 15.

Experimental Environment | Detail
---|---
Platform | GCP (Google Cloud Platform)
PyTorch version | 1.13
Machine Type | 16 vCPUs, 104 GB RAM
GPUs | NVIDIA T4 * 2
Table 14: Experimental enviroment OpenPrompt Hyperparameters | Detail
---|---
Random seed | 144
Batch size | 5
Training Epochs | 20
Training Strategy |  | Select the checkpoint out of 20,
---
where accuracy on dev set is the highest.
Optimizer | AdamW
Weight decay |  | 0.01 (except [“bias”, “LayerNorm.weight”]);
---
0 ([“bias”, “LayerNorm.weight”])
Learning rate | $3\times 10^{-5}$
Table 15: OpenPrompt hyperparameters