# PESCO: Prompt-enhanced Self Contrastive Learning
for Zero-shot Text Classification

Yau-Shian Wang Ta-Chung Chi Ruohong Zhang Yiming Yang
Carnegie Mellon University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We present PESCO, a novel contrastive learning framework that substantially
improves the performance of zero-shot text classification. We formulate text
classification as a neural text matching problem where each document is
treated as a query, and the system learns the mapping from each query to the
relevant class labels by (1) adding prompts to enhance label matching, and (2)
using retrieved labels to enrich the training set in a self-training loop of
contrastive learning. PESCO achieves state-of-the-art performance on four
benchmark text classification datasets. On DBpedia, we achieve 98.5% accuracy
without any labeled data, which is close to the fully-supervised result.
Extensive experiments and analyses show all the components of PESCO are
necessary for improving the performance of zero-shot text classification.

##  1 Introduction

Text classification is the task of assigning relevant category labels to each
input document. It is an important problem in machine learning research with a
wide spectrum of applications, including sentiment analysis Pang et al.
(2002); Maas et al. (2011); Socher et al. (2013); Tang et al. (2014), question
answering Rajpurkar et al. (2016, 2018), and intent classification Tur et al.
(2010), etc. Recently, deep neural networks have obtained remarkable
improvements in text classification, including CNNs Kim (2014); Zhang et al.
(2015), RNNs Tang et al. (2015); Yang et al. (2016), Transformers Vaswani et
al. (2017), and more, thanks to the successful modeling of contextualized
representations.

Despite the remarkable progress, training well-performing neural classifiers
still requires a large amount of human-labeled documents, which is costly and
time-consuming, especially for new application domains. This stimulates the
recent trend of exploring self-supervised pre-training neural models on text
classification tasks. In particular, pre-trained language models (PTLMs)
Devlin et al. (2019); Liu et al. (2019); Yang et al. (2019) clearly stand out
from other methods owing to the pre-training on large-scale unlabeled data.
Nevertheless, how to adapt PTLMs to downstream tasks with less supervision
remains an open question for the research community, inviting new ideas to
explore.

Prompt-based learning Brown et al. (2020); Shin et al. (2020); Liu et al.
(2021); Li and Liang (2021); Gao et al. (2021a) has been actively studied to
better adapt PTLMs to downstream tasks with the goal of reducing human
annotation effort. For example, PET Schick and Schütze (2020) is a prompt-
based method for few-shot text classification. It formulates the task as a
Cloze Test, where a PTLM is used to predict the output label(s) by completing
a prompt concatenated right after an input document. For example, the
sentiment of a product review is highly likely to be positive if a PTLM fills
the word “good” into the following input:

> [Review] $|$ It is a _ product.

This example shows that prompt-based learning could unleash the potential
power of a PTLM by constructing the input format of a downstream task in a way
that closely resembles the PTLM pre-training objective, which is masked
language modeling (MLM) in this case.

Motivated by the recent success of prompt-based learning, we propose PESCO, a
novel self-training framework for zero-shot classification that uses prompts
to enhance performance. The self-training consists of two iterative steps,
pseudo-label prediction and model update. To make label descriptions more
informative, we first put label descriptions into some predefined prompts and
call the enhanced descriptions label-prompts. As depicted in Figure 1, to
predict the pseudo-label of a document, PESCO formulates text classification
as a neural matching task. A pre-trained text encoder maps both documents and
label-prompts into a shared embedding space. A label whose embedding is
closest to the document is predicted as the pseudo-label.

To effectively update the text encoder with pseudo-labels, we propose the
Prompt-enhanced Label-aware Cloze Test (PLCT), a contrastive learning
framework for self-training. The text encoder is trained to match a document
and the text relevant to its pseudo-label. The relevant texts include pseudo-
label prompts and the key sentences from the documents assigned to the same
pseudo-label. The key sentence of each document is the sentence most related
to its pseudo-label.

In our experiments, we show that the iterative self-training consistently
improves the classification performance compared to the same model without
self-training and that our proposed approach substantially outperforms other
strong zero-shot classification baselines. On some datasets, the zero-shot
results are even on par with a fully supervised baseline. On the Dbpedia
dataset, in particular, PESCO achieves 98.5% accuracy without any labeled
data.

In summary, the contributions of this paper are twofold:

  1. 1.

We explore text classification in a neural matching formulation enhanced by
prompts. We demonstrate that even without any finetuning on the text encoder,
this straightforward formulation is an effective method for zero-shot text
classification.

  2. 2.

The potential of contrastive learning for self-training has not been explored.
We show that this is a promising direction for self-training and can achieve
state-of-the-art performance on zero-shot text classification.

##  2 Related Work

###  2.1 Contrastive Learning

Contrastive learning (CL) Chopra et al. (2005); Hadsell et al. (2006) is a
metric learning method that aims to pull closer similar inputs in the
embedding space. Recently, the most popular and efficient methods for CL
involve batch contrastive learning He et al. (2019); Chen et al. (2020), which
put similar inputs (positive pairs) and dissimilar inputs (negative pairs) in
the same batch, simultaneously minimizing the distance of representations from
positive pairs, while maximizing the distance of negative pairs.

The key to CL is how to construct positive samples. Based on downstream
applications, there are various ways to formulate the positive pairs. In self-
supervised pre-training, the positive pairs are usually formulated by data
augmentation. That is, different versions of a distorted sample are treated as
a positive pair. In supervised contrastive learning Khosla et al. (2020), the
examples belonging to the same class are viewed as a positive pair.

Figure 1: In this example, there are three classes, whose label descriptions
are “sports”, “business”, and “world” respectively. We convert the
descriptions into label-prompts by placing them into a template. The model
predicts a label whose label-prompt embedding is the most similar to the
document embedding.

In NLP, CL is usually used as an additional self-supervised pre-training to
PTLMs because the sentence embeddings from PTLMs without fine-tuning are not
ready to be used in downstream tasks Li et al. (2020). SimCSE Gao et al.
(2021b) employs dropout as minimal data augmentation and obtains state-of-the-
art unsupervised sentence representations. In supervised SimCSE, the sentences
with entailment relation are viewed as a positive pair. Other approaches for
data augmentation include sentence reformulation Wu et al. (2020), back
translation Fang et al. (2020), dual encoder Carlsson et al. (2021), language
model corruption Meng et al. (2021), and translation pairs Wang et al. (2022).

In addition, CL is a commonly used training algorithm for neural text
retrieval Xiong et al. (2021). Inverse cloze test (ICT) Lee et al. (2019) is
the most commonly used contrastive pre-training task for retrieval that
predicts a randomly selected sentence from the rest of the texts. It is also
possible to construct positive pairs by leveraging the document structures
Chang et al. (2020).

###  2.2 Self-training and Zero-Shot Text Classifcation

##### Self-training

Self-training Yarowsky (1995); Nigam and Ghani (2000); Lee (2013); Xie et al.
(2020) is a widely used approach for semi-supervised learning and can have
additive improvement to pre-training in both computer vision Zoph et al.
(2020) and NLP Du et al. (2021). The paradigm of self-training is first using
a pre-trained base model as “teacher” to generate pseudo-labels on unlabeled
data. The pseudo-label is then used to train a “student” model. The teacher-
student training is performed iteratively until convergence.

##### Zero-shot Text Classification

Zero-shot classification aims to classify text using only label names without
human annotation. Self-training has demonstrated impressive performance on
few-shot Mukherjee and Hassan Awadallah (2020) and zero-shot text
classification. Unlike a few-shot setting which can use supervised information
to obtain a base model, in zero-shot text classification, obtaining a base
model is non-trivial. LOTClass Meng et al. (2020) leverages PTLMs to augment
label descriptions with semantically related words and then find category-
indicative words among these related words to label documents. They generalize
the performance to the documents without category-indicative words via self-
training. iPET Schick and Schütze (2020) formulates text classification as a
cloze test to help PTLMs understand the task. They design several types of
prompts for each dataset, and each type of prompt trains an individual teacher
model to annotate documents using self-training. A student model aggregates
the knowledge from the teachers via knowledge distillation. In this work, we
propose a novel self-training method for zero-shot text classification that
integrates self-supervised pre-training into self-training in a contrastive
learning framework.

##  3 Zero-shot Classification as Matching

In our zero-shot setting, there are $N$ unlabeled documents
$X=\\{x_{1},x_{2},\cdots,x_{N}\\}$ and a set of label descriptions
$C=\\{c_{1},c_{2},\cdots,c_{L}\\}$, where $L$ denotes the number of classes.
We aim to learn a scoring function $g(x,c)$ so that relevant document and
label description pairs can have higher scores. A label whose label
description has the highest score is selected as model prediction:

$\hat{y}=\arg\max_{j}\ g(x,c_{j}),$ (1)

Inspired by the recent success of pre-trained sentence encoder Gao et al.
(2021b); Chuang et al. (2022) which has shown impressive performance on
matching relevant texts, we explore using pre-trained encoders as
$g(x,c_{j})$. Specifically, as illustrated in Figure 1, we formulate zero-shot
text classification as a neural text matching problem. Both document and label
descriptions are encoded into dense vectors by a shared encoder. The matching
score can be obtained by measuring cosine similarity between dense vectors.

However, label descriptions are usually a few words rather than a sentence
with full semantics, which makes PTLMs unable to fully understand the meaning
of the labels. To tackle this, query reformulation Nogueira and Cho (2017);
Petroni et al. (2020) is a commonly used technique in retrieval to enhance the
semantics of a query. This technique can be further incorporated with prompt-
based learning Schick and Schütze (2020), which has shown that adding prompts
to a text helps PTLMs understand classification tasks. We use a prompt
function $p(\cdot)$ to convert a label description $c$ into a prompt by
placing label descriptions into pre-defined templates. We design $T$ templates
for each dataset, and the scoring function is:

$g(x,c)=\frac{1}{T}\sum_{i=1}^{T}sim(f_{\theta}(x),f_{\theta}(p^{i}(c))),$ (2)

where $f_{\theta}(\cdot)$ is a text encoder with parameters $\theta$ that maps
an input text to a dense embedding, and $sim(\cdot)$ is a similarity function.
For the rest of our paper, we use cosine similarity as $sim(\cdot)$. For
simplicity, in the rest of the article, we use $p_{j}$ to refer
$p^{i}(c_{j})$, which is the “label-prompt“ of label $j$ with $i$ randomly
sampled from $\\{1,\cdots,T\\}$.

##  4 PESCO

Figure 2: The framework of the PLCT. (A) Suppose the pseudo-label
$\hat{y}_{1}$ for $x_{1}$ is $1$. We select $s^{2}_{1}$ as the key sentence
$k_{1}$ for the document $x_{1}$ because the embedding of $s^{2}_{1}$ is the
most similar to the embedding of label-prompt $p_{1}$. $\hat{x}_{1}$ is the
augmented version of $x_{1}$, which removes $s^{2}_{1}$ from $x_{1}$. (B) We
use $k$ and $\hat{x}$ from part (A) to construct an example batch of PLCT with
batch size $B=3$. Similar to self-supervised training, we use $\hat{x}_{1}$ to
retrieve $k_{1}$ because they are from the same document. We use $\hat{x}_{1}$
to retrieve $k_{2}$ because $x_{1}$ and $x_{2}$ have the same pseudo-label. We
also use $x_{1}$ to retrieve the its pseudo-label-prompt $p_{1}$. (C) We
separate PLCT into LCT and PCL losses.

PESCO is a simple but effective self-training framework for zero-shot text
classification. Algorithm 1 gives an overview of PESCO. In our iterative self-
training loop, we first use a pre-trained sentence encoder $f_{\theta}$ to
generate pseudo-labels (i.e. predicted labels) by the matching process
described in Section 3. We then use the pseudo-labels to update $f_{\theta}$
by Prompt-enhanced Label-aware Cloze Test (PLCT), which leverages pseudo-
labels to construct positive training pairs. We continue the self-training
process by iteratively generating pseudo-labels and updating the model using
the PLCT objective function.

###  4.1 Prompt-enhanced Label-aware Cloze Test

We propose Prompt-enhanced Label-aware Cloze Test (PLCT) to update our model
using pseudo-labels. As shown in Figure 2, PLCT consists of two losses, Label-
aware Cloze Test (LCT) loss and Prompt Contrastive Loss (PCL). To compute LCT,
for each document, we first select a key sentence from the document that is
most relevant to its pseudo label. In LCT, given a document, the positive
texts are the key sentences from the documents belonging to the same pseudo-
label. For PCL, the positive texts for a document are its pseudo-label prompt
(i.e. the label-prompt of a pseudo-label). We combine these two losses by
putting the positive texts of LCT and PCL into the same batch of a contrastive
loss.

####  4.1.1 Label-aware Cloze Test

LCT is inspired by Inverse Cloze Test Lee et al. (2019) which is a widely used
self-supervised pre-training task for neural text retrieval. It uses a
randomly selected sentence from a document to match the remaining texts. In a
document, as some sentences don’t contain useful information, using a randomly
selected sentence for training is not an optimal choice. Instead, we use
pseudo-label to select the key sentences. Note that we use “Cloze Test“
without “Inverse“ because we use the remaining long texts to match its
relevant short sentences, which can be viewed as label descriptions.

As illustrated in Figure 2-(A), given an input document
$x_{i}=\\{s_{i}^{1},s_{i}^{2},\cdots,s_{i}^{n}\\}$ consists of $n$ sentences
and its predicted pseudo label $\hat{y}_{i}$, its key sentence $k_{i}$ is
$s_{j}$, where:

$j=\arg\max_{n}\ g(s_{i}^{n},p_{\hat{y}_{i}}).$ (3)

Here, $g(\cdot)$ is the scoring function in Eq.(1). As key sentence $k_{i}$ is
more relevant to the pseudo-label than any other sentences in $x_{i}$,
optimizing this objective is similar to minimize the distance between a
document and its pseudo-label in embedding space, so $k_{i}$ can be viewed as
an augmented version of the pseudo-label prompt. Predicting the augmented
version can have additional training signal than simply predicting pseudo-
label prompt. We provide a real example of $\hat{x}$ and $k$ in Table. 1 and
more examples can be found in the Appendix Table 8.

Since key sentences are highly correlated to corresponding pseudo-label
prompts, given a document, it should not only match its key sentence but also
key sentences in documents assigned to the same pseudo-label as shown in
Figure 2 (C)-1. We use the supervised contrastive loss Khosla et al. (2020) to
optimize LCT, which extends the SimCLR Chen et al. (2020) to allow multiple
positive keys for a query in a supervised setting. Specifically, let
$\mathit{I}=\\{1,\cdots,B\\}$ be the set of the indices of the texts in a
batch, where $B$ denotes the batch size. The LCT loss $\mathcal{L}_{LCT}$ is
written as:

$\sum_{i\in\mathit{I}}\frac{\scriptstyle-1}{\scriptstyle|K(i)|}\sum_{\hat{k}\in
K(i)}\log{\frac{e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(\hat{k}))/\gamma}}{\sum_{j\in\mathit{I}}e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(k_{j}))/\gamma}}}.$
(4)

Here, $K(i)\equiv\\{k_{j},\forall j\in\mathit{I}:\hat{y}_{j}=\hat{y}_{i}\\}$
denotes the keys belonging to the same pseudo class $\hat{y}_{i}$, and
$\gamma$ denotes a temperature commonly-used in CL. To prevent trivial
training signal, the input document is $\hat{x}_{i}=x_{i}\setminus\\{k_{i}\\}$
rather than $x_{i}$, where the key sentence $k_{i}$ is removed.

Label Description |  $x$ | $k$
---|---|---
Family and Relationship |  how do you know if you’re in love? is it possible to know for sure? in my experience you just know. it’s a long term feeling of always wanting to share each new experience with the other person in order to make them happy, to laugh or to know what they think about it. it’s jonesing to call even though you just got off an hour long phone call with them. it’s knowing that being with them makes you a better person. it’s all of the above and much more. |  how do you know if you’re in love?
Table 1: An example of the document $\hat{x}$ and the selected pseudo positive
keys $k$ in Yahoo Answers. In this example, $k$ is very related to label
description.

####  4.1.2 Prompt Contrastive Loss

As the update target of self-training is to maximize the similarity between
$x_{i}$ and its pseudo-label-prompt $p_{\hat{y_{i}}}$ in embedding space, we
use the prompt contrastive loss (PCL) $\mathcal{L}_{PCL}$ to directly maximize
the similarity:

$\mathcal{L}_{PCL}=-\sum_{i\in\mathit{I}}\log{\frac{e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(p_{\hat{y_{i}}}))/\gamma}}{\sum_{c\in
C}e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(p(c)))/\gamma}}}.$ (5)

Depicted in Figure 2 (C)-2, this loss predicts $\hat{y_{i}}$ from
$\hat{x}_{i}$.

###  4.2 Combining LCT and PCL

Naturally, to combine LCT and PCL, the simplest way is to use
$\mathcal{L}_{PCL}+\mathcal{L}_{LCT}$ as the final training loss. However, we
found that minimizing this loss has limited improvement over minimizing
$\mathcal{L}_{LCT}$ or $\mathcal{L}_{PCL}$ alone. As depicted in Figure 2 (B),
we come up with a more effective approach that puts the positive texts from
these two losses into the same batch. By doing so, pseudo keys $k$ and pseudo
prompt $p$ can serve as mutually challenging negative samples, thus enhancing
the representative power through more difficult contrastive tasks. In our
experiment, this simple solution significantly improves the performance.

Specifically, we use $\hat{x}_{i}$ as a query to retrieve (1) the key $k_{i}$
from the same text $x_{i}$, (2) $K(i)$, the keys belonging to the same pseudo
class $\hat{y}_{i}$, and (3) the positive pseudo-label-prompt
$p_{\hat{y}_{i}}$. The PLCT loss $\mathcal{L}_{PLCT}$ is written as:

$\sum_{i\in\mathit{I}}\frac{\scriptstyle-1}{\scriptstyle|A(i)|}\sum_{a\in
A(i)}\log{\frac{e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(a))/\gamma}}{\sum_{m\in
M}e^{sim(f_{\theta}(\hat{x}_{i}),f_{\theta}(m))/\gamma}}}$ (6)

Here, $A(i)\equiv K(i)\cup\\{p_{\hat{y}_{i}}\\}$ is the set of positive texts
in the mini-batch for $x_{i}$, $M\equiv\\{k_{j},\forall
j\in\mathit{I}\\}\cup\\{p_{c},\forall c\in C\\}$ denotes the set of all the
candidate keys.

Interestingly, $\hat{x}_{i}$ can be viewed as a challenging data augmentation
of $x_{i}$ for predicting pseudo-label prompt because it removes the most
salient sentence from $x_{i}$. A model can make a prediction simply based on
one salient sentence, neglecting the information of remainder. This data
augmentation method forces the model to capture additional information.

Algorithm 1 PESCO

Require: Unlabeled texts $X$, label descriptions $C$.
Initialization: A pre-trained sentence encoder $f_{\theta}(\cdot)$.
Repeat until convergence:

  1. 1.

Use $f_{\theta}(\cdot)$ to generate hard pseudo-labels $\hat{y}$ with Eq.(1)
for all unlabeled texts without data augmentation.

  2. 2.

Sample $T_{t}$ training pairs $(x,\hat{y})$ from step 1 based on the pseudo-
label predicted probability. Use these pairs to update the $\theta$ of
$f_{\theta}(\cdot)$ that minimizes the $\mathcal{L}_{PLCT}$ in eq 6.

  3. 3.

With a more powerful $f_{\theta}(\cdot)$, go back to step 1.

Output: $f_{\theta}(\cdot)$

Dataset | Class Number | Test Examples
---|---|---
AG News | 4 | 7,600
DBPedia | 14 | 70,000
Yahoo Answers | 10 | 60,000
Amazon | 2 | 400,000
Table 2: Dataset statistics.

###  4.3 Self-training

Algorithm 1 describes PECOS self-training loop. Our self-training algorithm is
a simplified version of noisy student training Xie et al. (2020) that a single
model alternately serves as a student and a teacher. The key idea of noisy
student training is that the teacher uses clean data without data augmentation
to generate pseudo-labels, while the student learns to predict the pseudo-
label on augmented data.

We first use pre-trained sentence encoder to initialize $f_{\theta}(\cdot)$.
Then, in step 1, $f_{\theta}(\cdot)$ serves as a teacher to generate pseudo-
labels from clean data $x$ as described in Section 3. In step 2,
$f_{\theta}(\cdot)$ serves as a student that learns to increase the
probability of predicting pseudo-labels by minimizing $\mathcal{L}_{PLCT}$.
Step 2 is a noisy student training because the model takes $\hat{x}$ as input
rather than clean $x$. The self-training repeats step 1 and step 2 until
convergence. We use $f_{\theta}(\cdot)$ from the last iteration as our final
model.

In the algorithm, we set $T_{t}=d\cdot T_{t-1}$ that gradually increases $T$
until a threshold ${T}^{\prime}$. The probability of sampling a pseudo
training pair is proportional to the normalized scores outputed by the score
function, so a more confident pseudo training pair is more likely to be
sampled. When sampling pseudo training pairs, we found that it is important to
keep the ratio of all the labels balanced. If a class doesn’t have enough
instances to be sampled, then we upsample the class to keep it balanced.

Datasets |  Label Descriptions |  Prompts
---|---|---
AG news |  (1)World (2)Sports (3)Business (4)Technology and Science |  (1)Category: [desc] news. (2)[desc] news.
DBpedia |  (1)company (2)school and university (3) artist (4)athlete (5)politics (6)means of transportation (7)building (8)river and mountain and lake (9)village (10)animal species (11)plant and tree (12)album (13)film (14)novel and publication and book |  (1)Category: [desc]. (2)It is about [desc].
Yahoo Answers |  (1)Society and Culture (2)Science and Mathematics (3)Health (4)Education and Reference (5)Computers and Internet (6)Sports (7) Business and Finance (8)Entertainment and Music (9)Family and Relationships (10)Politics and Government |  (1)Category: [desc]. (2)It is about [desc].
Amazon-review-P |  bad, good |  (1)It is a [desc] product. (2)In summary, the product is [desc]
Table 3: The label descriptions and their prompts. [desc] in the templates
denotes the label descriptions.

##  5 Experiments

###  5.1 Experimental Setting

##### Implementation Details

Inspired by Yin et al. (2019) who formulate zero-shot text classification as
entailment prediction, we choose the version of SimCSE Gao et al. (2021b) pre-
trained on natural language inference (NLI) task 111We choose the model named
“sup-simcse-bert-base-uncased” at https://github.com/princeton-nlp/SimCSE. as
our text encoder for all datasets. Our experiments have shown that sentence
encoder fine-tuned on NLI performs better on zero-shot classification tasks.
We use the representation outputted by the last layer as our sentence
representation.

Following supervised contrastive learning Khosla et al. (2020), the value of
$\gamma$ in all equations is set to be $0.07$. For the value of $d$ in the
self-training section, we set it to be $2$ because we want the model to
annotate unlabeled data slowly. The details of other hyperparameters in the
Appendix B.

##### Datasets

We conduct experiments on various text classification datasets: (1)AG News:
topic classification on news article. (2)DBpedia: Ontology classification on
selected classes from DBpedia. (3)Yahoo Answers: question type classification.
(4)Amazon: binary sentiment classification on Amazon product review. The
statistics of these dataset are listed in Table 2.

We provide the label descriptions in Table 3. The label descriptions of Yahoo
Answers and AG news are mainly from the original dataset, and the label
description of DBpedia is mainly from LOTClass Meng et al. (2020).

###  5.2 Effect of Using Prompts

Id | Self-train | Methods | AG News | DBpedia | Yahoo Answers | Amazon
---|---|---|---|---|---|---
$[1]$ | No | SimCSE w/o prompt | 69.7 | 73.8 | 55.2 | 88.3
$[2]$ | No | SimCSE w/ prompt | 76.3 | 76.0 | 56.5 | 88.3
$[3]$ | No | PET | 79.4 | 75.2 | 56.4 | 87.1
$[4]$ | Yes | iPET | 86.0 | 85.2 | 68.2 | 95.2
$[5]$ | Yes | LOTClass | 86.4 | 91.1 | – | 91.6
$[6]$ | Yes | PESCO w/o prompt | 87.1 | 96.0 | 69.9 | 95.1
$[7]$ | Yes | PESCO | 89.6 | 98.5 | 71.1 | 95.2
$[8]$ | – | Supervised | 94.2 | 99.3 | 77.3 | 97.1
Table 4: Test-set accuracy of zero-shot text classification methods. The Self-
train column indicates whether a method performs self-training on unlabeled
data.

We investigate whether supplementing the label description with the prompt can
help the model better understand the meaning of the label, and thus improve
the performance. In Table 3, we provide the label descriptions and the prompts
we use. For each dataset, we manually design two prompts, where the ’[desc]’
in the templates is the label description. For example, given a label
description “Health”, the prompting function converts it into either “It is
about Health” or “Category: Health”.

Our experiments showed that the choice of prompts doesn’t affect performance
much as long as reasonable prompts are given. For example, in AG news, without
self-training, the accuracy of using “Category: <label> news”, “This is about
<label> news”, and “<label> news” are 76.4, 76.0, and 78.0 respectively.
Furthermore, our scoring function, as described in Eq.(2), combines the scores
of different prompts, which further reduces the gap. The performance gap among
different prompts is less than 2% without self-training and less than 1% after
self-training.

In Table 4, we analyze the effect of using prompts on SimCSE without self-
training. By comparing [1] with [2], we find that using prompts for retrieval
improves the performance on most of the datasets, especially on AG News. We
find that without the word “news”, the model can not understand the meaning of
the class only with the description “world”. Using the prompt-enhanced SimCSE
[2] as the initial base model provides a better start for self-training.
However, comparing with the performance gap of [1] and [2] in Table 4, we
observed that the gap between [6] and [7] becomes smaller, which indicates
that the effect of using prompts decreases after self-training.

Id | Methods | AG News | DBpedia | Yahoo Answers | Amazon
---|---|---|---|---|---
$[1]$ | PESCO | 89.6 | 98.5 | 71.1 | 95.2
$[2]$ | PESCO - R | 87.0 | 97.1 | 69.1 | 95.0
$[3]$ | LCT | 88.0 | 89.1 | 69.6 | 94.3
$[4]$ | LCT - R | 80.7 | 86.9 | 68.6 | 93.3
$[5]$ | PCL | 87.8 | 89.4 | 68.7 | 95.1
$[6]$ | LCT+PCL | 88.2 | 97.0 | 69.8 | 95.2
$[7]$ | PESCO w/o aug | 87.8 | 96.7 | 68.6 | 93.5
Table 5: Contrastive losses of different methods. The methods end with “-R”
means their pseudo positive key sentences are randomly selected instead of
picking the most salient sentence.

###  5.3 Zero-shot Text Classification

In Table 4, we compare our results against two state-of-the-art zero-shot text
classification baselines, LOTClass Meng et al. (2020) and iPET Schick and
Schütze (2020). We select these two methods as our baselines because they both
employ self-training for zero-shot classification. In [1], [2], and [3], they
do not employ self-training on unlabeled data, so the Self-train column is
“No”. In [7], we report the best results over 5 runs on PESCO single model
performance without an ensemble. We also report the average, maximum, and
minimum accuracy over 5 runs in Appendix Table 6. In [8], to see the gap
between zero-shot and fully-supervised settings, we train a typical BERT
Devlin et al. (2019) classifier on a labeled training set. We jointly finetune
BERT and a linear classifier on top of BERT [CLS] output layer.

##### Effect of Self-training

First, by comparing [7] against [2] in Table 4, we find that the proposed
self-training framework significantly improves the performance by more than
10% on average. On DBpedia, self-training improves performance substantially
by 20%, and it even achieves 98.5% accuracy. This demonstrates that self-
training is an effective method to enhance performance after general pre-
training, closing the gap between fully supervised training.

##### Comparison against LOTClass

Comparing [7] PESCO against [5] LOTClass, PESCO significantly improves the
zero-shot text classification performance on all datasets. LOTClass leverages
PTLMs to find the category-indicative words which are semantically related to
label descriptions. The documents containing category-indicative words are
classified as the corresponding category. Our method uses a pre-trained
sentence encoder to define the relevance between document and category, which
is more effective and requires less human heuristics.

##### Comparison against iPET

Our main baseline is [4] iPET, which uses [3] PET as a base model to generate
initial pseudo-labels followed by a series of self-training steps. We find
that our base model [2] achieves similar performance with [3] on all datasets
except Ag News, on which ours lags behind by 3%. The lesson here is that using
text retrieval as a means of text classification gives a similar performance
to that using cloze tests. Next, our full model [7] is also better than [4]
iPET on three datasets while achieving similar performance on the Amazon
dataset, demonstrating the effectiveness of our method. Also, we notice that
PET requires a massive model ensemble (e.g. 15 models) to achieve the reported
accuracy. We run their code with a PvP ensemble without using various random
seeds for ensembling. Even with this simplified setting, iPET still needs far
more disk space (1.4 GB vs 26 GB) and more training time than us in that we do
not need to train various models for model ensembling in each self-training
step.

Note that It is not feasible to test our method using Roberta-base/large
because language models without SimCSE finetuning poorly capture the semantic
meaning of texts in cosine similarity space and cannot be used for retrieval.
On the other hand, simCSE is finetuned for sentence embeddings, making
language models lose text generation ability. Because iPET and LOTClass
require language models to generate tokens, using SimCSE-Roberta for iPET or
LOTClass is also not feasible.

###  5.4 Ablation Study and Analysis

##### Comparison of different contrastive losses

The results of different contrastive learning losses are shown in Table 5. In
the table, LCT means we only use $\mathcal{L}_{LCT}$ in Eq.( 4) to train our
model, PCL means we use $\mathcal{L}_{PCL}$, and LCT+PCL means we sum the
$\mathcal{L}_{LCT}$ and $\mathcal{L}_{PCL}$ as our loss function rather than
using PLCT loss which puts keys and label-prompts in the same batch. The
methods end with “-R” means the pseudo positive sentences $k$ are randomly
selected from the documents instead of picking the most salient sentences.

In LCT, although it doesn’t explicitly minimize the distance between an input
document and its predicted pseudo-label-prompt, optimizing this loss still
obtains performance similar to PLC. This implies the selected key sentences
can serve as augmented version of label-prompts.

Furthermore, we analyze the difference in the performance between using
randomly selected sentences and the most salient sentences. By comparing [1]
and [2], and [3] and [4], we can see that the model has a significant
performance drop in predicting randomly selected sentences. This demonstrates
the importance of choosing a salient sentence as the training target.

Finally, to demonstrate the effectiveness of putting pseudo-label-prompts and
key sentences in the same batch, we compare [1] against [6]. [1] yields better
performance than [6], which implies using this more challenging contrastive
task allows the model to learn more general representations.

##### Effect of Data Augmentation

In Table 5, [7] PESCO w/o aug means we use $x_{i}$ as a query to retrieve its
positive examples $A(i)$ instead of using $\hat{x}_{i}$ as a query. Comparing
[1] and [7], removing the most salient sentence from a document is an
effective data augmentation method that can greatly improve performance. This
is consistent with previous literature Xie et al. (2020) that updating student
models with noisy data is important in self-training.

##  6 Conclusion

This paper presents a novel approach to zero-shot text classification, which
significantly improves the SOTA results on four benchmark datasets by
formulating the classification task as a prompt-enhanced retrieval problem and
by combining the strengths of pre-trained language models and contrastive
learning over pseudo-labeled data in a self-training loop. Our experiments in
comparison with representative baselines and ablation analysis show evidence
for the effectiveness of the proposed approach.

##  7 Limitations

The main limitation of our method is that it heavily depends on the quality of
the label description. If a label description does not precisely describe the
meaning of the label, our method cannot work. For some classification tasks
such as microaggression detection, their labels have abstract meaning that is
difficult to be understood by pre-trained language models. Similarly, our
method cannot work on the domain that is not covered by the pre-training
corpora of language models, such as the medical domain.

Another limitation of our method is that PLCT loss cannot handle short texts.
If a text consists of only one sentence, PLCT loss will no longer work because
LCT requires a document to be more than one sentence. In this case, PCL loss
can still be used for self-training.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Carlsson et al. (2021) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021.  Semantic re-tuning with contrastive tension.  In _International Conference on Learning Representations_.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020\.  Pre-training tasks for embedding-based large-scale retrieval.  In _International Conference on Learning Representations_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_.
  * Chopra et al. (2005) S. Chopra, R. Hadsell, and Y. LeCun. 2005.  Learning a similarity metric discriminatively, with application to face verification.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , volume 1, pages 539–546 vol. 1.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022.  DiffCSE: Difference-based contrastive learning for sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4207–4218, Seattle, United States. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2021) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021.  Self-training improves pre-training for natural language understanding.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5408–5418, Online. Association for Computational Linguistics.
  * Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020.  Cert: Contrastive self-supervised learning for language understanding.
  * Gao et al. (2021a) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gao et al. (2021b) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2_ , CVPR ’06, page 1735–1742, USA. IEEE Computer Society.
  * He et al. (2019) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019.  Momentum contrast for unsupervised visual representation learning.  _arXiv preprint arXiv:1911.05722_.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Lee (2013) Dong-Hyun Lee. 2013.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _ICML Workshops_.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096. Association for Computational Linguistics.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020.  On the sentence embeddings from pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9119–9130, Online. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 34.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020.  Text classification using label names only: A language model self-training approach.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9006–9017, Online. Association for Computational Linguistics.
  * Mukherjee and Hassan Awadallah (2020) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020.  Uncertainty-aware self-training for few-shot text classification.  In _Advances in Neural Information Processing Systems (NeurIPS 2020)_ , Online.
  * Nigam and Ghani (2000) Kamal Nigam and Rayid Ghani. 2000.  Analyzing the effectiveness and applicability of co-training.  In _Proceedings of the Ninth International Conference on Information and Knowledge Management_ , CIKM ’00, page 86–93, New York, NY, USA. Association for Computing Machinery.
  * Nogueira and Cho (2017) Rodrigo Nogueira and Kyunghyun Cho. 2017.  Task-oriented query reformulation with reinforcement learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 574–583, Copenhagen, Denmark. Association for Computational Linguistics.
  * Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002.  Thumbs up? sentiment classification using machine learning techniques.  In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)_ , pages 79–86. Association for Computational Linguistics.
  * Petroni et al. (2020) Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020.  How context affects language models’ factual predictions.  In _Automated Knowledge Base Construction_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Schick and Schütze (2020) Timo Schick and Hinrich Schütze. 2020.  Exploiting cloze questions for few-shot text classification and natural language inference.  _Computing Research Repository_ , arXiv:2001.07676.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Tang et al. (2015) Duyu Tang, Bing Qin, and Ting Liu. 2015.  Document modeling with gated recurrent neural network for sentiment classification.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1422–1432, Lisbon, Portugal. Association for Computational Linguistics.
  * Tang et al. (2014) Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014.  Learning sentiment-specific word embedding for Twitter sentiment classification.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1555–1565, Baltimore, Maryland. Association for Computational Linguistics.
  * Tur et al. (2010) Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010.  What is left to be understood in atis? In _2010 IEEE Spoken Language Technology Workshop_ , pages 19–24.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang et al. (2022) Yau-Shian Wang, Ashley Wu, and Graham Neubig. 2022.  English contrastive learning can learn universal cross-lingualsentence embeddings.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.  Clear: Contrastive learning for sentence representation.  _ArXiv_ , abs/2012.15466.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. 2020.  Self-training with noisy student improves imagenet classification.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _International Conference on Learning Representations_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1480–1489, San Diego, California. Association for Computational Linguistics.
  * Yarowsky (1995) David Yarowsky. 1995.  Unsupervised word sense disambiguation rivaling supervised methods.  In _33rd Annual Meeting of the Association for Computational Linguistics_ , pages 189–196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3914–3923. Association for Computational Linguistics.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. 2020.  Rethinking pre-training and self-training.  In _Advances in Neural Information Processing Systems_ , pages 3833–3845.

| AG News | DBpedia | Yahoo | Amazon
---|---|---|---|---
avg | 88.7 | 96.9 | 70.5 | 94.3
max | 89.6 | 98.5 | 71.1 | 95.2
min | 87.7 | 96.1 | 70.0 | 93.9
Table 6: Average/minimum/maximum accuracy over 5 runs. | AG News | DBpedia | Yahoo Answers | Amazon
---|---|---|---|---
Learning rate | 1e-5 | 1e-5 | 5e-6 | 5e-6
Document length | 156 | 128 | 192 | 128
Batch size | 32 | 32 | 32 | 32
Epsilon | 1e-6 | 1e-8 | 1e-8 | 1e-8
${T}^{\prime}$ | 0.2N | 0.5N | 0.1N | 0.1N
Epoch | 5 | 5 | 2 | 1
Table 7: Hyperparameters.

##  Appendix A Discussion

##### Text Classification as neural text retrieval

Formulating text classification as neural retrieval is straightforward but not
widely explored by previous work. In this work, we show that this formulation
can also obtain good performance with a well-pre-trained sentence encoder. The
benefit of this formulation over cloze test is that we don’t need to restrict
the label description to only one word. PET requires a carefully selected word
(verbalizer) to represent each class. If a classification task has hundreds or
even more than thousands of categories, it is not feasible to manually select
a word to represent each class. Furthermore, if the meaning of a category in a
classification task is too abstract or complex, we cannot simply represent it
with a single word. Our formulation allows the model to describe categories
using sentences or even short texts and maybe a better choice for more
challenging classification tasks.

##### Contrastive Learning for Self-training

The effect of contrastive learning for self-training is not well-studied by
previous work. Contrastive learning obtains impressive results on unsupervised
representation learning. In a supervised setting, it is also robust to noisy
labels and noisy data, and it also shows impressive performance on a few-shot
classification. Considering these good properties of contrastive learning, we
believe contrastive learning is a promising direction for self-training and
propose PESCO to explore its potential on zero-shot text classification.

##  Appendix B Hyperparameters

Figure 3: The effect of different batch sizes. Figure 4: Training epoch versus
validation set accuracy on AG News dataset.

As indicated by previous work Chen et al. (2020), using a larger batch size
generally yields better performance because it includes more negative samples.
We analyze how different batch size influences the performance of PESCO in
Figure 3. We found that PESCO is not very sensitive to batch size. Using a
smaller batch size only reduces the accuracy by less than 2%. Also, we observe
that our algorithm converges after 1000 steps (1 epoch) of training, and
additional training steps only slightly increase the performance. In other
datasets, our algorithm also converges after 1 training epoch.

We list the hyperparameters of our model in Table 7. We use AdamW as our
optimizer. The ${T}^{\prime}$ is the threshold mentioned in Section 4.3, we
set it proportional to $N$, where $N$ is the total number of unlabeled data in
the corresponding dataset. We find that the number of training epoch only
slightly influence the final performance that usually influences the accuracy
by less than 1%. In Figure 4, we plot the training epoch versus validation set
accuracy. Although we train 5 epochs on the AG news to obtain the best result,
the model actually converges in the early training stage. Similar training
curves can be observed in all the datasets.

Label Description |  $x$ | $k$
---|---|---
Family and Relationship |  where is the best place to look for love? it might be easy to use the internet- there are many good matching web sites that can help |  where is the best place to look for love?
Entertainment and Music |  what is the best place to get guitar lessons in the south bay area? looking for a great instructor and relatively affordable price. i have no experience but have a desire to learn. it’s really according to what you are looking for. certain teachers specialize in acoustic vs. electric (for example). your best bet is to place a request on a service such as click for lessons that will show you several teacher bios and let you decide for yourself. |  what is the best place to get guitar lessons in the south bay area?
Business and Finance |  does anyone know a good apartment rental agency around washington dc? i’ve had personal experience with archstone apartments and summit (just bought by camden) apartments in the past two years. while neither one is stellar, both were acceptable. both of these were in the northern virginia area - bedroom communities for d.c. best of luck apartment hunting! the housing market around here is absolutely insane. |  does anyone know a good apartment rental agency around washington dc?
Sports |  why are there 5 rings in the olympics symbol? what does it represent? i heard few theories about it but not sure what is the correct one the 5 rings were introduced at the the 1920 games in antwerp games. the rings included at least one color from the flag of every participating country. |  why are there 5 rings in the olympics symbol?
Table 8: More examples of the distorted document $\hat{x}$ and the selected
pseudo positive keys $k$ in Yahoo Answers. It happens that $k$ seems to be the
most important sentence of the texts, so their semantics are closest to label
descriptions.


# Content Addressable Parallel Processors on a FPGA

Ayush Salik  BINDS Lab, CICS
University of Massachusetts, Amherst
Email<EMAIL_ADDRESS>Manor Askenazi  Biomedical Hosting LLC
Arlington, MA
Email<EMAIL_ADDRESS>Edward Rietman  BINDS Lab, CICS
University of Massachusetts, Amherst
and Small-Technology Incubator LLC
Grantham, NH
Email<EMAIL_ADDRESS>
###### Abstract

In this short article, we report on the implementation of a Content
Addressable Parallel Processor using a FPGA. While Content addressable
memories have been implemented in FPGAs, to our knowledge this is the first
implementation in FPGA of Caxton C. Foster’s vision of parallel _processing_ ,
particularly the notions of parallel write as well as the combining of output
values, which are usually missing in more typical CAM implementations, such as
the ones designed for network routing. The resulting CAPP is made accessible
to a host computer over a USB/UART interface, using a straightforward serial
protocol that is demonstrated using a Python-based driver.

###### Index Terms:

FPGA, Verilog, CAPP, CAM

##  I Introduction

Content Addressable Parallel Processors (CAPPs) constitute an alternative to
the standard von Neumann architecture, featuring parallel-processing based on
content-addressable memory. Unlike Random Access Memory (RAM) which works by
performing operations on a word in memory by referring to its physical
address, a CAPP is able to select multiple memory locations simultaneously
based on pattern matching the contents of the memory being accessed.
Consequently, a CAPP can perform operations like writing into and reading from
multiple selected memory cells in constant time. The original intent of the
CAPP design was to serve as a general purpose computer, capable of parallel
processing. Furthermore, it was hoped that, by providing native support for
parallel pattern-matching and multi-write capability, the software for such
machines would be relatively easy to write (at least in comparison with other
parallel architectures). In practice, this did not occur and CAPPs eventually
found use primarily in a much more limited form of application specific CAMs,
primarily in the area of computer networking devices, e.g. in the form of fast
lookup tables used in network switches and routers.

The goal of our project was to expose a true CAPP capable of multi-write and
parallel read, as a convenient USB peripheral, to enable renewed
experimentation with this class of architecture. We therefore designed a
Verilog module for a parameterized (hence, scalable) CAPP. To this module we
added a USB/UART interface module which we manage using an FSM-based protocol.
The combined system was implemented on a TinyFPGA-BX [1] which we then control
over the USB/UART using a simple Python-based driver. We believe that CAPPs
have potential in graph theory, neural network caching, high-speed indexing,
regex computations etc. and we hope that an open-source, expandable CAPP
design which can be synthesized on an FPGA using open-source tools, can
provide a basis for renewed research into the application of CAPPs to these
diverse and challenging domains. Our implementation is fully open source and
can be found here [2].

##  II Content Addressable Parallel Processor

The circuitry of a CAPP can be broken down into three main parts. The tags,
the cells and the search registers. The search registers are the least complex
circuits in a CAPP. They consist of two different registers, the comparand and
the mask. The comparand is the word to search for, and the mask contains
locations of bits in the word to ignore during the search. The design for
these are given in figure 1.

Figure 1: Design of the CAPP search registers

The second most complex circuitry is found in the tags. Each tag is a one bit
register and there is one tag associated with each memory cell. If the bit of
a cell is high after a search, it means that the word in that cell matched the
search criteria. In other words, after a search, the cells with high tag bits
are the ones that constitute the answer set. As all the tag bits of a CAPP
have to change in constant time (i.e. in parallel), match lines from the
search registers go through each cell to create mismatch lines which turn
irrelevant tags off whenever the search signal is set. The circuitry also
includes logic for manually setting all bits high and selecting the first tag
bit that is on. The design for these are given in Figure 2.

Figure 2: Design of the CAPP tag registers

Arguably the most complex circuitry is found in the memory cells themselves,
as it encapsulates the main logic for writing, reading and searching. Each bit
of each cell is surrounded by two write lines, two search lines and one read
line. The n-th bits across all cells share these lines, i.e. each position has
a set of lines across all the cells. The value of each bit in each cell is
changed to 0 or 1 depending on which write line is set high. Conversely,
depending on the two search lines and the bit value, a mismatch line that is
connected to the tag bit of the cell is set high or low (see 4). This allows
the CAPP to search or write in parallel. The output of all the read lines is
the bitwise OR of all cells that have their tag bits on as shown in Figure 3.
The parallel writing and reading of results is precisely what distinguishes
CAPPs from the more common CAM architecture (as described in the
introduction).

Figure 3: Design of the parallel read logic Figure 4: Design of the parallel
matching logic

The design described above is identical to the introductory CAPP described in
Caxton Foster’s seminal book on the topic of CAPPs [3]. We implemented a
direct translation to Verilog and then used nextpnr [4] for the placing and
routing of our Verilog design. The choice of nextpnr was due to its ability to
take into account timing constraints which were important for the correct
implementation of the Serial UART module needed to function reliably despite
being connected to a CAPP module which requires potentially more than one
clock cycle to execute a request communicated from the host computer. The
resulting layout was synthesized and tested on the TinyFPGA-BX using a Python-
based driver to communicate with the resulting USB-CAPP device. The Finite
State Machine at the core of the communication protocol between the host
computer and the CAPP is described in the next section.

##  III The Finite State Machine

The FSM manages the interactions between the CAPP and the host computer and it
is responsible for orchestrating the individual CAPP operations necessary to
implement complex arbitrary algorithms on the host computer. It has numerous
states, such as those for sending or receiving data, loading data into the
CAPP, searching the memory (which in turn requires setting the comparand and
mask) as well as reading the combined output. To implement the communication
with the host computer we use base UART module provided in David Things’
Github repository [5]. To reduce complexities and minimize our LUT-budget, we
adopted the 48MHz clock speed used by the UART module throughout this project.

Figure 5: Overview of the FSM underlying the CAPP-host protocol

As shown in Figure 5, our FSM will typically break down a task into multiple
short procedural tasks to reduce both magnitude and variance of the latency
associated with each atomic state/task. In other words, we attempted to have a
repertoire of tasks which all had (similarly) short latencies. Consequently,
an overall task may necessitate several atomic states before returning the
CAPP to its default ready state. For example, the operations for searching
involve several steps, which are typically initiated by the Python "driver"
and include:

  * •

Resetting the tags (set tags high, followed by set tags low)

  * •

Setting the comparand

  * •

Setting the mask

  * •

Sending the search signal

Notice that setting the comparand and mask require several clock cycles as the
memory is based on 4-byte words and the CAPP can only send one byte per CAPP
clock cycles. Furthermore, the FSM diagram shows a single RECEIVE state that
can be used to receive both the comparand and the mask. Technically, in a
formal FSM diagram, this should be represented by two distinct states,
however, the Verilog code re-uses some logic when implementing these
operations and we wanted the FSM-diagram to reflect this.

As seen above, resetting the tags is, in fact, a two step operations:

  * •

Set Tags High: change SET to high

  * •

Set Tags Low: change SET to low

where SET refers to the SET line shown in Figure 2.

Some operations that appear to be atomic in Figure 5 are in fact composed of
multiple micro-steps, including an "IDLE" (delay) step. For example, the
Search command is in fact composed of three micro-steps:

  * •

SEARCH_1:

    * –

set SEARCH to high

    * –

set delay to 5 clock cycles

  * •

IDLE:

    * –

wait for duration of delay

    * –

go to SEARCH_2

  * •

SEARCH_2:

    * –

set SEARCH to low

where SEARCH refers to the "Perform Search" line in Figure 1. IDLE represents
another case of a "shared" micro-step which is reused by two commands (as was
the case with RECEIVE). Though it is not represented in Figure 5 (in the
interest of clarity), IDLE is in fact used by both the Search and Select
commands.

##  IV Results

The Content Addressable Parallel Processor was successfully implemented on a
TinyFPGA-BX. It was tested both by manual interaction with the serial USB-UART
as well as through a python driver which wraps the protocol in a higher-level
object-oriented API. The python driver file also contains a simple test
program that serves both as a tutorial and a test-suite for the FPGA based
CAPP.

##  V Discussion

Historically, CAPPs have clearly lost out to von Neumann machines. Arguably,
this was because the increased price associated with the more complex memory
cells was too prohibitive, especially in the early days of computing. However,
over the years, manufacturing and designing costs have plummeted while the
difficulty of writing correct parallel software has remains as high as it ever
was. This makes it a good time to reconsider alternative architectures, which,
by exposing powerful primitives such as content addressable read and write
operations should simplify the development of parallel algorithms. We have
therefore revived Caxton Foster’s CAPP research project in the form of an
FPGA-based co-processor, which we think will be able to support new,
interesting and competitive implementation of algorithms in areas such as:
fast lookup tables, neural networks, parallel regex operations, graph
algorithms and more. Our implementation is fully open source and can be found
here [2].

## Acknowledgment

We acknowledge Kartik Thakore of Sharecare Inc. for technical feedback. ER was
funded by a contract from Doc.AI Inc. MA was funded by a contract from Small-
Technology Incubator LLC. PowerPoint templates of basic Logic Gates were
designed by Oliver Mannay and downloaded from: https://www.tes.com/en-
us/teaching-resource/logic-gate-symbols-for-powerpoint-11139006

## References

  * [1] “TinyFPGA-BX.” [Online]. Available: https://github.com/tinyfpga/TinyFPGA-BX
  * [2] “Content-Addressable-Parallel-Processors.” [Online]. Available: https://github.com/asalik13/Content-Addressable-Parallel-Processors
  * [3] C. C. Foster, _Content addressable parallel processors_ , ser. Computer science series. New York: Van Nostrand Reinhold, 1976.
  * [4] D. Shah, E. Hung, C. Wolf, S. Bazanski, D. Gisselquist, and M. Milanovic, “Yosys+nextpnr: An open source framework from verilog to bitstream for commercial fpgas,” in _2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_ , 2019, pp. 1–4.
  * [5] “TinyFPGA BX USB Serial.” [Online]. Available: https://github.com/davidthings/tinyfpga_bx_usbserial