# A Contrastive Cross-Channel Data Augmentation Framework for Aspect-Based
Sentiment Analysis

Bing Wang1 , Liang Ding2, Qihuang Zhong4, Ximing Li1, Dacheng Tao2,3
1College of Computer Science and Technology, Jilin University
2JD Explore Academy 3The University of Sydney 4Wuhan University
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work was done when Bing and Qihuang
were interning at JD Explore Academy. Liang Ding is the corresponding author.

###### Abstract

Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis
task, which focuses on detecting the sentiment polarity towards the aspect in
a sentence. However, it is always sensitive to the multi-aspect challenge,
where features of multiple aspects in a sentence will affect each other. To
mitigate this issue, we design a novel training framework, called Contrastive
Cross-Channel Data Augmentation (C3da), which leverages an in-domain generator
to construct more multi-aspect samples and then boosts the robustness of ABSA
models via contrastive learning on these generated data. In practice, given a
generative pretrained language model and some limited ABSA labeled data, we
first employ some parameter-efficient approaches to perform the in-domain
fine-tuning. Then, the obtained in-domain generator is used to generate the
synthetic sentences from two channels, i.e., Aspect Augmentation Channel and
Polarity Augmentation Channel, which generate the sentence condition on a
given aspect and polarity respectively. Specifically, our C3da performs the
sentence generation in a cross-channel manner to obtain more sentences, and
proposes an Entropy-Minimization Filter to filter low-quality generated
samples. Extensive experiments show that our C3da can outperform those
baselines without any augmentations by about 1% on accuracy and Macro-F1. Code
and data are released in https://github.com/wangbing1416/C3DA.

A Contrastive Cross-Channel Data Augmentation Framework for Aspect-Based
Sentiment Analysis

Bing Wang1††thanks:  Work was done when Bing and Qihuang were interning at JD
Explore Academy. , Liang Ding2††thanks:  Liang Ding is the corresponding
author., Qihuang Zhong4, Ximing Li1, Dacheng Tao2,3 1College of Computer
Science and Technology, Jilin University 2JD Explore Academy 3The University
of Sydney 4Wuhan University<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Sentiment Analysis (SA) is a typical Natural Language Understanding (NLU) task
to predict the sentence-level sentiment polarities Pang and Lee (2007); Liu
(2012). However, since a single review sentence always exists multiple
polarities, Aspect-based Sentiment Analysis (ABSA), a fine-grained sentiment
analysis task Ma et al. (2017); Fan et al. (2018); Sun et al. (2019); Wang et
al. (2020); Li et al. (2021b); Zhang et al. (2022a), is further introduced to
detect the sentiment polarities towards given aspects (entities) in a review
sentence. Gathering a review "while the ambiance and atmosphere were great,
the food and service could have been a lot better.", the sentiment polarities
of "ambiance" and "atmosphere" are Positive and we get Negative polarity for
aspects "food" and "service".

It is crucial to capture the aspect-specific contextual features for an ABSA
model. Unfortunately, as shown in the aforementioned example, the multi-aspect
challenge that there are multiple aspects in a sentence always deteriorates
the model’s generalization, especially when the expressed polarities of
multiple aspect words in a sentence are opposite. Meanwhile, since ABSA is a
low-resource task, it is hard to train a robust model under fewer samples.

Some existing works focused on the multi-aspect challenge, such as Lu et al.
(2011); Hu et al. (2019) in which novel model structures were designed to
capture aspect-specific sentiment information. Additionally, Jiang et al.
(2019) manually collected a large-scale high-quality multi-aspect multi-
sentiment (MAMS) dataset. Undoubtedly, the data-centric MAMS is an effective
approach to tackle the multi-aspect problem, but the human-annotated and non-
expandable limitations still hinder the robust training for ABSA models.
Therefore, given limited labeled data, it is critical to investigate how to
collect more in-domain multi-aspect samples automatically. On the other hand,
capturing the aspect-specific information from multi-aspect sentences is also
necessary for robust ABSA models.

In response to the aforementioned problems, we propose a novel training
framework, namely Contrastive Cross-Channel Data Augmentation (C3da), to
generate more in-domain multi-aspect samples and train robust ABSA models
based on these generated data. Firstly, inspired by successful generative
pretraining models Raffel et al. (2020); Lewis et al. (2020), we employ an
representative pretraining models T5 Raffel et al. (2020) as the multi-aspect
data generator. In practice, due to the limited ABSA labeled data, it is hard
to fine-tuning the entire T5 model effectively. Thus, we further introduce
some parameter-efficient methods, e.g. prompt Lester et al. (2021a), prefix Li
and Liang (2021) and LoRA Hu et al. (2022), to tune the T5 generator. In this
way, a domain -specific generator is obtained and we can apply it to collect
more in-domain multi-aspect samples.

To be more specific, there are two channels to generate expected multi-aspect
samples in our framework, i.e. Aspect Augmentation Channel (AAC) and Polarity
Augmentation Channel (PAC). In the AAC, the generator is encouraged to
generate the synthetic sentence towards the given sentence and aspect, while
in the PAC, the synthetic sentence towards the given sentence and polarity is
obtained. To further boost the generated samples, we attempt to collect the
multi-aspect samples in a cross-channel manner. Specifically, in the first
generation, the source sentence is fed to the double channels and obtain an
aspect-specific sentence and a polarity-inverted sentence respectively. In the
second generation, both generated sentences are injected to the another
channel to get the ultimate sentences. Finally, we propose an Entropy-
Minimization Filter (EMF) to filter some low-quality generated samples. And a
contrastive training objective that can draw away the different aspect’s
embeddings in a sentence is also leveraged to alleviate the multi-aspect
problem and help train the final robust ABSA model.

We conduct sufficient experiments on three popular ABSA datasets, i.e.
Restaurant, Laptop Pontiki et al. (2014) and Twitter Dong et al. (2014), to
prove that our C3da framework can boost the model’s robustness and predictive
performance, and outperform other NLP data augmentation strategies.
Specifically, with the help of our C3da, RoBERTa-based models can achieve
averaged performance improvement 1.28% and 1.62% in terms of accuracy and
macro-F1 respectively, while the improvements of BERT-based models are also
over 0.87% and 1.10%. Some in-depth discussions and case studies are also
executed. Contributions of this paper are threefold:

  * •

We recast the vanilla ABSA training scheme with a data augmentation-based
training framework and a contrastive training objective to tackle the multi-
aspect challenge.

  * •

We design a novel cross-channel data augmentation method based on generative
large-scale pretrained language models to generate high-quality in-domain
multi-aspect samples, which has great potential to benefit other fine-grained
NLU tasks.

  * •

Extensive experiments on three widely-used datasets show the effectiveness of
our C3da.

##  2 Related Work

Figure 1: The overall C3da framework. In our generator, learnable modules can
be optimized in the fine-tuning stage and will also be frozen in the
generation stage, and the PLM encoder-decoder structure is always fixed.

###  2.1 Language Models and Parameter Efficient

Large Pre-trained Language Models (PLMs) are still research cores in various
natural language processing tasks. The PLMs tend to be variants of Transformer
Vaswani et al. (2017), and are trained on massive unlabeled raw sentences
under some linguistic unsupervised objectives. According to the model
structure, these Transformer-based PLMs are roughly divided into (1) Encoder-
only LMs Devlin et al. (2019); Liu et al. (2019) can be utilized to capture
token-level and sentence-level features, (2) Decoder-only LMs Radford et al.
(2018, 2019) tend to design an auto-regressive objective to cater to text
generation tasks and (3) Encoder-Decoder LMs Lewis et al. (2020); Raffel et
al. (2020) need two sentences (source and target) to perform conditional
generation.

The above PLMs always follow the pre-train + fine-tune (FT) paradigm to adapt
the downstream tasks, however, tuning a whole model is a tricky challenge and
is not applicable to low-resource NLP tasks. Therefore, a novel pre-train +
prompt-tune (PT) Liu et al. (2021c) concept is proposed to break this
challenge, PT-based models can also be divided into (1) manual discrete
prompts design human-made templates to adapt different pre-trained objective
Liu et al. (2021a), and (2) soft prompts inject some learnable pseudo tokens
or matrices to frozen PLMs Liu et al. (2021d); Lester et al. (2021b); Li and
Liang (2021); Zhong et al. (2022b).

###  2.2 Aspect-based Sentiment Analysis

Aspect-based Sentiment Analysis (ABSA) is a kind of Text Classification task,
that benefits from a better aspect-aware text representation. Therefore, an
amount of neural network-based models were proposed to break the ABSA
challenge, for example, graph-based models Sun et al. (2019); Wang et al.
(2020); Li et al. (2021b); Zhong et al. (2022a) conduct graph convolutional
operations on dependency trees to encode the semantic information, and
attention-based models Ma et al. (2017); Chen et al. (2017); Fan et al.
(2018); Song et al. (2019); Liu et al. (2021b) focus on interaction between
aspect terms and context tokens, _etc._

In recent years, pre-trained language models changed the situation of ABSA
because they are better text encoders and can capture context features.
Therefore, researchers explored more views to improve ABSA performance, except
that PLM is regarded as an encoder to replace the vanilla Glove + BiLSTM
paradigm. For example, Dai et al. (2021) leverages RoBERTa to re-construct
dependency trees, Yan et al. (2021); Li et al. (2021a); Zhang et al. (2021)
try to integrate various ABSA subtasks into a unified generative framework,
and Zhou et al. (2021); Seoh et al. (2021) combine ABSA with other NLP tasks,
_etc._

###  2.3 Data Augmentation

To improve the scale of training samples, some Data Augmentation (DA) methods
in the NLP community are proposed Zhang et al. (2018); Wei and Zou (2019);
Kobayashi (2018); Wu et al. (2019); Anaby-Tavor et al. (2020); Ding et al.
(2021); Cao et al. (2021); Wang et al. (2022); Zhang et al. (2022b). EDA Wei
and Zou (2019) transforms source sentences with Synonym Replacement
approaches, _etc._ CBERT Wu et al. (2019) introduces a new conditional masked
language model task. Meanwhile, some PLM-based generative augmentation
frameworks Anaby-Tavor et al. (2020); Wang et al. (2022); Zhang et al. (2022b)
also have been explored. Also, recent work reveals the complementarity between
PLMs and classical approaches, e.g., back translation Liu et al. (2021e) and
random initialization Zan et al. (2022), to augment both the accuracy and
generalization.

##  3 Our C3da Framework

#### Task Description of ABSA

Given a group of triplets
$\Omega^{src}=\\{\mathbf{s}_{i},\mathbf{a}_{i},\mathbf{p}_{i}\\}_{i=1}^{N}$, a
sentence $\mathbf{s}_{i}=\\{s_{i1},\cdots,s_{iL}\\}$ and an aspect-span
indicator $\mathbf{a}_{i}\in\\{0,1\\}^{L}$ ($1$ denotes this token is an
aspect, vice versa) will be fed to a trainable model to obtain a hidden
embedding
$\mathbf{h}_{i}=\mathcal{F}(\mathbf{s}_{i},\mathbf{a}_{i},\Theta)\in\mathbb{R}^{L\times
D}$, where $N$, $L$ and $D$ denote the instance scale, sentence length and
hidden embedding size, and $\mathcal{F}(\cdot,\Theta)$ is a prediction model
(_e.g.,_ BERT Devlin et al. (2019)). Then, a ground-truth polarity label
$\mathbf{p}_{i}\in\\{0,1,2\\}$ is utilized to conduct supervised learning to
estimate task-specific parameters $\Theta$.

Table 1: An example of the concatenation operation. $\mathbf{s}$ |  I’ve been here 3 times for lunch and it is one of my favorites in the city.
---|---
$[\mathbf{s},\textit{service}]$ |  I’ve been here 3 times for lunch and it is one of my favorites in the city. $<eos>$ service
$[\mathbf{s},\textit{Negative}]$ |  I’ve been here 3 times for lunch and it is one of my favorites in the city. $<eos>$ so bad

###  3.1 Overall Learning Objective

In this section, we introduce the overall process of our proposed method, and
more details will be further described in Section 3.2 and 3.3, and our C3da
framework is shown in Figure 1.

Our C3da framework consists of two steps: (1) Data Augmentation and (2)
Contrastive Training. We utilize the representative pre-trained T5 model
Raffel et al. (2020) as our generator in the data augmentation stage111Note
that we can employ various generative pretraining models as the generator. T5
is used in this work, since it is widely-used and powerful.. Given a sentence
$\mathbf{s}_{i}$, we expect to generate an another sentence
$\mathbf{\widehat{s}}_{i}=\mathcal{G}(\mathbf{s}_{i};\Pi)$ that expresses the
opposite polarity and has a different aspect from the source sentence, where
$\mathcal{G}(\cdot;\Pi)$ is the T5 generator.

Basically, we conduct the Supervised Classification Training (SCT) on source
sentences and augmented sentences, the objective is as follows:

$\displaystyle\mathcal{L}_{SCT}=\frac{1}{N}\sum\nolimits_{i=1}^{N}$
$\displaystyle\ell_{CE}(\mathbf{h}_{i}\mathbf{W}_{s}+\mathbf{b}_{s},\mathbf{p}_{i})$
$\displaystyle+\alpha$
$\displaystyle\ell_{CE}(\mathbf{h}_{i}^{p}\mathbf{W}_{s}+\mathbf{b}_{s},\mathbf{p}_{i}),$
(1)

$\mathbf{h}_{i}^{p}=\mathcal{F}\left([\mathbf{s}_{i},\mathbf{\widehat{s}}_{i}],\mathbf{a}_{i},\Theta\right),$
(2)

where $[\cdot,\cdot]$ and $\ell_{CE}(\cdot)$ are the concatenate operation and
a Cross-Entropy loss function, $\mathbf{W}_{s}\in\mathbb{R}^{D\times M}$ and
$\alpha$ denote the classification head and a hyper-parameter, respectively.

To obtain more robust performance, we leverage the Contrastive Training (CT)
formulated as the triplet loss Schultz and Joachims (2003) as follows:

$\displaystyle\mathcal{R}_{CT}=\frac{1}{N}\sum\nolimits_{i=1}^{N}max\\{$
$\displaystyle d(\mathbf{h}_{i},\mathbf{h}_{i}^{p})-$ $\displaystyle
d(\mathbf{h}_{i},\mathbf{h}_{i}^{n})+\xi,0\\},$ (3)

where $\mathbf{h}_{i}^{n}$ is the average-pooled sentence representation of
$\mathbf{\widehat{s}}_{i}$ in the sentence
$[\mathbf{s}_{i},\mathbf{\widehat{s}}_{i}]$, $\xi$ is a hyper-parameter called
margin that can control whether an instance should be trained, and
$d(\cdot,\cdot)$ denotes a distance measure function. Specially, we apply a
negative cosine similarity to measure the embedding distance $d(\cdot,\cdot)$.

$d(\mathbf{h}_{i},\mathbf{h}_{i}^{p})=-\frac{\mathbf{h}_{i}\mathbf{h}_{i}^{p}}{\|\mathbf{h}_{i}\|\times\|\mathbf{h}_{i}^{p}\|}.$
(4)

Finally, the overall learning objective can be formulated as:

$\mathcal{L}=\mathcal{L}_{SCT}+\beta\mathcal{R}_{CT},$ (5)

where $\beta$ is a controllable hyper-parameter.

###  3.2 Cross-Channel Data Augmentation

Our motivation is to generate high-quality in-domain multi-aspect samples. In
practice, the cross-channel data augmentation consists of two steps: 1) tune
the pretrained backbone model to adapt our cross-channel sentence generation;
2) generate sentences with opposite polarity and different aspects.

#### Pre-Trained Generator

We employ pre-trained encoder-decoder model T5 as our backbone. ABSA is a low-
resource task, therefore, tuning a whole large language model (_e.g.,_ T5)
with scarce samples will cause an over-fitting problem. Inspired by Lester et
al. (2021a); Wang et al. (2022), we utilize the parameter-efficient strategies
to tune the generation model. In particular, we consider several efficient
tuning methods, including prefix-tuning Li and Liang (2021), prompt-tuning
Lester et al. (2021a), and LoRA Hu et al. (2022). We grid-search the optimal
efficient tuning method for each dataset, detailed comparisons can be found at
Appendix A.

#### Model Fine-Tuning

To make the T5 model capable of generating new sentences conditioned on the
original sentence and given aspect or polarity, we first construct the fine-
tuning data based on the ABSA training set. Concretely, we randomly sample two
instances $\\{\mathbf{s}_{i}\\}$ and
$\\{\mathbf{s}_{j},\mathbf{a}_{j},\mathbf{p}_{j}\\}$ from the source triplets
$\Omega^{src}$. Then, we can tune the T5 model to generate $\mathbf{s}_{j}$
conditioned on $[\mathbf{s}_{i},\mathbf{a}_{j}]$ or
$[\mathbf{s}_{i},\mathbf{p}_{j}]$. As aforementioned, parameter-efficient
strategies are used to avoid over-fitting. Taking the cutting-edge prompt-
tuning Lester et al. (2021a) as an example, the massive parameters in the
generator should be fixed in this stage, and we only tune the parameters of
prompts.

#### Sentence Generation

The sentence generation for data augmentation stage consists of Aspect
Augmentation Channel (AAC) and Polarity Augmentation Channel (PAC).

AAC generates an aspect-specific sentence
$\mathbf{s}_{i}^{a}=\mathcal{G}([\mathbf{s}_{i},\mathbf{\widehat{A}}],\Pi)$222AAC
can generate one or more sentences, we take generating a single sentence as an
example., where $\Pi$ is the trainable parameters of the T5 generator
$\mathcal{G}(\cdot)$. PAC generates a polarity-inverted sentence
$\mathbf{s}_{i}^{p}=\mathcal{G}([\mathbf{s}_{i},\mathbf{\widehat{P}}],\Pi)$,
where $\mathbf{\widehat{A}}$ is a random-sampled in-domain aspect and
$\mathbf{\widehat{P}}$ is a group of seed spans that have the inverted
polarity from $\mathbf{s}_{i}$, such as so good. Then, these sentences will be
fed into the another channel to obtain
$\mathbf{s}_{i}^{pa}=\mathcal{G}([\mathbf{s}_{i}^{p},\mathbf{\widehat{A}}],\Pi)$
and
$\mathbf{s}_{i}^{ap}=\mathcal{G}([\mathbf{s}_{i}^{a},\mathbf{\widehat{P}}],\Pi)$
that can satisfy our motivation. We give an example of the concatenation
operation $[\mathbf{s},\cdot]$ in Table 1.

Note that for sentence generation, both the generator backbone and prompts
will be frozen. The detailed Cross-Channel Data Augmentation method is shown
in Algorithm 1.

Algorithm 1 Cross-Channel Data Augmentation.

1:Source triplets
$\Omega^{src}=\\{\mathbf{s}_{i},\mathbf{a}_{i},\mathbf{p}_{i}\\}_{i=1}^{N}$;
Initial pre-trained T5 generator with soft prompt
$\mathcal{G}(\cdot,\Pi^{0})$; Pre-defined aspect set $\mathcal{A}$; Fine-tune
iterations $\tau$.

2:Augmented sentences $\Omega^{aug}=\\{\mathbf{\widehat{s}}_{i}\\}_{i=1}^{N}$.

3:for $iter$ in $\tau$ do $\triangleright$ Stage I: Model Fine-tuning

4:
$\\{\mathbf{s}_{i}\\},\\{\mathbf{s}_{j},\mathbf{a}_{j},\mathbf{p}_{j}\\}\leftarrow
Sample(\Omega^{src})$

5: $\nabla^{a}\leftarrow
loss\left(\mathbf{s}_{j},\mathcal{G}([\mathbf{s}_{i},\mathbf{a}_{j}],\Pi^{iter})\right)$

6: $\nabla^{p}\leftarrow
loss\left(\mathbf{s}_{j},\mathcal{G}(\left[\mathbf{s}_{i},\mathbf{p}_{j}\right],\Pi^{iter})\right)$

7: $\Pi^{iter+1}\leftarrow Train(\Pi^{iter},\nabla^{a},\nabla^{p})$

8:end for

9:$\Pi\leftarrow\Pi^{\tau}$

10:for $\\{\mathbf{s}_{i},\mathbf{a}_{i},\mathbf{p}_{i}\\}$ in $\Omega^{src}$
do

11: $\triangleright$ Stage II: Sentence Generation

12: $\mathbf{\widehat{A}}\leftarrow Sample(\mathcal{A})$

13: $\mathbf{\widehat{P}}\leftarrow-\mathbf{p}_{i}$ $\triangleright$
$-\mathbf{p}$ denotes opposite polarity from $\mathbf{p}$

14:
$\mathbf{s}_{i}^{a}\leftarrow\mathcal{G}\left([\mathbf{s}_{i},\mathbf{\widehat{A}}],\Pi\right)$,
$\mathbf{s}_{i}^{p}\leftarrow\mathcal{G}\left([\mathbf{s}_{i},\mathbf{\widehat{P}}],\Pi\right)$

15:
$\mathbf{s}_{i}^{pa}\leftarrow\mathcal{G}\left([\mathbf{s}_{i}^{p},\mathbf{\widehat{A}}],\Pi\right)$,
$\mathbf{s}_{i}^{ap}\leftarrow\mathcal{G}\left([\mathbf{s}_{i}^{a},\mathbf{\widehat{P}}],\Pi\right)$

16:
$\Omega^{aug}\leftarrow\Omega^{aug}+Filter(\mathbf{s}_{i}^{a},\mathbf{s}_{i}^{p},\mathbf{s}_{i}^{pa},\mathbf{s}_{i}^{ap})$

17:end for

18:return $\Omega^{aug}$;

###  3.3 Entropy-Minimization Filtering

Cross-Channel Data Augmentation generates four sentences for each source
sentence, and we can leverage these sentences to train a prediction model
$\mathcal{F}(\cdot,\Theta)$. However, it is necessary to filter out some low-
quality samples that carry noisy (or difficult) information, which can be
estimated by language modeling Moore and Lewis (2010) and norm of word
embedding Liu et al. (2020).

To ensure the certainty of synthetic sentences, we design an Entropy-
Minimization Filter (EMF) to filter those noisy sentences. We establish a
hypothesis that noisy sentences always express an ambiguous polarity such that
our prediction model gives a more smooth polarity probability distribution. To
be specific, the information entropy of prediction distribution as Equation 6
is an appropriate measure. A sentence with larger prediction entropy should be
filtered, and a hyper-parameter $k$ can control how many augmented sentences
are selected to participate in training for each sample.

$\mathcal{H}_{i}=-\mathbb{E}_{\mathbf{y}_{i}}\big{[}\log_{2}p(\mathbf{y}_{i})\big{]}=-p(\mathbf{y}_{i})\log_{2}p(\mathbf{y}_{i}),$
(6)

$p(\mathbf{y}_{i})=softmax(\mathbf{h}_{i}\mathbf{W}_{s}+\mathbf{b}_{s}).$ (7)

##  4 Experiment

Table 2: Empirical results of C3da and other cutting-edge data augmentation methods. The best scores of each metric are indicated in bold, and $\uparrow$ denotes the improvement over BERT-base and RoBERTa-base baselines. Model | Restaurant | Laptop | Twitter
---|---|---|---
acc |  F1 |  acc |  F1 |  acc |  F1
BERT-base |  86.31 |  80.22 |  79.66 |  76.11 |  76.50 |  75.23
\+ EDA Wei and Zou (2019) |  86.42 |  79.63 |  79.59 |  75.79 |  76.26 |  75.16
\+ BT Fan et al. (2021) |  86.47 |  80.29 |  79.67 |  76.35 |  76.63 |  75.47
\+ CBERT Wu et al. (2019) |  86.27 |  80.00 |  79.83 |  76.12 |  76.44 |  75.36
\+ SCon Liang et al. (2021) |  86.51 |  80.55 |  80.23 |  76.48 |  - |  -
\+ C3da (Ours) |  86.93↑0.62 |  81.23↑1.01 |  80.61↑0.95 |  77.11↑1.00 |  77.55↑1.05 |  76.53↑1.30
RoBERTa-base |  86.38 |  80.29 |  80.10 |  76.24 |  76.63 |  75.37
\+ EDA Wei and Zou (2019) |  86.43 |  80.21 |  80.38 |  76.59 |  76.47 |  75.36
\+ BT Fan et al. (2021) |  86.50 |  80.59 |  80.22 |  76.73 |  76.59 |  75.47
\+ CBERT Wu et al. (2019) |  86.77 |  80.51 |  80.54 |  76.57 |  76.73 |  75.37
\+ C3da (Ours) |  87.11↑0.73 |  81.63↑1.34 |  81.83↑1.73 |  78.46↑2.22 |  78.31↑1.38 |  76.67↑1.30

###  4.1 Experimental Settings and Baselines

We conduct our experiments on three public datasets, i.e. Restaurant, Laptop
from SemEval 2014 ABSA task Pontiki et al. (2014) and Twitter Dong et al.
(2014). In practice, we run the experiments over 5 random seeds and report the
average values of accuracy (acc) and Macro-F1 (F1) to avoid stochasticity.

Two widely-used pretraining models, i.e. BERT Devlin et al. (2019) and RoBERTa
Liu et al. (2019), are regarded as our prediction backbone, and we compare our
framework C3da with the following four data augmentation methods:

  * •

EDA Wei and Zou (2019) is a token-level transformation method. It follows some
patterns such as Synonym Replacement, Random Insertion, _etc._ For the ABSA
task, we deliberately avoid corrupting the aspect words.

  * •

Back Translation (BT) Sennrich et al. (2016) is a sentence-level augmentation
method, which translates a sentence to another language and translates it back
to the original language. We follow Fan et al. (2021) that translates English
to French and translates it back to English.

  * •

CBERT Wu et al. (2019) fully excavates the power of mask language model (MLM)
objective to replace some tokens, so it is also a token-level replacement
approach.

  * •

SCon Liang et al. (2021) design aspect-invariant/-dependent data augmentation
for ABSA and deploy a supervised contrastive learning objective. We reproduce
it according to their released code and default best settings.

###  4.2 More Empirical Details

We implement our experiment by following the released ABSA
Pytorch333https://github.com/songyouwei/ABSA-PyTorch repository and our pre-
trained models, such as bert-base-uncased, roberta-base and t5-base, are from
HuggingFace444https://huggingface.co/models. Meanwhile, we utilize a flexible
toolkit OpenDelta555https://github.com/thunlp/OpenDelta to adapt various
parameter-efficient methods to our T5 generator.

In the data augmentation stage, we leverage an Adafactor optimizer. A T5
generator is fine-tuned for 100 epochs, and batch size is fixed to 16. As
training a prediction model BERT or RoBERTa, we adapt an Adam optimizer with a
learning rate of $2\times 10^{-5}$, and the dropout rate is set to 0.3.
Additionally, we train the final prediction model for 15 epochs with the batch
size as 16.

A single NVIDIA A100 is used to conduct our all experiments. The model fine-
tuning and sentence generation stages will generally spend 2 - 3 hours, and it
spends about 1 hour to train a prediction model.

Figure 2: Sensitivity Analysis of the contrastive training margin $\xi$. The
ribbon upper-side and below-side denote the maximum and minimum performance in
5 seeds respectively, and the centerline is the average value. Figure 3:
Sensitivity Analysis of the number of selected sentences $k$ in EMF.

###  4.3 Main Result

Table 2 shows our main experimental results. To be fair, for each data
augmentation method, we generate 1 synthetic sample per sentence. Overall, our
C3da framework consistently and significantly improve the preformance of both
BERT-based and RoBERTa-based models, e.g. the result of Restaurant even
outperforms the RoBERTa-base baseline 2.22% in term of F1. Moreover, we
observe that the performance order of various data augmentation methods is
C3da > CBERT > BT > EDA, which indicates: (1) EDA semantic-regardlessly
removes or inserts tokens, which will inject more noise into final prediction
models, thus leading to the poor performance. (2) Comparing BT and CBERT, BT
always achieves higher F1, and CBERT will get more advantageous accuracy.
Since BT is a sentence-level method, it focuses on how to transform a source
sentence into a form-different and semantic-reserved sentence, but neglects
the useful aspect information. On the contrary, CBERT will only change some
tokens, which is a semantic-reversed and aspect-reserved approach. Table 3
summarizes the characteristics of some augmentation methods.

Turn to compare two metrics, another finding is that C3da has more significant
performance in term of Macro-F1. Proverbially, since Macro-F1 is a metric to
measure the generalization of a model, the higher improvements on Macro-F1
prove that our framework can effectively improve the model’s robustness in
this view. Concurrently, the effect of C3da is better on Restaurant and
Twitter. The difference between these datasets is that Laptop is a small
dataset, indicating that our C3da is sensitive to the data scale and C3da may
have better improvement if there is a larger-scale dataset.

Table 3: A comparison of various data augmentation methods. Semantic-R. and Aspect-R. denote semantic reserved methods and aspect reserved methods, and T.- / S.- discriminates token-level and sentence-level methods respectively. Methods |  Semantic-R. |  Aspect-R. |  T.- / S.-
---|---|---|---
EDA |  $\times$ |  $\checkmark$ |  T.
BT |  $\checkmark$ |  $\times$ |  S.
CBERT |  $\checkmark$ |  $\checkmark$ |  T.
C3da |  $\checkmark$ |  $\checkmark$ |  S.

###  4.4 Ablation Study

Table 4: Case study. The aspect words and corresponding polarities are noted as red color and blue color, respectively. case | the  falafal was rather over cooked and dried but the  chicken was fine.
---|---
Aspect & Polarity | falafal | chicken
|  Positive |  Neutral |  Negative |  Positive |  Neutral |  Negative
w/o C3da |  0.098 |  0.109 |  0.793 |  0.643 |  0.047 |  0.310
with C3da |  0.071 |  0.092 |  0.837 |  0.698 |  0.076 |  0.226
case | good  food but the  service was dreadful!
Aspect & Polarity | food | service
|  Positive |  Neutral |  Negative |  Positive |  Neutral |  Negative
w/o C3da |  0.502 |  0.173 |  0.325 |  0.191 |  0.236 |  0.573
with C3da |  0.589 |  0.211 |  0.200 |  0.177 |  0.186 |  0.637

The core modules in our framework C3da are Data Augmentation (DA), Contrastive
Learning (CL), and Entropy-Minimization Filter (EMF). In this section, we
perform ablation study to investigate the effect of these modules, and the
results are listed in Table 5. BERT-base is regarded as the prediction model
in this study, and w/o denotes without some modules.

Firstly, w/o DA & CL removes our data augmentation framework, it is equivalent
to a vanilla BERT-base-only prediction model. As shown in Table 5, w/o DA & CL
always presents poor performance, even decrease by about 1% on Laptop’s acc
and F1 and Restaurant’s F1. It proves our C3da framework imports more high-
quality data items and is effective to boost the performance. Then, w/o CL
remains augmented sentences and revokes the contrastive training objective
$\mathcal{R}_{CL}$. Overall, this objective improves the results by about 0.3%
on each dataset, conforming that contrastive learning can powerfully optimize
the aspect-specific features. Finally, w/o EMF conducts both supervised
training and contrastive training on all generated sentences. Intuitively,
more samples can consolidate the predictive power of a classification model,
but some emotionless and illogical sentences also carry plenty of noise. As we
speculate, w/o EMF degrades the model’s performance by 0.5% - 1.0%, the
outcome on Restaurant’s accuracy is even worse than the result without DA &
CL. More sensitivity studies about EMF will be shown in Section 4.5.

Additionally, we also investigate the performance of generating sentences with
only AAC and only PAC settings and report their results in Table 5. We show
that both them are beneficial, where only AAC slightly outperforms that of the
only PAC, demonstrating that the multi-aspect data are more crucial than the
multi-polarity data.

Table 5: Ablation study of C3da. The prediction model is BERT-base in this study. Model |  Restaurant |  Laptop |  Twitter
---|---|---|---
| acc
C3da |  86.93 |  80.61 |  77.08
w/o DA & CL |  86.31 |  79.66 |  76.50
w/o CL |  86.69 |  80.35 |  76.87
w/o EMF |  86.45 |  79.84 |  76.50
only AAC |  86.63 |  80.10 |  77.16
only PAC |  86.44 |  79.94 |  77.19
Model | F1
C3da |  81.23 |  77.11 |  75.76
w/o DA & CL |  80.22 |  76.11 |  75.23
w/o CL |  81.00 |  76.88 |  75.63
w/o EMF |  80.21 |  76.45 |  75.17
only AAC |  80.61 |  76.69 |  75.93
only PAC |  80.24 |  76.17 |  75.91

###  4.5 Sensitivity Analysis

As mentioned above, we employ 4 hyper-parameters in our C3da framework, and
$\xi$ and $k$ are more crucial. To investigate the influence of both hyper-
parameters, we further conduct sensitivity analysis in this section.
Additionally, we fix $\alpha$ and $\beta$ to 0.5 and 2.0, respectively.
Limited by space, their sensitivity analysis will be described and analyzed in
Appendix B.

#### Contrastive Training Margin $\xi$

Margin $\xi$ in Equation 3.1 is an important hyper-parameter to control the
model’s distinctive ability to discriminate negative samples and positive
samples. Under a little margin, the prediction model will indolently balance
$d(\mathbf{h}_{i},\mathbf{h}_{i}^{p})$ and
$d(\mathbf{h}_{i},\mathbf{h}_{i}^{n})$ to cause under-optimization, while a
large margin makes a model converge hard.

To investigate the impact of the margin $\xi$, we implement experiments on
Restaurant and Laptop, and range $\xi$ from 0.1 to 0.6. As shown in Figure 2,
our C3da always gets the best performance when $\xi$ is 0.3, and as it expands
or shrinks, the results consistently show a downward trend.

#### Number of Selected Sentences $k$ in EMF

As the description in Section 4.4, $k$ in EMF is a key hyper-parameter to
decide to introduce more useful sentences or more noisy information, we select
$k$ sentences for each source sentence with the entropy-minimization filter
method.

We evaluate C3da on $k\in\\{1,2,3,4\\}$ and show the results in Figure 3. The
best performance invariably appears in $k=2$ or $k=1$. Meanwhile, the
performance decreases significantly when the value of $k$ is 3, 4.

###  4.6 Case Study

C3da is a DA-based framework, and our purpose is to break the multi-aspect
challenge. Therefore, we implement a case study to evaluate whether C3da can
solve this problem, and more cases with different DA methods and cases with
varying fine-tuning epochs will be shown in Appendix C.

We select two samples from the Restaurant test set, and these two sentences
all have two aspects with different sentiment polarities. We will compare the
performance of our C3da on their polarity prediction distribution. To observe
the Table 4, we summarize that (1) C3da can consistently and significantly
enhance the prediction model’s robustness, such as the Positive probability of
the aspect food improve about 0.077. Therefore, our framework can correct the
polarity prediction when a sentence expresses an ambiguous polarity. (2) When
a sentence is short, the interaction between two aspects with contradictory
polarities will be more intimate. In this scenario, sentences will express
more ambiguous sentiment and the effectiveness of our framework will be more
significant.

###  4.7 Computing Cost Analysis

Our method will prolong each sentence, longer sentences will undoubtedly
increase computing costs. Therefore, we attempt to compare the computing time
of our framework and other augmentation methods. In the data preparing stage,
C3da will fine-tune a generator and inference on the training set, it will
spend 2 - 3 hours, and it is a once-for-all process. In the prediction
training stage, we find that EDA, Back Translation and CBERT do not change the
sentence length, their prediction time cost $0.98\times\sim 1.05\times$
(without any augmentation is $1\times$). By comparison, C3da spends
$1.14\times$ time cost, which is completely acceptable.

##  5 Conclusion

In our work, we focus on solving the multi-aspect problem in the ABSA task. To
address this challenge, we propose a data-centric training framework and
design a novel C3da method to implement data augmentation. The proposed
framework conduct both supervised training and contrastive training on
augmented samples. The key idea of C3da is to generate some sentences that
express the opposite sentiment polarity from the source sentences. Extensive
experiments on widely-used benchmarks demonstrate that our framework can
effectively improve the prediction model’s robustness and predictive
performance. In the future, it will be interesting to apply our C3da framework
to improve other fine-grained natural language understanding tasks.

## Acknowledgements

We are grateful to the anonymous reviewers and the area chair for their
insightful comments and suggestions. Bing Wang and Ximing Li were supported by
the project from the National Natural Science Foundation of China (NSFC)
(No.62006094).

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue! In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_ , pages 7383–7390.
  * Cao et al. (2021) Yu Cao, Liang Ding, Zhiliang Tian, and Meng Fang. 2021.  Towards efficiently diversifying dialogue generation via embedding augmentation.  _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7443–7447.
  * Chen et al. (2017) Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017.  Recurrent attention network on memory for aspect sentiment analysis.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 452–461.
  * Dai et al. (2021) Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, and Xipeng Qiu. 2021.  Does syntax matter? A strong baseline for aspect-based sentiment analysis with roberta.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1816–1829.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186.
  * Ding et al. (2021) Liang Ding, Di Wu, and Dacheng Tao. 2021.  Improving neural machine translation by bidirectional training.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 3278–3284, Online and Punta Cana, Dominican Republic.
  * Dong et al. (2014) Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014.  Adaptive recursive neural network for target-dependent twitter sentiment classification.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 49–54.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021.  Beyond english-centric multilingual machine translation.  _Journal of Machine Learning Research_ , 22:107:1–107:48.
  * Fan et al. (2018) Feifan Fan, Yansong Feng, and Dongyan Zhao. 2018.  Multi-grained attention network for aspect-level sentiment classification.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 3433–3442.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The International Conference on Learning Representations_.
  * Hu et al. (2019) Mengting Hu, Shiwan Zhao, Li Zhang, Keke Cai, Zhong Su, Renhong Cheng, and Xiaowei Shen. 2019.  CAN: constrained attention networks for multi-aspect sentiment analysis.  In _Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing_ , pages 4600–4609.
  * Jain et al. (2016) Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016.  Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications.  In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 935–944.
  * Jiang et al. (2019) Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019.  A challenge dataset and effective models for aspect-based sentiment analysis.  In _Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing_ , pages 6279–6284. Association for Computational Linguistics.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 452–457, New Orleans, Louisiana.
  * Lester et al. (2021a) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021a.  The power of scale for parameter-efficient prompt tuning.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Lester et al. (2021b) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021b.  The power of scale for parameter-efficient prompt tuning.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Li et al. (2021a) Chengxi Li, Feiyu Gao, Jiajun Bu, Lu Xu, Xiang Chen, Yu Gu, Zirui Shao, Qi Zheng, Ningyu Zhang, Yongpan Wang, and Zhi Yu. 2021a.  Sentiprompt: Sentiment knowledge enhanced prompt-tuning for aspect-based sentiment analysis.  _arXiv preprint_ , abs/2109.08306.
  * Li et al. (2021b) Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, and Eduard H. Hovy. 2021b.  Dual graph convolutional networks for aspect-based sentiment analysis.  In _Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , pages 6319–6329.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , pages 4582–4597.
  * Liang et al. (2021) Bin Liang, Wangda Luo, Xiang Li, Lin Gui, Min Yang, Xiaoqi Yu, and Ruifeng Xu. 2021\.  Enhancing aspect-based sentiment analysis with supervised contrastive learning.  In _ACM International Conference on Information and Knowledge Management,_ , pages 3242–3247.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017\.  Focal loss for dense object detection.  In _IEEE International Conference on Computer Vision_ , pages 2980–2988.
  * Liu (2012) Bing Liu. 2012.  _Sentiment Analysis and Opinion Mining_.  Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
  * Liu et al. (2021a) Jian Liu, Zhiyang Teng, Leyang Cui, Hanmeng Liu, and Yue Zhang. 2021a.  Solving aspect category sentiment analysis as a text generation task.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 4406–4416.
  * Liu et al. (2021b) Juhua Liu, Qihuang Zhong, Liang Ding, Hua Jin, Bo Du, and Dacheng Tao. 2021b.  Unified instance and knowledge alignment pretraining for aspect-based sentiment analysis.  _arXiv preprint_.
  * Liu et al. (2021c) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021c.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint_ , abs/2107.13586.
  * Liu et al. (2021d) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021d.  GPT understands, too.  _arXiv preprint_ , abs/2103.10385.
  * Liu et al. (2020) Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020.  Norm-based curriculum learning for neural machine translation.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 427–436, Online.
  * Liu et al. (2021e) Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2021e.  On the complementarity between pre-training and back-translation for neural machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP_ , pages 2900–2907.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _arXiv preprint_ , abs/1907.11692.
  * Lu et al. (2011) Bin Lu, Myle Ott, Claire Cardie, and Benjamin K. Tsou. 2011.  Multi-aspect sentiment analysis with topic models.  In _IEEE International Conference on Data Mining workshops_ , pages 81–88.
  * Ma et al. (2017) Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017.  Interactive attention networks for aspect-level sentiment classification.  In _International Joint Conference on Artificial Intelligence_ , pages 4068–4074.
  * Moore and Lewis (2010) Robert C. Moore and William Lewis. 2010.  Intelligent selection of language model training data.  In _Proceedings of the ACL Conference Short Papers_ , pages 220–224, Uppsala, Sweden.
  * Pang and Lee (2007) Bo Pang and Lillian Lee. 2007.  Opinion mining and sentiment analysis.  _Found. Trends Inf. Retr._ , 2(1-2):1–135.
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  Semeval-2014 task 4: Aspect based sentiment analysis.  In _International Workshop on Semantic Evaluation_ , pages 27–35.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _OpenAI blog_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Schultz and Joachims (2003) Matthew Schultz and Thorsten Joachims. 2003.  Learning a distance metric from relative comparisons.  _Advances in neural information processing systems_ , 16.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 86–96, Berlin, Germany.
  * Seoh et al. (2021) Ronald Seoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021.  Open aspect target sentiment classification with natural language prompts.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 6311–6322.
  * Song et al. (2019) Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019.  Attentional encoder network for targeted sentiment classification.  _arXiv preprint_ , abs/1902.09314.
  * Sun et al. (2019) Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2019.  Aspect-level sentiment analysis via convolution over dependency tree.  In _Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing_ , pages 5678–5687.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Annual Conference on Neural Information Processing Systems_ , pages 5998–6008.
  * Wang et al. (2020) Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020.  Relational graph attention network for aspect-based sentiment analysis.  In _Annual Meeting of the Association for Computational Linguistics_ , pages 3229–3238.
  * Wang et al. (2022) Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. 2022.  Promda: Prompt-based data augmentation for low-resource NLU tasks.  _arXiv preprint_ , abs/2202.12499.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  In _Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing_ , pages 6381–6387.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional BERT contextual augmentation.  In _International Conference on Computational Science_ , volume 11539 of _Lecture Notes in Computer Science_ , pages 84–95.
  * Yan et al. (2021) Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021.  A unified generative framework for aspect-based sentiment analysis.  In _Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , pages 2416–2429.
  * Zan et al. (2022) Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022.  On the complementarity between pre-training and random-initialization for resource-rich machine translation.  In _International Conference on Computational Linguistics (COLING)_.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021.  Towards generative aspect-based sentiment analysis.  In _Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , pages 504–510.
  * Zhang et al. (2022a) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022a.  A survey on aspect-based sentiment analysis: Tasks, methods, and challenges.  _arXiv preprint_ , abs/2203.01054.
  * Zhang et al. (2022b) Zheng Zhang, Liang Ding, Dazhao Cheng, Xuebo Liu, Min Zhang, and Dacheng Tao. 2022b.  Bliss: Robust sequence-to-sequence learning via self-supervised input representation.  _arXiv preprint_.
  * Zhong et al. (2022a) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Hua Jin, and Dacheng Tao. 2022a.  Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis.  _arXiv preprint_.
  * Zhong et al. (2022b) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022b.  PANDA: prompt transfer meets knowledge distillation for efficient model adaptation.  _CoRR_ , abs/2208.10160.
  * Zhou et al. (2021) Deyu Zhou, Jianan Wang, Linhai Zhang, and Yulan He. 2021.  Implicit sentiment analysis with event-centered text representation.  In _Conference on Empirical Methods in Natural Language Processing_ , pages 6884–6893.

Table 6: Case study. The upper table compares results of different data augmentation methods, the bold spans is changed part; The below table shows some generated sentences with C3da. Method |  Examples [dataset, polarity]
---|---
source |  not only was the  food outstanding, but the little  ’perks’ were great. [Restaurant, Positive]
No-tuning |  the  kitchen is very well equipped with all the  equipment you need to cook up a storm.
Full-Tuning |  The  food is okay and the  prices here are mediocre.
Prefix-tuning |  The  kitchen is a little small, but the  food is good.
Prompt-tuning |  The  kitchen is a little dated, but the  food is good.
LoRA |  The  seats are a bit cramped, but the  food was delicious.
source |  it is easy to  start up and does not overheat as much as other laptops. [Laptop, Positive]
No-tuning |  a  laptop that is easy to start up and does not overheat as much as other laptops.
Full-Tuning |  I also purchased  iWork to go with it which has programs for word processing, spreadsheets, and presentations ( similar to Microsoft Office ).
Prefix-tuning |  The  battery isn’t very strong, but it is very light.
Prompt-tuning |  but the  screen is so dark, and the  service center is a little numb.
LoRA |  The  software is very easy to use, the  start up is very fast, the  graphics are fantastic.

##  Appendix A Various Parameter-Efficient Methods Result

We adapt three parameter efficient methods for our T5 generator:

  * •

Prefix-tuning Li and Liang (2021) fixes PLMs parameters, and inject an
additional trainable prefix tokens for each transformer hidden layer. We set
the number of prefix tokens to default 6.

  * •

Prompt-tuning Lester et al. (2021b) only prepends a prompt family to the input
embedding layer. The length of the soft tokens is 100 in our experiments.

  * •

Low-Rank Adaptation (LoRA) Hu et al. (2022) trains a low-rank matrix to
lightweight the PLMs. To be specific, we fix the hyper-parameter rank $r$ to 8
and cancel the dropout operation ($dropout=0$).

  * •

Full-tuning fine-tunes all parameters in the T5 backbone.

  * •

No-tuning directly generates sentences with a pre-trained T5 model without
fine-tuning.

Figure 4: Convergence Analysis. Table 7: Convergence loss. "-T." is short for "-Tuning". Method |  Full-T. |  Prefix-T. |  Prompt-T. |  LoRA
---|---|---|---|---
Loss |  0.05102 |  0.70009 |  0.69182 |  0.46218
Figure 5: Sensitivity Analysis of the objective coefficient $\alpha$ in
supervised classification training. Figure 6: Sensitivity Analysis of the
contrastive training objective coefficient $\beta$ in our overall objective.

###  A.1 Case Study

More instances from different fine-tuning methods are listed in Table 6, the
domains and polarities are attached to the source sentences. Under our
observation, we found some phenomena: (1) the sentences from Full-tuning are
always longer, and Full-tuning habitually outputs the same sentences, which
proves that Full-tuning will cause an over-fit problem by conjecture. (2)
three parameter efficient methods consistently generate multi-aspect samples,
however, it is doubtful whether more aspects in a sentence will promote the
model’s robustness more significantly. (3) most of the generated sentences can
achieve our motivation (express an opposite polarity).

###  A.2 Convergence Analysis

We analyze the fine-tuning loss for those aforementioned fine-tuning methods
(without No-tuning) with T5ForConditionGeneration.loss interface in Hugging
Face on Restaurant. Figure 4 shows the loss curve, we adapt a percentile
filter with 20 pts to conduct the curve smooth, and the losses of ultimate
convergence are listed in Table 7. Obviously, Full-tuning will achieve a lower
loss, because all parameters are tuned, and the scale of parameters is far
greater than the number of data items, it is a precursor of an over-fitting
problem. By comparing three fine-tuning methods, we prefer a method with lower
convergence loss, Prefix-tuning and Prompt-tuning are indistinguishable,
however, LoRA obtains an obvious superiority.

##  Appendix B More Sensitivity Analysis

Objective coefficients $\alpha$ and $\beta$ are also controllable hyper-
parameters, so we conduct sensitivity analysis on them. Our experimental
results are shown in Figure 5 and Figure 6, the prediction model is
consistently BERT-base and
$\alpha\in\\{0.1,0.3,0.5,1,2\\},\beta\in\\{0.5,1,2,5,10\\}$. According to the
figures, the best performance always when $\alpha=0.3$ or $0.5$ and $\beta=1$
or $2$.

##  Appendix C More Case Studies

#### Different DA methods

We investigate different data augmentation frameworks from the view of the
model’s prediction results in Section 4.3. We will also list an example to
compare these DA baselines and throw more generated sentences from our C3da in
Table 8. The example in the upper table is from Restaurant, and we adapt LoRA
to the T5 generator, and more examples of C3da below are from Restaurant and
Laptop.

We summarize that EDA deletes "my", transforms "co-workers" to its synonym
"workers" and randomly inserts a comma to an unreasonable position. This
method adds some corruptions without any semantic relations, and it is
possible to generate an illogical sample; Compared to EDA, CBERT is also a
token-level method, but a semantic-reserved sentence is consistently created;
Back Translation is a sentence-level method, and a correct sentence can be
generated. However, it erases the aspect words for ABSA task. By comparison,
our C3da not only reserves the origin semantic information but also introduces
some high-quality in-domain corpus-aware information and adversarial signals.
We also observe that most generated sentences from C3da exist two or more
aspects, and this phenomenon is widespread in several datasets.

Table 8: Case study. The upper table compares results of different data
augmentation methods, the blue spans is changed part; The below table shows
some generated sentences with C3da.

DA methods |  examples
---|---
source |  my co-workers had to wait almost an hour for delivery.
EDA |  workers had to, wait an hour for delivery.
B.T. |  my colleague waited nearly an hour to deliver it.
CBERT |  my  mother had to wait  half an hour for delivery.
C3da |  my co-workers had to wait almost an hour for delivery.  but the perks are great, and you get to enjoy all the perks.

source sentences |  C3da generated sentences
---|---
if you love seafood, you would love this place! |  the  sauce was good, but the  meats were a little fatty.
we had a girls, night dinner here for restaurant week. |  try the  samosas at a reasonable price.
it is easy to start up and does not overheat as much as other laptops. |  but the  screen is so dark, and the  service center is a little numb.

#### Cases with Varying Fine-tuning Epochs $\tau$

We also explore the generation results with varying fine-tuning epochs $\tau$.
As shown in Table 9, given a source sentence “but the staff was so horrible to
us.“, the T5 generator tends to generate an irrelative sentence when we have
not yet fine-tuned it. In the middle of the fine-tuning stage, the generator
is inclined to retell the source sentences or generate some irrational
sentences. Finally, our generation model will eventually converge when
$\tau=80$.

Table 9: Case study. Given a source sentence “but the staff was so horrible to us.“, we can observe the trend of its augmented sentence with fine-tune epochs $\tau$. epochs $\tau$ |  sentence
---|---
source |  but the staff was so horrible to us.
0 |  i cried out for the suffragellah woman.
20 |  the staff is so horrible to us.
40 |  the as-is, the service isn’t great.
60 |  the food is very good.
80 |  the food was great, the service was excellent.

##  Appendix D Instance-level Loss Re-weight for Long-tail Aspects

There is an another problem when we implement sentence generation, our
generated sentences in one dataset often fit to few aspects, such as food in
Restaurant. To investigate this issue, we show the aspect information for
three public ABSA datasets in Table 10, and we found that food in Restaurant
actually maintain maximum frequency. Similarly, service, battery and screen
_etc._ report this long-tail distribution problem. And most of aspects in
Twitter is [UNK], it means that extensive aspect information will be unusable,
this causes some models don’t work, especially the attention-based models.

Similar to focal loss Lin et al. (2017), we adapt an instance-level loss re-
weight method to our fine-tuning stage. Inspired by Jain et al. (2016), we
multiple a multiplier to each instance’s objective, the multiplier is shown in
Equation (8).

$\Delta_{j}\ =\ \frac{1}{1+Ce^{-A\log(M_{j}+B)}},$ (8)

$C\ =\ (\log M-1)(B+1)^{A},$ (9)

where $M_{j}$, $M$ are the number of data points annotated with aspect $j$,
and aspect items, respectively.

Table 10: The aspect words and its frequency statistic of three public datasets. We only show the top-10 aspect words (with the T5 tokenizer) and hide some special tokens such as ’a’, ’i’, _etc._ dataset |  aspect words: frequency
---|---
Restaurant |  food: 419, [UNK]: 404, service: 204, wait: 94, menu: 80, dinner: 75, wine: 72, staff: 68 , pizza: 65, place: 60
Laptop |  [UNK]:267, battery: 97, screen: 81, use: 58, life: 56, windows: 55, price: 55, software: 54 , keyboard: 54, drive: 53
Twitter |  [UNK]: 5328, spear: 894, bri: 887, ney: 887, lo: 394, han: 393, lind: 388, say: 388 , in: 370, ry: 360