An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification
Zhuowei Chen Lianxi Wang Yuben Wu Xinfeng Liao Yujia Tian Junyang Zhong

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
    Mingze Gao^1,2,3,†  Jingyu Liu^2   Mingda Li^2  Jiangtao Xie^4   Qingbin Liu^2 
   Bo Zhao^2   Xi Chen^2   Hui Xiong^1,3, * 

^1 The Hong Kong University of Science and Technology (Guangzhou), China

      ^2Tencent PCG

     ^3The Hong Kong University of Science and Technology, China

    ^4 Dalian University of Technology, China

    Received 16 July 2024; accepted 04 September 2024
=================================================================================================================================================================================================================================================================================================================================================













§ ABSTRACT 



Sentiment classification (SC) often suffers from low-resource challenges such as domain-specific contexts, imbalanced label distributions, and few-shot scenarios. The potential of the diffusion language model (LM) for textual data augmentation (DA) remains unexplored, moreover, textual DA methods struggle to balance the diversity and consistency of new samples. Most DA methods either perform logical modifications or rephrase less important tokens in the original sequence with the language model. In the context of SC, strong emotional tokens could act critically on the sentiment of the whole sequence. Therefore, contrary to rephrasing less important context, we propose DiffusionCLS to leverage a diffusion LM to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens. This approach ensures a balance between consistency and diversity, avoiding the introduction of noise and augmenting crucial features of datasets. DiffusionCLS also comprises a Noise-Resistant Training objective to help the model generalize. Experiments demonstrate the effectiveness of our method in various low-resource scenarios including domain-specific and domain-general problems. Ablation studies confirm the effectiveness of our framework's modules, and visualization studies highlight optimal deployment conditions, reinforcing our conclusions.




§ INTRODUCTION


Sentiment classification is a crucial application of text classification (TC) in Natural Language Processing (NLP) and can play a crucial role in multiple areas. However, NLP applications in domain-specific scenarios, such as disasters and pandemics, often meet with low-resource conditions, especially domain-specific problems, imbalance data distribution, and data deficiency <cit.>.  Recently, the birth of pre-trained language models (PLMs) and large language models (LLMs) have advanced the NLP field, giving birth to numerous downstream models based on them. On the one hand, these PLMs take the models to a new height of performance, on the other hand, since these models are highly data-hungry, they struggle to perform satisfactorily on most tasks under noisy, data-sparse and low-resource conditions <cit.>.




To address these challenges, one effective approach is data augmentation (DA), which enriches the diversity of the dataset without explicitly collecting new data <cit.>. Classic rule-based DA methods employ logical modifications to obtain pseudo samples, such as EDA <cit.>, and AEDA <cit.>. Model-based DA methods develop rapidly as the transformer architecture dominates the NLP field, most of these methods execute DA through corrupt-then-reconstruct (CTR), as examples shown in Table <ref>. Namely, masked language model (MLM) <cit.>, and GENIUS <cit.> which applies BART as the sample generator. Also, <cit.> finetunes GPT-2 and generates pseudo samples with label prompts, called LAMBADA. 


However, these methods often struggle with domain-specific tasks and uneven label distributions. Some methods generate samples solely relying on pre-trained knowledge, like GENIUS. The other though finetuned on the downstream dataset, these methods generate samples only conditioned on the label itself, such as LAMBADA, leading to strong label inconsistency, especially in data-sparse settings. Also, most CTR methods focus on replacing minor tokens in sequences but keeping the crucial tokens stationary to generate high-quality pseudo samples. 

In contrast, we corrupt the most label-related tokens first and reconstruct the whole sentence conditioned on the context and label prompt, as shown in Table <ref>, to diversify the key label-related tokens rather than less important contexts. This approach not only augments sample diversity but also upholds consistency through selective masking. Inspired by DiffusionBERT <cit.>, which is designed to recover the most informative tokens from those with less informatics, we propose DiffusionCLS. Additionally, building upon the findings of <cit.>, we further introduce consistency and diversity as crucial elements for quality of samples. High-quality pseudo samples must align with their labels and domain contexts, minimizing noise introduction. Integrating these samples enhances dataset diversity, thereby positively impacting the model performance.

DiffusionCLS initially finetunes PLM with a diffusion objective, functioning as a sample generator, followed by training the TC model in a noise-resistant manner. By fine-tuning the diffusion LM, we can then input original samples with their crucial tokens corrupted and use the label as a generation prompt to get new samples. This method diversifies the original dataset by replacing strong label-related tokens and also steers the model towards producing high-quality pseudo samples that comply with the diversity-consistency rule.


The major contributions of this paper can be summarized as follows:

    
  * We propose DiffusionCLS, which comprises a diffusion LM-based data augmentation module for SC, generating diverse but consistent pseudo samples by substituting diverse strong label-related contexts.
    
    
  * Designed and integrated a noise-resistant training method within the proposed DiffusionCLS, which significantly improves the SC model's performance with pseudo samples. 
    
    
  * Comprehensive experiments on domain-specific and multilingual datasets validate DiffusionCLS's superior performance in SC tasks. Detailed ablation studies highlights the effectiveness of its individual modules.

    
  * A visualization study is conducted to discuss the diversity-consistency trade-off, which further validates the effectiveness of DiffusionCLS.









§ RELATED WORK








 §.§ Low-Resource Text Classification










Motivated by the observation that data is often scarce in specific domains or emergent application scenarios, low-resource TC <cit.> has recently attracted considerable attention. Low-resource TC involves effectively categorizing text in scenarios where data is scarce or limited. 
<cit.> and <cit.> have explored several methods for low-resource TC, which mainly involve traditional machine learning techniques to increase data quantity and diversity.
 
Recently, since the studies by <cit.> and <cit.> demonstrated the impressive performance of PLMs across various NLP tasks, a significant amount of work has leaned towards using PLMs to address low-resource TC problems <cit.>. However, PLMs requires amounts of annotated samples for finetuning, data-sparce significantly impacts models' performances and DA could mitigate such problems.






 §.§ Textual Data Augmentation











To address low-resource challenges, various data augmentation methods have been proposed, including Easy-Data-Augmentation (EDA) <cit.>, Back-Translation (BT) <cit.>, and CBERT <cit.>. However, these methods, relying on logical replacements and external knowledge, often introduce out-domain knowledge and domain inconsistency. Moreover, these methods focus only on a specific original input, resulting in limited diversity.

Another type of data augmentation method includes representation augmentation approaches. These methods generate pseudo-representation vectors by interpolating or perturbing the representations of original samples. For instance, <cit.> proposed the groundbreaking technique known as mixup, and <cit.> recently proposed AWD, an advanced approach in textual DA.

Recent advancements in generative models have led to research on GPT-based paraphrasing data augmentation methods, such as LAMBADA <cit.>, which fine-tuned GPT-2 model to generate new samples. However, LAMBADA generates new samples based solely on specific labels, neglecting information from the original samples. Another research direction involves not fine-tuning PLMs but combining the language modeling capability of pretrained models with the generative diversity of diffusion models <cit.>, which significantly improves the capability of the generative encoder, i.e., MLM.

Since diffusion LMs can generate new sequences from masked original sequences, which matches the goal of retaining key information and rephrasing secondary information in generative data augmentation. Therefore, on top of diffusion LM, we propose DiffusionCLS, simultaneously considering label and domain consistency and generating pseudo samples by partially paraphrasing strong label-related tokens. Extensive experiments verify the effectiveness of our method and hopefully be extended to numerous NLP tasks.










§ METHODOLOGY


Sentiment classification models often overfit and lack generalization due to sample deficiency. To address this, we propose DiffusionCLS, consisting of Label-Aware Noise Schedule, Label-Aware Prompting, Conditional Sample Generation, and Noise-Resistant Training. A diffusion LM-based sample generator is integrated to generate new samples from the original dataset, enhancing TC model performance.

Figure <ref> illustrates DiffusionCLS. The diffusion LM-based sample generator generates new samples for data augmentation, while the TC model is trained for the specific TC task. Label-Aware Prompting and Label-Aware Noise Schedule are crucial for training the sample generator, and Conditional Sample Generation and Noise-Resistant Training contribute to the training of the TC model.



 §.§ Sample Generator

To generate usable samples for further TC model training, there are two crucial rules of success to satisfy, diversity and consistency. Therefore, we expect the generated samples to be as diverse as possible with consistency to the TC label and original domain simultaneously. However, higher diversity also leads to a higher difficulty in maintaining consistency. 

As <cit.> excavated the potential of combining diffusion models with LMs for sequence generation, we built the sample generator from the discrete diffusion model scratch. Precisely, we design the Label-Aware Noise Schedule for the diffusion LM, which helps the model to generate diverse and consistent samples. Additionally, we integrate Label-Aware Prompting into the training regime, enabling the model to grasp label-specific knowledge, subsequently serving as the guiding condition for sample generation. These two modules help the generator to surpass the diversity-consistency challenge and excel in performance.





  §.§.§ Label-Aware Noise Schedule

A proper algorithm of noise schedule could guide the diffusion LM to capture more accurate semantic relations. Moreover, the effectiveness of time-agnostic decoding has been demonstrated, indicating that incorporating implicit time information in the noise schedule process is effective <cit.>. Since the generated samples are also expected to stay consistent with the TC label and the original domain, we proposed Label-Aware Noise Schedule. 

The Label-Aware Noise Schedule begins by integrating a proxy model that has been fine-tuned for the TC task. This proxy model allows for the determination of the importance of each token in the TC process, quantified through attention scores between the [CLS] token and other tokens, which are derived from the last layer of proxy model and calculated as follows.


    w_i = 1/H∑_h=1^Hs_i^h,

where s_i^h represents the i-th token attention score in the h-th attention head, and w_i denotes the weight measuring the importance of the i-th token.

Motivated by <cit.>'s DiffusionBERT, we incorporates absorbing state in the LM noise schedule. In our method, during the masking transition procedure, each token in the sequence remains unchanged or transitions to [MASK] with a certain probability. The transition probability of token i at step t can be denoted as:

    q_t^i = 1 - t/T - λ· S(t) · w_i,



    S(t) = sintπ/T,

where q_t^i represents the probability that a token is being masked, and T denotes the total step number. λ is introduced to control the impact of w_i, as a hyper-parameter. 




By introducing strong label-related w_i, the diffusion model is guided to recover the tokens with lower weight first, then recover the tokens that are strongly related to the classification task later.












The probability of a token being masked is tied to its attention score relative to the [CLS] token, reflecting its contribution to the TC objective. Figure <ref> shows that masking probabilities depend on the token's label-related information. Label-Aware Noise Scheduling guides the model to recover the most label-related key tokens from those less crucial to the classification task.



  §.§.§ Label-Aware Prompting


However, such a noise schedule still poses a challenge to the conditional generation process. The diversity-consistency trade-off becomes more intense when important tokens are masked. With fewer unmasked tokens provided, the model naturally has a higher possibility of generating tokens that would break the label consistency.



To address this challenge, we propose Label-Aware Prompting, a method that offers supplementary conditional information during both training and inference phrases. This additional information aids the model in generating samples that uphold label consistency.

As Figure <ref> illustrated, following the masking of samples in the noise schedule process, the labels of these samples are concatenated with their respective masked sequences. 





 §.§ Text Classification Model




In this work, we adopt encoder-based PLM as our backbone model and finetuned them for the TC task. Though diffusion LM is strong enough to maintain consistency and diversity at the same time, the introduction of pseudo samples unavoidably introduced noise data to the training of the TC model. To mitigate such a problem, we design a contrastive learning-based noise-resistant training method, further improving the scalability of the proposed DiffusionCLS.



  §.§.§ Reflective Conditional Sample Generation

We implement label prompting as a prior for the sample generator, akin to Label-Aware Prompting. Additionally, we introduce a novel reflective conditional sample generation module within the training loop of the TC model. This module dynamically generates masked sequences for the sample generator, integrating insights from label annotations and attention scores derived from the TC model simultaneously, calculating weights for each token with Eq.<ref>.





However, generating pseudo samples from varying degrees of masking will result in various degrees of context replacement flexibility, thus impacting the consistency and diversity of pseudo samples. Essentially, providing a proper amount of conditional information will lead to plausible samples. Thus, we perform multiple experiments to search for the best condition, which will be further discussed in Section <ref>.



  §.§.§ Noise-Resistant Training


The introduction of pseudo samples unavoidably introduced noise data to the training of the TC model. To mitigate such a problem, we design a contrastive learning-based noise-resistant training method, further improving the scalability of the proposed DiffusionCLS.



Figure <ref> demonstrates the Noise-resistant Training. Specifically, besides including supervision signals from labels of original and generated samples, we also guide the model to enlarge the gap between samples with different labels.



Consider a dataset comprising m distinct categories C = {c_1,c_2,...,c_m}, we can obtain k samples from the original training set, and the corresponding subscript list is I = {1, 2,..., k-1, k}. Essentially, a batch of sentences S = {s_1, s_2, ..., s_k-1, s_k}, their corresponding label sequence L = [l_1, l_2, ..., l_k-1, l_k] with l_i ∈ C, and negative set for each sample N_i = {j ∈ I| l_j≠ l_i}.  From this, we derive semantic representations H = {h_1, h_2, ..., h_k-1, h_k} from the TC model. Furthermore, employing a sample generator yields B new samples for each original sample s_i, denoted as G_i = {g_0^s_i,g_1^s_i, ..., g_B-1^s_i, g_B^s_i}, where g_0^s_i = s_i.

Contrastive Loss. To avoid expanding the impact of noise samples, we calculate contrastive loss from only the original samples. With the aim to enlarge the gap between samples from different categories, the contrastive loss can be calculated as:

    L_c = 1/Klog∑_i∈ I∑_j ∈ N_iexp(sim(h_i,h_j)/τ)  ,

where sim() denotes the consine similarity function and τ is a  hyper-parameter as a scaling term.

Classification Loss. We also allows supervision signals directly affects the training of the TC model through the cross entropy loss, which can be denoted as:






    L_e = -1/K(B+1)∑_i∈ I∑_b=0^B∑_c∈ C y^i_b,clog(ŷ^i_b,c),

where y^i_b,c is the label indicator, and ŷ^i_b,c is the predicted probability of b-th pseudo sample of the original sample i being of class c.

Training Objective. From two losses mentioned above, we formulated the overall training objective for the TC model, which can be denoted as:


    L = L_c + L_e.





§ EXPERIMENTS





 §.§ Datasets and Baselines





To measure the efficiency of the propose DiffusionCLS, we utilize both domain-specific and domain-general datasets comprising samples in Chinese, English, Arabic, French, and Spanish. Namely, domain-specific SMP2020-EWECT[https://smp2020ewect.github.io], India-COVID-X[https://www.kaggle.com/datasets/surajkum1198/twitterdata], SenWave <cit.>, and domain-general SST-2 <cit.>. Additionally, to compare with the most cutting-edge low-resource TC methods, we utilize SST-2 dataset to evaluate our method in the few-shot setting. Dataset statistic and descriptions are demonstrated in Appendix <ref>.



To thoroughly explore and validate the capabilities of DiffusionCLS, we compare our method with a range of data augmentation techniques, from classic approaches to the latest advancements for low-resource TC. Specifically, we take Resample, Back Translation <cit.>, Easy Data Augmentation (EDA) <cit.>, SFT GPT-2 referenced to LAMBADA <cit.>, AEDA <cit.>, and GENIUS <cit.> as our baselines. Also, we compare our method in the few-shot setting with a couple of cutting-edge methods, namely, SSMBA <cit.>, ALP <cit.>, and SE <cit.>. More details of our baselines are demonstrated in Appendix <ref>.









 §.§ Experiment Setup


We set up two experimental modes, entire data mode and partial data mode, to reveal the effectiveness of our method in different scenarios. Since severe imbalanced distribution challenges existed, we take macro-F1 and accuracy as our major evaluation metrics.

Also, we conduct 5-shot and 10-shot experiments on SST-2 to investigate the performance of DiffusionCLS in extreme low-resource conditions. For evaluation, we use accuracy as the metric and report the average results over three random seeds to minimize the effects of stochasticity.

Additionally, we setup comparisons between variant augmentation policies, namely, generate new samples until the dataset distribution is balanced, and generate n pseudo samples for each sample (n-samples-each), which denoted as B/D and G/E in Table <ref>, and n=4 in our experiments.
Other related implementation details are described in Appendix <ref>.




 §.§ Results and Analysis

The results of entire-data-setting experiments on datasets SMP2020-EWECT and India-COVID-X are mainly demonstrated in Table <ref>, which we compare DiffusionCLS with other strong DA baselines. For experiments with partial-data and few-shot settings, results are majorly showed in Figure <ref> and Table <ref>.



























Results in Entire Data Mode. As shown in Table <ref>, in general, the proposed DiffusionCLS outperforms most DA methods on domain-specific dataset SMP2020-EWECT and India-COVID-X, especially under G/E policy. Notably, the DiffusionCLS positively impacts the TC model across all policies and datasets, which most baselines fail.

Our method excels in dealing with the challenge of uneven datasets. Under severe uneven distribution and domain-specific scenarios, i.e., the dataset SMP2020-EWECT, most DA baselines fail to impact the classification model positively except DiffusionCLS, which achieves the best performance. Also, our method achieves competitive performance under data-sparse and domain-specific scenarios, i.e., in the dataset India-COVID-X, most DA methods bring improvement to the classification model, and our DiffusionCLS ranked second.

Rule-based DA methods such as EDA, rather lack diversity bringing overfit problems or solely relying on out-domain knowledge therefore breaking consistency and impacting the task model negatively. For model-based methods, though most methods significantly increase the diversity of the generated samples, they rather generate samples solely depending on pretraining knowledge and in-context-learning techniques or generate samples only conditioned on the label itself, posing a challenge of maintaining consistency. 



Results in Partial Data Mode and Few-shot Settings. As shown in Figure <ref> and Table <ref> in Appendix <ref>, the proposed DiffusionCLS method consistently improves the classification model. Notably, DiffusionCLS matches the PLM baseline performance on the Arabic SenWave dataset using only 50% of the data samples.






We also compare DiffusionCLS with the most cutting-edge few-shot methods on SST-2 dataset under 5-shot and 10-shot setting, the results are shown in Table <ref>. Though our method fails to surpass all few-shot baselines, it still achieves competitive performance with those designed for the few-shot task. 

Since DiffusionCLS requires diffusion training to adapt to domain-specific tasks, extreme sample insufficiency may introduce noise, negatively impacting the model. However, our method positively impacts the TC model in most low-resource cases by effectively utilizing pre-trained and in-domain knowledge, from severe imbalanced label distribution to severe sample insufficiency.




 §.§ Ablation Study


To validate the effectiveness of modules in the proposed DiffusionCLS, we conduct ablation studies to study the impacts of each module. Table <ref> presents the results of the ablation experiments. In each row of the experiment results, one of the modules in DiffusionCLS has been removed for discussion, except D.A., which removes all modules related to the generator and only applies noise-resistance training.

Overall, all modules in the proposed DiffusionCLS works positively to the TC model, compared with the pure PLM model, the application of DiffusionCLS leads to 2.11% and 3.66% rises in F1 values on dataset SMP2020-EWECT and India-COVID-X respectively.

The results of ablation studies further validate that the Label-Aware Prompting effectively improves the quality of pseudo samples. Also, the Noise-Resistant Training reduces the impact of noise pseudo samples.







 §.§ Discussions and Visualizations


Generating pseudo samples from more masked tokens provides more flexibility for generation and tends to result in more diverse samples, however, it will enlarge the possibility of breaking the consistency since less information is provided.

To analyze the optimal amount of masks for generating new pseudo samples, we conduct experiments on the India-COVID-X dataset. During conditional sample generation, we gather masked sequences from 32 noise-adding steps, group them into sets of eight, and evaluate how varying masking levels impact the model's performance.



As shown in Figure <ref>, our observations indicate a unimodal trend. The model's performance improves with increased masking, peaks at the 4th group, and then declines with further masking. This reflects the diversity-consistency trade-off, more masked tokens create more diverse samples, but overly diverse samples may be inconsistent with original labels or domain.

To explore the relationship between generated pseudo samples and original samples, we conduct 2D t-SNE visualization. Figure <ref> shows that as masking increases, pseudo samples gradually diverge from the original samples, indicating increased diversity. 










§ CONCLUSION

In this work, we propose DiffusionCLS, a novel approach tackling SC challenges under low-resource conditions, especially in domain-specific and uneven distribution scenarios. Utilizing a diffusion LM, DiffusionCLS captures in-domain knowledge to generate high-quality pseudo samples maintaining both diversity and consistency. This method surpasses various kinds of data augmentation techniques. Our experiments demonstrate that DiffusionCLS significantly enhances SC performance across various domain-specific and multilingual datasets. Ablation and visualization studies further validate our approach, emphasizing the importance of balancing diversity and consistency in pseudo samples. DiffusionCLS presents a robust solution for data augmentation in low-resource NLP applications, paving a promising path for future research.



§ LIMITATIONS

Like most model-based data augmentation methods, the performance of data generators is also limited in extreme low-resource scenarios. This limitation persists because the model still necessitates training on the training data, even with the potential expansion of the dataset through the inclusion of unlabeled data, data deficiency impacts the data generator negatively.








§ EXPERIMENT SETUP, IMPLEMENTATION, AND DATASET STATISTICS




 §.§ Experiment Setup

The low-resource challenge in TC includes problems like insufficient annotated samples, domain-specific adaptation problems, and imbalanced distribution. To measure the capability of the proposed DiffusionCLS to mitigate these problems, we conducted experiments on three domain-specific datasets with respect to the problems mentioned above, as shown in Table <ref>. 



 §.§ Implementation

For implementation, we take bert-base-uncased[https://huggingface.co/google-bert/bert-base-uncased] and chinese-roberta-wwm[https://huggingface.co/hfl/chinese-roberta-wwm-ext] from the huggingface platform respectively for English and Chinese dataset training.

Also, hyper-parameters settings of our work are demonstrated in Table <ref> and Table <ref>.










 §.§ Datasets

For our experiments, we utilize multilingual datasets, both domain-specific and domain-general, to evaluate the proposed DiffusionCLS. Data statistics and their challenges are demonstrated in Table <ref> and Table <ref>.

    
  * SMP2020-EWECT[https://smp2020ewect.github.io]. This Chinese dataset includes 8,606 pandemic-related posts, categorized into neutral, happy, angry, sad, fear, and surprise, with highly imbalanced label distribution.
    
  * India-COVID-X[https://www.kaggle.com/datasets/surajkum1198/twitterdata]. This dataset contains cleaned English tweets from India X platform on topics such as coronavirus, COVID-19, and lockdown. The tweets have been labeled into four sentiment categories with relatively balanced label distribution.
    
  * SenWave<cit.>. This dataset includes about 5,000 English tweets and approximately 3,000 Arabic tweets in the specific domain of the pandemic and lockdown, which are annotated with sentiment labels. English-translated French and Spanish annotated samples are also included. We extract all single label samples for experiments.
    
  * SST-2<cit.>. It includes 11,855 movie review sentences parsed by the Stanford parser, with 215,154 unique phrases annotated by three human judges.










§ BASELINES






    
  * Non-Generative Methods 
    
        
  * SSMBA<cit.>. Uses a corruption and reconstruction function to augment data by filling in masked portions.
    
        
  * ALP<cit.>. Employs Lexicalized Probabilistic Context-Free Grammars to generate syntactically diverse augmented samples.
        
        
  * SE<cit.>. Utilizes a self-evolution learning-based mixup technique to create adaptive pseudo samples for training.
        
        
  * AEDA<cit.>. Randomly insert punctuations into the original sentences to produce new samples.
    

    
  * Generative Methods
    
            
  * GPT-2<cit.>. Fine-tunes GPT-2 with prompt-based SFT, prompting labels to generate pseudo samples.

            
  * GENIUS<cit.>. 
        A conditional text generation model using sketches as input, which can fill in the missing context for a given sketch.
        
    

    
  * Representation Augmentation Methods
    
        
  * mixup<cit.>. Mixup is a representational DA technique that creates new training samples by linearly interpolating between pairs of examples and their labels.

        
  * AWD<cit.>. AWD generates challenging positive examples for low-resource text classification by diluting strong positive word embeddings with unknown-word embeddings.
    









§ EXPERIMENT RESULTS WITH PARTIAL DATA MODE


The proposed DiffusionCLS method consistently enhances the classification model, achieving higher accuracy with only 50% training data than the raw PLM on dataset SMP2020-EWECT. Detailed results are shown in Table <ref>.




