2023

These authors contributed equally to this work. These authors contributed
equally to this work. [1]Hao He

[1]MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong
University, Shanghai, China 2]School of Software Engineering, Sun Yat-sen
University, Guangzhou, China

# Accurate Use of Label Dependency in Multi-Label Text Classification Through
the Lens of Causality

Caoyun Fan<EMAIL_ADDRESS>Wenqing Chen<EMAIL_ADDRESS>Jidong
Tian<EMAIL_ADDRESS>Yitian Li<EMAIL_ADDRESS><EMAIL_ADDRESS>Yaohui Jin<EMAIL_ADDRESS>*  [

###### Abstract

Multi-Label Text Classification (MLTC) aims to assign the most relevant labels
to each given text. Existing methods demonstrate that label dependency can
help to improve the model’s performance. However, the introduction of label
dependency may cause the model to suffer from unwanted prediction bias. In
this study, we attribute the bias to the model’s misuse of label dependency,
i.e., the model tends to utilize the correlation shortcut in label dependency
rather than fusing text information and label dependency for prediction.
Motivated by causal inference, we propose a CounterFactual Text Classifier
(CFTC) to eliminate the correlation bias, and make causality-based
predictions. Specifically, our CFTC first adopts the predict-then-modify
backbone to extract precise label information embedded in label dependency,
then blocks the correlation shortcut through the counterfactual de-bias
technique with the help of the human causal graph. Experimental results on
three datasets demonstrate that our CFTC significantly outperforms the
baselines and effectively eliminates the correlation bias in datasets.

###### keywords:

Multi-Label Text Classification, Label Dependency, Correlation Shortcut,
Counterfactual De-bias

##  1 Introduction

Multi-Label Text Classification (MLTC) is a fundamental but challenging task
(Wang et al, 2020) in Natural Language Processing (NLP), and it has be applied
in many real-world scenarios, such as text categorization (Wang et al, 2019),
sentiment analysis (Wankhade et al, 2022), emotion recognition (Alswaidan and
Menai, 2020) and so on. Therefore, it is necessary to design an accurate and
efficient multi-label text classifier for practical applications.

Table 1: An example of MLTC. In this example, If some labels are known (listed in _Given Labels_), we are more likely to predict some labels (listed in _Positive Labels_) and exclude others (listed in _Negative Labels_).  Text | Germany beat Argentina on Gotze’s goal to win World Cup.
---|---
Given Labels | Sport, World Cup
Positive Labels | Football
Negative Labels | Technology, Education, Economics

Initially, MLTC was decomposed into multiple independent binary classification
tasks (Boutell et al, 2004). Soon the researchers realized that Label
Dependency (LD) (Chen and Ren, 2021) could be exploited to improve
performance. Intuitively, knowing some labels makes it easier to predict other
labels, because the labels tend to have dependency (Yang et al, 2018). Taking
Table 1 as an example, the given labels ‘Sports, World Cup’ provide additional
information, which makes the following predictions more reliable (more likely
to predict ‘Football’ and exclude ‘Technology, Education, Economics’).
Therefore, researchers in the NLP community made great efforts in exploiting
label dependency. There are two mainstream methods: the first is the explicit
method, i.e., the model makes predictions based on the text and the previous
predictions, and a series of studies (Yang et al, 2018, 2019; Wang et al,
2021a) treated previous predictions as auxiliary information to support
decision making process in different forms; the second is the implicit method,
i.e., designing specific modules (e.g. attention mechanisms (Xiao et al, 2019;
Liu et al, 2021b), graph networks (Liu et al, 2021a; Vu et al, 2022)) to
capture implicit information in label dependency. Both methods demonstrated
the performance gain from label dependency. However, the LD-based models
suffer from two unwanted biases:

  * •

Exposure Bias (Yang et al, 2018): some incorrect predictions lead to the error
accumulation in the following predictions.

  * •

Stereotypes Bias (Zhang et al, 2021): the model tends to generate high-
frequency label combinations while ignoring low-frequency label combinations.

Figure 1: An example of the LD-based model dominated by label dependency. When
the text and partial labels are fed to the model, the first prediction is
Sport, as it is a high-frequency combination with World Cup (Stereotypes
Bias), and then the incorrect prediction Sport leads to the following error
Basketball (Exposure Bias). The text is ignored in this process.

In this study, we attribute these biases to the fact that the LD-based models
may be dominated by label dependency and ignore the text, as shown in Fig. 1.
Through the lens of causality, MLTC’s underlying mechanism is a ‘text
$\rightarrow$ label’ causality path, and label dependency should serve as
auxiliary information for this path. However, label dependency, as a
statistical cue (Niven and Kao, 2019) between labels, has an additional ‘label
$\rightarrow$ label’ correlation shortcut. For example, if the labels ‘Sport,
World Cup’ are known in Table 1, the LD-based model can predict the label
‘Football’ even without the text, because there is a strong correlation
between them. In fact, the co-occurrence relationship between labels is sparse
(in Appendix B), which means that the statistical correlations between partial
labels can be easily captured by the LD-based models. Since neural networks
lack the ability to distinguish between correlation and causality (Feder et
al, 2021), label dependency could be misused, and thus lead to unwanted
correlation bias due to the correlation shortcut (Shah et al, 2020).

To avoid unwanted bias, accurate use of label dependency is crucial for LD-
based models. In the human decision making process, humans usually pre-design
an appropriate causal graph (Yao et al, 2021) for a specific task, and then
eliminate the correlation bias outside the causal graph to make causality-
based predictions. Taking MLTC as an example, we would deliberately avoid
‘guessing’ the predictions based only on label dependency, because our causal
graph should not contain this correlation shortcut. It naturally occurs to us
that if the causal graph is introduced into the LD-based model, the model
could imitate the human decision making process to introspect whether the
predictions are deceived by the correlation shortcut. Specifically, when the
LD-based model is provided with the text and Label Information111In this
paper, Label Information (LI) refers to the information extracted based on
label dependency.  (LI), we employ counterfactual inference technique (Niu et
al, 2021; Wang et al, 2021c) to simulate two states:

Observation: _What would the prediction be, if the model gets the text and
LI?_ Intervention: _What would the prediction be, if the model could only get
LI by intervening on the text?_

In the observation state, the model confuses correlation and causality, while
in the intervention state, the model has to rely only on the correlation
shortcut to make predictions. We simulate the human de-bias process by
comparing these two states.

In this paper, we proposed a novel counterfactual framework called
CounterFactual Text Classifier (CFTC) to implement the process described
above. Specifically, Our CFTC was designed from two perspectives: extracting
LI and using LI. We designed a novel predict-then-modify backbone in order to
extract more complete and precise LI from label dependency, and employed the
counterfactual intervention on the causal graph of MLTC to remove the
correlation bias to enable label dependency to be used accurately.
Furthermore, we extensively examined our CFTC on three datasets: AAPD (Yang et
al, 2018), RCV1 (Lewis et al, 2004) and Reuters-21578 (Lewis, 1997), and the
results revealed that CFTC obtained superior performance than other baselines
under the same premise of other settings. Our contributions are:

  * •

We analyze the LD-based model’s decision processes through the lens of
causality and attribute the bias to the misuse of label dependency.

  * •

We employ a novel predict-then-modify backbone to extract more precise LI, and
propose a counterfactual framework to block the correlation shortcut
introduced by LI so that label dependency can be used accurately.

  * •

We evaluate our proposed method named CFTC on three datasets: AAPD, RCV1 and
Reuters-21578, and the experimental results demonstrate the effectiveness of
our CFTC.

##  2 Related Work

###  2.1 Multi-Label Text Classification

For MLTC task, a common solution was to decompose it into multiple independent
binary classification tasks, which was well known as Binary Relevance (BR)
(Boutell et al, 2004). Researchers soon realized the importance of label
dependency. Label Powerset (LP) (Tsoumakas and Katakis, 2007) viewed MLTC as a
multi-class classification problem by classifying data on all unique label
combinations. Classifier Chains (CC) (Read et al, 2009) exploited the chain
rule and make predictions relying on the previous prediction. (Yang et al,
2018, 2019; Nam et al, 2017) viewed MLTC as a sequence generation task and
utilized the Seq2Seq model as a multi-class classifier. However, both CC and
Seq2Seq-based methods relied on a predefined label order. As such methods were
sensitive to the label order, many studies (Tsai and Lee, 2020) attempted to
tackle the label order dependency problem. Recently, many studies proposed
approaches that are not based on the Seq2Seq architecture to exploit label
dependency. ML-Reasoner (Wang et al, 2021a) employed multiple rounds of
predictions to obtain the final prediction. CorNet (Xun et al, 2020) utilized
BERT (Devlin et al, 2019) and added an extra module to learn label dependency,
enhance raw label predictions. LACO (Zhang et al, 2021) and HiMatch (Chen et
al, 2021a) extracted label dependency using the hierarchical structure among
labels and explicitly modeled the label dependency in a multi-task framework.
LDGN (Ma et al, 2021) learned label-specific components based on the
statistical label co-occurrence in Graph Convolution Network (GCN). LELC (Liu
et al, 2021a) simplified the process of model learning by the label
correlation matrix. (Ozmen et al, 2022) indicated that the presence or absence
of each label is valuable information for MLTC. However, these methods ignore
the potential bias introduced by the correlation shortcut when exploiting
label dependency.

###  2.2 Counterfactual Inference

Counterfactual inference (Wang et al, 2022) is a branch of causal inference
(Luo et al, 2019; Yao et al, 2021), which could remove the bias in inference.
Usually, counterfactual inference requires a causal graph (Li and Yue, 2020)
reflecting the causal relationships between variables, and counterfactual
inference means that some of the variables are fixed to values that violate
the fact, thus obtaining inferences that do not fit the causal graph. A series
of studies attempted to incorporate counterfactual inference into deep
learning: in computer vision, (Niu et al, 2021) reduced language bias by
subtracting the direct language effect from the total causal effect in Visual
Question Answering, (Yue et al, 2021) performed a counterfactual intervention
on class attributes and obtained excellent performance in Zero-Shot Learning;
in recommendation system, (Wang et al, 2021c) used counterfactual inference to
distinguish the impact of exposure features and content features, (Yue et al,
2021) eliminated confounding factors in the recommendation system using
counterfactual inference; in NLP, (Qian et al, 2021) reduced the document-
level label bias and word-level keyword bias in text classification by
counterfactual inference; (Wang and Culotta, 2021) extracted causal features
from the text by constructing counterfactual data, (Paranjape et al, 2022)
developed a Retrieve-Generate-Filter (RGF) technique to create counterfactual
evaluation and training data with minimal human supervision. In this study, we
attempt to introduce counterfactual inference into MLTC in order to eliminate
the correlation bias.

##  3 Causal Analysis of MLTC

###  3.1 Problem Formulation

MLTC studies the classification problem that each text is associated with a
set of labels simultaneously. Formally, a MLTC dataset can be denoted as
$D=\left\\{\mathcal{T},\mathcal{Y}\right\\}$, where $\mathcal{T}$ is the text
set and $\mathcal{Y}$ is the class set. For each text $T_{i}\in\mathcal{T}$,
it is made up of $m$ words $T_{i}=\\{w_{i}^{1},w_{i}^{2},\dots,w_{i}^{m}\\}$,
and is annotated with the corresponding label $Y_{i}\in\\{0,1\\}_{|L|}$. The
target of MLTC is to learn a mapping function
$F:\mathcal{T}\rightarrow\mathcal{Y}$ to minimizing the empirical risk as:

$\min\frac{1}{N}\sum_{i=1}^{N}\delta(Y_{i},F(T_{i})),$ (1)

where $\delta(\cdot)$ refers to the loss function used in the training
process.

###  3.2 Causal Graph and Intervention

From the perspective of the LD-based model, MLTC can be formalized as a two-
step process: extracting label information based on label dependency and
fusing label information with text to make decisions. We notate this process
as:

$Y_{T+LI}=F(T,LI),$ (2)

where $LI$ refers to Label Information extracted from label dependency. This
format is intuitive because LI brings additional valuable information.

(a) Human

(b) LD-based Model

(c) Intervention

Figure 2: Causal graphs and Counterfactual Intervention in MLTC. $LI$ and $FI$
refer to Label Information and Fusion Information, respectively, and $Y$ is
the prediction. ‘Grey’ refers to the counterfactual part.

In the ideal state, the model follows the human decision making process to
fuse the text and LI (Text $\rightarrow$ FI $\leftarrow$ LI) and make
predictions based on the causality path (FI $\rightarrow$ Y). However, since
the LD-based model cannot distinguish between causality and correlation, the
neural network’s preference for correlation features (Du et al, 2021) may
cause the model to be dominated by the label correlation shortcut (LI
$\rightarrow$ Y), i.e., the conditional probability of the prediction
collapses as:

$p(Y|T,LI)\approx p(Y|LI),$ (3)

which is the source of unwanted correlation bias. Following the concepts of
causal inference, we construct the causal graphs of human and the LD-based
model in MLTC as illustrated in Fig. 2(a) & 2(b). The only difference between
these two causal graphs is the LD-based model’s predictions can be interfered
with by the LI $\rightarrow$ Y shortcut (the red line in Fig. 2(b)).
Therefore, we expect LD-based models to imitate humans by deliberately
blocking this shortcut.

However, Eq. 2 confuses the effect of text and LI for prediction, so we cannot
distinguish whether the shortcut interferes with the model’s predictions.
According to the counterfactual framework (Niu et al, 2021; Wang et al, 2021b,
c), we estimate the label correlation shortcut by blocking the correct text
information as:

$Y_{T^{*}+LI}=F(T^{*},LI),$ (4)

where $T^{*}$ means the counterfactual text222In this paper, we only require
that $T^{*}$ does not contain the correct text information in $T$, rather than
a semantic counterfactual text.  for the text $T$. Eq. 4 describes the
scenario in Fig. 2(c): the model does not have access to the correct text
information and only makes predictions based on the correct LI. This process
imitates human ‘guessing’ the predictions based only on label dependency.
Thus, $F(T^{*},LI)$ is a reasonable estimate of the label correlation
shortcut. Then, we employ the counterfactual de-bias technique (Niu et al,
2021) to remove the label correlation bias outside the human causal graph as:

$Y_{cd}=F(T,LI)-F(T^{*},LI).$ (5)

Intuitively, the model obtains de-biased predictions by subtracting the
interference of label correlation shortcut, similar to the causal effect
estimation (Cheng et al, 2022). In terms of learning strategy, the
counterfactual inference decouples causality and correlation in datasets based
on the causal graph in Fig. 2(a), so the model could eliminate the label
correlation bias and make causality-based predictions.

##  4 Methodology

Figure 3: The architecture of the CounterFactual Text Classifier (CFTC). The
predict-then-modify backbone (drawn in blue) extracts and utilizes LI based on
the text and label dependency. The model eliminates the label correlation bias
through Mask Mechanism (MM) and Counterfactual De-bias modules (drawn in red).

To implement counterfactual de-bias process in Section 3, we introduce our
CounterFactual Text Classifier (CFTC), as shown in Fig. 3. In order to make
full use of label dependency and extract precise LI, we employ the predict-
then-modify backbone in Section 4.1. To remove the correlation bias from label
dependency, we introduce the counterfactual de-bias in our CFTC in Section
4.2. The training details are described in Section 4.3.

###  4.1 Predict-Then-Modify Backbone

The premise of using label dependency is to extract precise LI from label
dependency. However, both explicit and implicit methods of LD-based models
have limitations in extracting LI: the available labels in explicit methods
(e.g., Seq2Seq models) are incomplete and shallow; the label features
extracted by implicit methods are not guaranteed to be pure, as text
information may be mixed in.

To overcome these limitations, we adopt the predict-then-modify backbone in
our CFTC: the initial prediction $\widehat{Y}_{T}$ is first obtained by the
text only, then LI is extracted from $\widehat{Y}_{T}$, the final prediction
is obtained based on the text and LI. This backbone makes two complete
predictions, and the latter prediction utilizes information from the previous
prediction, so we name it predict-then-modify backbone.

In this backbone, LI is extracted from the initial prediction
$\widehat{Y}_{T}$ (Section 4.1.1), which ensures the completeness and purity
of LI; the LI extractor based on graph neural network (Section 4.1.2) ensures
that the deeper LI would not be missed; the attention mechanism (Section
4.1.3) ensures the effective fusion of text and LI.

####  4.1.1 Text Encoder

Just like the traditional method, we employ a text encoder to transform the
original text $T=\\{w^{1},w^{2},\dots,w^{m}\\}$ into text information $H$.
It’s worth mentioning that CFTC is encoder-agnostic and most text encoders are
available for CFTC. Here, we take BiLSTM (Hochreiter and Schmidhuber, 1997) as
an example.

First, each word $w^{i}$ in the text would be converted into a word vector
$e^{i}\in\mathbb{R}^{D_{W}}$, so the text would be transformed into a series
of word vectors as $E=\\{e^{1},e^{2},\dots,e^{T}\\}$. Then, the forward hidden
state $\overrightarrow{h^{i}}\in\mathbb{R}^{D_{T}}$ and the backward hidden
state $\overleftarrow{h^{i}}\in\mathbb{R}^{D_{T}}$ at step $i$ can be computed
with two LSTM models from both directions as:

$\begin{split}\overleftarrow{h^{i}}&=\overleftarrow{\text{LSTM}}(e^{i},\overleftarrow{h^{i+1}}),\\\
\overrightarrow{h^{i}}&=\overrightarrow{\text{LSTM}}(e^{i},\overrightarrow{h^{i-1}}),\end{split}$
(6)

where $\overrightarrow{h^{i-1}}$ and $\overleftarrow{h^{i+1}}$ represent the
previous hidden states from two directions, respectively. The final hidden
state at step $i$ is the concatenation of two hidden states as
$h^{i}=[\overrightarrow{h^{i}}\oplus\overleftarrow{h^{i}}]\in\mathbb{R}^{2D_{T}}$,
where $\oplus$ denotes the concatenation operation. A series of hidden states
$\\{h^{1},h^{2},\dots,h^{m}\\}\in\mathbb{R}^{2D_{T}\times|m|}$ is considered
as text information $H$ extracted from the text encoder. Finally, text
information $H$ is fed into the scoring module $f_{T}(\cdot)$ to obtain each
label’s score $s_{T}\in\mathbb{R}^{|L|}$ as:

$s_{T}=f_{T}(\text{pool}(H))=W_{T}\cdot\text{pool}(H)+b_{T},$ (7)

where $W_{T}\in\mathbb{R}^{|L|\times 2D_{T}}$ and $b_{W}\in\mathbb{R}^{|L|}$
represent the weight parameter and the bias parameter in $f_{T}(\cdot)$,
respectively. The initial prediction $\widehat{Y}_{T}$ can be derived from
$s_{T}$ by activation function $\text{Sigmoid}(\cdot)$ and the threshold $\mu$
as:

$\widehat{Y^{i}_{T}}=\begin{cases}1,&\text{if}\
\text{Sigmoid}(s_{T}^{i})\geq\mu\mathchar 24635\relax\\\ 0,&\text{if}\
\text{Sigmoid}(s_{T}^{i})<\mu.\end{cases}$ (8)

####  4.1.2 Graph Neural Network LI Extractor

We consider that $\widehat{Y}_{T}$ as a prediction is the shallowest LI, and
mutual interactions between labels (Ma et al, 2021) can be exploited to
extract deeper LI. To capture the implied interactions of labels, we employ
the label co-occurrence matrix (Ma et al, 2021; Liu et al, 2021a) as prior
knowledge and apply a graph neural network to extract deeper LI. The label co-
occurrence matrix $M\in\mathbb{R}^{|L|\times|L|}$ is the statistic of co-
occurrence between labels, where $M_{ij}$ denotes the conditional probability
of a text belonging to label $L_{i}$ when it belongs to label $L_{j}$.
Following the normalization method in (Kipf and Welling, 2017), the label co-
occurrence matrix $M$ is normalized as:

$\widehat{M}=D^{-\frac{1}{2}}\cdot M\cdot D^{-\frac{1}{2}},$ (9)

where $D$ is a diagonal degree matrix of $M$. The visualization of the label
co-occurrence matrix is in Appendix B.

Specifically, we utilize GCN (Kipf and Welling, 2017) to extract deep LI from
shallow $\widehat{Y}_{T}$. First, $\widehat{Y}_{T}$ needs to be embedded into
LI space. (Ozmen et al, 2022) indicated that the presence or absence of each
label is valuable information, so we design two embeddings
$\\{E_{i}^{in},E_{i}^{out}\\}\in\mathbb{R}^{D_{L}}$ for each label $L_{i}$ to
represent whether $L_{i}$ appears in $\widehat{Y}_{T}$, and each label chooses
the proper embedding based on $\widehat{Y}_{T}$ to compose
$E_{L}\in\mathbb{R}^{|L|\times D_{L}}$, which is initialized to $LI_{0}$ as
the shallowest LI. Then, $\widehat{M}$ is employed as the adjacency matrix in
multi-layer GCN. Each GCN layer takes the LI of the previous GCN layer
$LI_{i}$ as input to extract deeper $LI_{i+1}$. The layer-wise propagation
rule is as follows:

$LI_{i+1}=\sigma(\widehat{M}\cdot LI_{i}\cdot W_{i}),$ (10)

where $\sigma(\cdot)$ denotes the ReLU activation function, and $W_{i}$ is the
learnable parameter in the GCN layer. Suppose there are $n$ layers of GCN and
$LI$ is finally represented as:

$LI=\text{pool}(LI_{n})\in\mathbb{R}^{D_{L}}.$ (11)

The knowledge in the label co-occurrence matrix is sourced from the
deterministic label distribution, so the initial LI here is embedded from the
discrete prediction $\widehat{Y}_{T}$, rather than the continuous label-
specific features (Xiao et al, 2019; Ma et al, 2021). This design prevents LI
from mixing with text information, which helps our LI extractor to obtain more
precise and consistent LI.

####  4.1.3 LI-Attention Information Fusion

In the predict-then-modify backbone, the final prediction is obtained based on
the text and LI. Given text information $H$ and $LI$, the most common
information fusion module is concatenating these two parts and feeding it to
scoring function $f_{T+LI}(\cdot)$ as:

$s_{T+LI}=f_{T+LI}(\text{pool}(H)\oplus LI).$ (12)

However, (Chen et al, 2020) pointed out that simple feature aggregation
operations (e.g. $\text{pool}(\cdot)$) would limit the model’s performance
because each label’s related component is different in a text. Considering the
semantic information of labels determine the semantic connection between
labels and texts (Xiao et al, 2019), we propose the LI-attention information
fusion module to capture each text’s feature. Specifically, we explicitly
represent the semantic connection between $H$ and $LI$ by $LI$-$H$ attention
score $A\in\mathbb{R}^{m}$ as:

$A=\text{softmax}(H^{T}\cdot(W_{a}\cdot LI)),$ (13)

where $W_{a}\in\mathbb{R}^{2D_{T}\times D}$ is a weight parameter. The LI-
specific text information $h_{TI}$ can be obtained by a linear combination of
$H$ with the help of $A$ as:

$h_{TI}=\sum_{k=1}^{m}A_{k}h^{k}.$ (14)

Then, the score fusing text and LI can be expressed as:

$\begin{split}s_{T+LI}&=f_{T+LI}\left(h_{TI}\oplus LI\right)\\\
&=W_{T+LI}\cdot\left(h_{TI}\oplus LI\right)+b_{T+LI},\end{split}$ (15)

where $W_{T+LI}\in\mathbb{R}^{|L|\times(2D_{T}+D_{L})}$ and
$b_{T+LI}\in\mathbb{R}^{|L|}$ represent the weight parameter and the bias
parameter in $f_{T+LI}(\cdot)$, respectively.

###  4.2 Counterfactual De-bias

Despite the extraction and utilization of more complete and precise LI in
Section 4.1, the predict-then-modify backbone still adheres to the LD-based
model’s decision causal graph in Fig. 2(b), so the label correlation bias
still exists. Therefore, we should block the label correlation shortcut to
eliminate this bias, as mentioned in Section 3.

To measure the effect of the label correlation shortcut, we assume a
counterfactual text $X^{*}$ for each text in Section 3. Ideally, LI-specific
counterfactual text information $h_{TI}^{*}$ can be obtained using a process
similar to that of extracting $h_{TI}$: first, the counterfactual text
information $H^{*}$ is extracted from $X^{*}$ using the text encoder in
Section 4.1.1, then, $h_{TI}^{*}$ is obtained by combining the existing LI
through the attention module in Section 4.1.3. However, the reality is that
the counterfactual texts do not exist in most datasets. Although
counterfactual data augmentation methods (Wang and Culotta, 2021; Chen et al,
2021b) are widely discussed, the introduction of data augmentation methods
would present additional challenges, for example, additional bias may be
created in the data augmentation process.

To solve this problem, we employ a general proxy text information
$h_{TI}^{{}^{\prime}}$ as an alternative to all counterfactual text
information. This method of designing proxy information for counterfactual
inference is widely employed in the field of computer vision (Tang et al,
2020; Niu et al, 2021; Yang et al, 2021). In this study, we extend this method
to NLP problem. According to the counterfactual intervention in Fig. 2(c), the
motivation for the intervention is to prevent the model from getting the
correct text information, rather than a semantic counterfactual text. Since
$h_{TI}^{{}^{\prime}}$ is general, it would not carry any text information for
a specific text, so this alternative fits the motivation. In this situation,
the model can only make predictions based on the correct LI.

Specifically, we design a trainable parameter
$h_{TI}^{{}^{\prime}}\in\mathbb{R}^{2D_{T}}$ to represent the proxy text
information, and imitating the counterfactual intervention process in Fig.
2(c), $h_{TI}$ in Eq. 15 is replaced by $h_{TI}^{{}^{\prime}}$, and the label
correlation scores can be obtained using $f_{T+LI}(\cdot)$ based on
$h_{TI}^{{}^{\prime}}$ and $LI$ as:

$\begin{split}s_{T^{*}+LI}&=f_{T+LI}(h_{TI}^{{}^{\prime}}\oplus LI)\\\
&=W_{T+LI}\cdot(h_{TI}^{{}^{\prime}}\oplus LI)+b_{T+LI}.\end{split}$ (16)

In the counterfactual inference, $s_{T^{*}+LI}$ is the estimate of the label
correlation shortcut. Based on the counterfactual de-bias technique in Eq. 5,
we explicitly block the label correlation shortcut by subtracting
$s_{T^{*}+LI}$ from $s_{T+LI}$, and the counterfactual de-bias score is
denoted as:

$s_{cd}=s_{T+LI}-s_{T^{*}+LI}.$ (17)

Essentially, the counterfactual de-bias process provides the human decision
logic to the LD-based model in the form of the causal graph, which prevents
the model from making inferences based on the correlation shortcut in
datasets. However, this decision logic is difficult to be learned by the model
through data-driven methods.

#### Mask Mechanism

The predict-then-modify backbone, while blocking the correlation shortcut
through the counterfactual de-bias technique, may cause the FI $\rightarrow$ Y
path in Fig. 2(b) to lack expressiveness. Specifically, during the training
process, CFTC could extract accurate LI with the help of the ground truth, and
accurate LI can then reconstruct the ground truth (Y $\rightarrow$ LI
$\rightarrow$ Y). As a result, the expressiveness of the FI $\rightarrow$ Y
path is likely to be weakened, because LI provides sufficient information.
Although the label correlation shortcut would be blocked by counterfactual de-
bias (Eq. 17), weakening the expressiveness of the FI $\rightarrow$ Y path is
harmful to CFTC, because it is the unique causal path in MLTC.

To strengthen the FI $\rightarrow$ Y path, we insert the Mask Mechanism (MM)
before the LI Extractor (Section 4.1.2). Here, mask means to invert the result
of whether the text matches the label or not. The purpose of the mask
mechanism is to actively create uncertainties in LI so that the text
information can be utilized to remove these uncertainties, which means the LD-
based model has to strengthen the FI $\rightarrow$ Y path for prediction.

We use two mask mechanisms in our CFTC: probability-based mask and random
mask. In the probability-based mask, labels with low prediction confidence are
more likely to be masked, and we achieve it through the Gumbel-Softmax trick
(Jang et al, 2017). Specifically, we achieve this by changing the probability
distribution of each label, when we get $s_{T}$ in Eq. 7, the probability
after the probability-based mask of each label is calculated as:

$\begin{split}\sigma_{T}^{i}&=\log(\text{Sigmoid}([s_{T}^{i},-s_{T}^{i}])),\\\
\widehat{p_{M}}^{i}&=\text{Softmax}((\sigma_{T}^{i}+g)/\tau),\end{split}$ (18)

where $g\in\mathbb{R}^{2}$ is sampled from Gumbel$(0,1)$ distribution333The
Gumbel$(0,1)$ distribution can be sampled using inverse transform sampling by
drawing $u\sim\text{Uniform}(0,1)$ and computing $g=-\log(-\log(u))$.  and
$\tau$ is the temperature to control the probability distribution. With the
Gumbel-Softmax trick, LI has a certain probability of containing the opposite
information to $s_{T}$, when the label’s confidence is low, LI is more
unreliable and the model should rely more on text information for prediction.

After the probability-based mask, in order to increase the diversity of LI, we
randomly mask a certain percentage of labels. For the selected label $L_{i}$,
the random mask is formalized as:

$\widehat{p_{M}}^{i}=1-p_{T}^{i}.$ (19)

After the mask mechanism, we can obtain the masked initial prediction
$\widehat{Y}_{m}$ by activation function Sigmoid($\cdot$) and the threshold
$\mu$. Because of the mask mechanism, LI extracted from $\widehat{Y}_{m}$
carries uncertainties, and we consider that these uncertainties facilitate the
text information to be fully utilized. The mask mechanism serves to weaken the
correlation shortcut and strengthen the causal path, which ensures the
counterfactual de-bias module in Fig. 2(c) could work properly.

###  4.3 Training and Inference

In this paper, $\text{pool}(\cdot)$ is mean-pooling function, and
$\text{Sigmoid}(\cdot)$ function converts the scores into probabilities of
prediction $\widehat{P}$. We use Binary Cross-Entropy Loss (BCELoss) to
calculate the loss of the predictions. Specifically,
$Y=\\{y_{1},y_{2},\dots,y_{L}\\}$ are the ground-truth labels of a text, where
$y_{i}=0,1$ denotes whether the text matches $L_{i}$ or not, and the loss can
be calculated as:

$\mathcal{L}=\frac{1}{L}\sum_{i=1}^{L}y_{i}\log(\widehat{p}_{i})+(1-y_{i})\log(1-\widehat{p}_{i}).$
(20)

In addition to supervising
$\\{\widehat{P}_{T},\widehat{P}_{T+LI},\widehat{P}_{cd}\\}$, due to the
introduction of the trainable counterfactual text information $h^{*}_{LI}$, we
also supervise $\widehat{P}_{T^{*}+LI}$. Therefore, the final loss is the
combination of four predictions:

$\mathcal{L}=\mathcal{L}_{T}+\alpha\cdot\mathcal{L}_{T+LI}+\beta\cdot\mathcal{L}_{T^{*}+LI}+\gamma\cdot\mathcal{L}_{cd},$
(21)

where $\alpha,\beta,\gamma$ are the weights of each loss, respectively. In the
inference process, we use $\widehat{P}_{cd}$ as predictions because the label
correlation shortcut should be blocked.

##  5 Experiments

###  5.1 Datasets

Table 2: Statistics of datasets. $N$ and $L$ denote the total number of samples and labels, respectively. $\overline{W}$ is the average number of words per sample, and $\overline{L}$ is the average number of labels per sample.  Dataset | $N$ | $L$ | $\overline{W}$ | $\overline{L}$ |
---|---|---|---|---|---
AAPD | 55840 | 54 | 163 | 2.41 |
RCV1 | 804414 | 103 | 124 | 3.24 |
Reuters-21578 | 10789 | 90 | 160 | 1.13 |

We conducted our experiments on three datasets:

Arxiv Academic Paper Dataset (AAPD) was built by (Yang et al, 2018). It
consists of abstracts and corresponding topics of papers in the field of
computer science, which is organized into 54 related topics. AAPD is available
in444https://git.uwaterloo.ca/jimmylin/Castor-data/tree/master/datasets/AAPD.

Reuters Corpus Volume I (RCV1) was built by (Lewis et al, 2004). It consists
of over 80K manually categorized news made available by Reuters Ltd for
research purposes, and each news is assigned multiple topics. RCV1 is
available
in555http://www.ai.mit.edu/projects/jmlr/papers/volume5/lewis04a/lyrl2004_rcv1v2_README.htm.

Reuters-21578 was built by (Lewis, 1997). Its initial version contains 21578
documents and 90 categories. After eliminating the documents without
categories, the final version contains 10788 documents. Reuters-21578 is
available
in666http://www.daviddlewis.com/resources/testcollections/reuters21578/.

The statistics of the datasets are listed in Table 2.

###  5.2 Evaluation Metrics

Following the previous works (Yang et al, 2018; Wang et al, 2021a), we adopted
Hamming Loss (Schapire and Singer, 1998) and Micro-$F_{1}$ as our main
evaluation metrics. We also recorded Micro-$P$ and Micro-$R$ as secondary
metrics to further analyze the experimental results.

Hamming Loss is the fraction of labels that are incorrectly predicted,
including predicted irrelevant labels and missed relevant labels. It can be
computed as:

$\text{Hamming
Loss}(Y,\widehat{Y})=\frac{1}{L}\sum_{i=1}^{L}\mathbf{1}(y_{i},\widehat{y}_{i}),$
(22)

where $\mathbf{1}(a,b)$ is the indicator function. The indicator is equal to 1
if $a=b$, otherwise it is equal to 0.

Micro-$P$, Micro-$R$ and Micro-$F_{1}$ are common evaluation metrics in
classification tasks. Specifically, given True Positive $TP_{i}$, False
Positive $FP_{i}$, False Negative $FN_{i}$, and True Negative $TN_{i}$ for
class $i$, Micro-$P$, Micro-$R$ and Micro-$F_{1}$ can be represented as:

$\begin{split}\text{Micro-}P&=\frac{\sum_{i=1}^{L}TP_{i}}{\sum_{i=1}^{L}TP_{i}+FP_{i}},\\\
\text{Micro-}R&=\frac{\sum_{i=1}^{L}TP_{i}}{\sum_{i=1}^{L}TP_{i}+FN_{i}},\\\
\text{Micro-}F_{1}&=\frac{\sum_{i=1}^{L}2TP_{i}}{\sum_{i=1}^{L}2TP_{i}+FP_{i}+FN_{i}}.\end{split}$
(23)

###  5.3 Baselines

In order to verify the effectiveness of CFTC, we selected several multi-label
classification algorithms as baselines. We divided the baselines into two
groups based on whether label dependency is utilized.

The first group of baselines did not use label dependency, and only focus on
texts:

  * •

Binary Relevance (BR) (Gonçalves and Quaresma, 2003) trains a binary
classifier (linear SVM) for each label, and each classifier is independent;

  * •

Label Powerset (LP) (Boutell et al, 2004) views MLTC as a multi-class
classification problem;

  * •

CNN (Kim, 2014) extracts text features by multiple convolution kernels, which
is a common way to extract text features;

  * •

BiLSTM (Hochreiter and Schmidhuber, 1997) applies a Long Short-Term Memory
network to extract text features, and this approach takes into account the
sequential structure of the text;

  * •

CNN-RNN (Chen et al, 2017) extracts local and global text features by CNNs and
RNNs;

  * •

BERT (Devlin et al, 2019) applies Bidirectional Encoder Representations from
Transformers to extract text features, and BERT is pre-trained on a large-
scale corpus.

The second group of baselines attempted to utilize label dependency, and
except for LBA, none of these methods employed pre-trained language models:

  * •

Classifier Chains (CC) (Read et al, 2009) transforms the MLTC problem into a
sequence of binary classification tasks;

  * •

ML-GCN (Chen et al, 2019) captures and explores label dependency by Graph
Convolutional Network and co-occurrence matrix;

  * •

LSAN (Xiao et al, 2019) learns the label-specific text features with the help
of self-attention and label-attention mechanism;

  * •

SGM (Yang et al, 2018) views MLTC as a sequence generation task and utilizes
seq2seq model as a multi-class classifier to use label dependency;

  * •

ML-Reasoner (ML-R) (Wang et al, 2021a) uses the label predictions from the
previous round to utilize label dependency;

  * •

LBA (Liu et al, 2021b) designs the bi-directional attentive module for the
finer-grained token-level text representation and label embedding to utilize
label dependency.

###  5.4 Implementation Details

Table 3: Experiment results of different models on AAPD. The best performance is highlighted in bold, and the best performance without the pre-trained language model is highlighted by underline.  Models | Hamming Loss $\downarrow$ | Micro-$P$ $\uparrow$ | Micro-$R$ $\uparrow$ | Micro-$\bm{F_{1}}$ $\uparrow$
---|---|---|---|---
_w/o LD_ |  |  |  |
BR | 0.0266 | 71.0 | 63.2 | 66.9
LP | 0.0255 | 74.5 | 65.5 | 69.7
CNN | 0.0259 | 72.8 | 67.6 | 70.0
BiLSTM | 0.0254 | 74.3 | 67.2 | 70.3
CNN-RNN | 0.0261 | 72.6 | 66.9 | 69.7
BERT | 0.0230 | 79.1 | 66.3 | 72.2
_LD-based_ |  |  |  |
CC | 0.0256 | 75.8 | 62.9 | 66.8
ML-GCN | 0.0247 | 78.9 | 61.3 | 69.0
LSAN | 0.0246 | 75.2 | 67.5 | 70.9
SGM | 0.0251 | 74.8 | 67.5 | 71.0
ML-R | 0.0255 | 74.6 | 65.5 | 69.8
LBA | 0.0228 | 78.8 | 67.0 | 72.1
$\text{CFTC}_{BiLSTM}$ | 0.0237 | 77.0 | 66.6 | 71.4
$\text{CFTC}_{BERT}$ | 0.0222 | 79.3 | 68.4 | 73.4

BiLSTM and BERT were employed as the encoders in our CFTC to extract text
features, respectively. We chose a 3-layer GCN to transform the initial
prediction into LI, and the hidden size of GCNs was set to 300. The word
embedding dimension was 300 for BiLSTM and 768 for BERT. We set the batch size
to 64 (16 for BERT), and the learning rate of the Adam optimizer to 1e-4 (5e-5
for BERT). After training models for 50 epochs (10 epochs for BERT), we
selected the best model on the training set for testing. Since the text
encoder part served to extract text features and was prone to overfitting, we
optimized the encoder by $\mathcal{L}_{T}$ and optimized the decoder by
$\mathcal{L}_{T+LI}$, $\mathcal{L}_{T^{*}+LI}$ and $\mathcal{L}_{cd}$, or pre-
trained the encoder and froze the parameters on training. In this paper, we
masked 5% of labels in the mask mechanism, the threshold of probability was
set to 0.5, and we set the weight $\alpha=\beta=0.1$ and $\gamma=1.0$ in the
training process.

##  6 Results and Analysis

This section is mainly about the performance of CFTC. We reported the main
experimental results of CFTC and other baselines on three datasets (Section
6.1). Besides, we validated the contribution of each module in CFTC through
ablation experiments (Section 6.2). To demonstrate the effectiveness of the
counterfactual de-bias technique, we compared the difference in label co-
occurrence frequencies before and after counterfactual de-bias in CFTC
(Section 6.3). We showed the advantages of CFTC in eliminating the label
correlation bias through case study (Section 6.4).

Table 4: Experiment results of different models on RCV1. The best performance is highlighted in bold, and the best performance without the pre-trained language model is highlighted by underline.  Models | Hamming Loss $\downarrow$ | Micro-$P$ $\uparrow$ | Micro-$R$ $\uparrow$ | Micro-$\bm{F_{1}}$ $\uparrow$
---|---|---|---|---
_w/o LD_ |  |  |  |
BR | 0.0086 | 90.4 | 81.6 | 85.8
LP | 0.0087 | 89.6 | 82.4 | 85.8
CNN | 0.0083 | 88.1 | 85.1 | 86.6
BiLSTM | 0.0079 | 89.0 | 85.1 | 87.0
CNN-RNN | 0.0087 | 87.8 | 83.8 | 85.8
BERT | 0.0071 | 90.5 | 86.5 | 88.5
_LD-based_ |  |  |  |
CC | 0.0087 | 88.7 | 82.8 | 85.7
ML-GCN | 0.0080 | 88.0 | 86.1 | 87.0
LSAN | 0.0079 | 91.3 | 82.5 | 86.7
SGM | 0.0079 | 88.5 | 86.0 | 87.2
ML-R | 0.0079 | 89.7 | 84.5 | 87.0
LBA | 0.0073 | 90.0 | 85.9 | 88.0
$\text{CFTC}_{BiLSTM}$ | 0.0074 | 89.3 | 86.1 | 88.0
$\text{CFTC}_{BERT}$ | 0.0068 | 90.5 | 87.4 | 88.9
Table 5: Experiment results of different models on Reuters-21578. The best performance is highlighted in bold, and the best performance without the pre-trained language model is highlighted by underline.  Models | Hamming Loss $\downarrow$ | Micro-$P$ $\uparrow$ | Micro-$R$ $\uparrow$ | Micro-$\bm{F_{1}}$ $\uparrow$
---|---|---|---|---
_w/o LD_ |  |  |  |
BR | 0.0049 | 86.2 | 78.8 | 82.3
LP | 0.0054 | 78.7 | 81.0 | 79.8
CNN | 0.0038 | 89.0 | 82.3 | 85.5
BiLSTM | 0.0040 | 90.4 | 78.4 | 84.0
CNN-RNN | 0.0038 | 90.3 | 81.3 | 85.5
BERT | 0.0031 | 93.4 | 83.3 | 88.0
_LD-based_ |  |  |  |
CC | 0.0045 | 85.2 | 81.7 | 83.4
ML-GCN | 0.0043 | 86.0 | 81.8 | 83.5
LSAN | 0.0041 | 87.4 | 82.1 | 84.7
SGM | 0.0052 | 80.7 | 75.9 | 78.8
ML-R | 0.0041 | 91.3 | 77.5 | 83.8
LBA | 0.0030 | 92.7 | 86.5 | 88.5
$\text{CFTC}_{BiLSTM}$ | 0.0037 | 91.4 | 81.5 | 86.2
$\text{CFTC}_{BERT}$ | 0.0029 | 91.7 | 87.8 | 88.8

###  6.1 Main Results

We compared the performance of CFTC as well as other compared baselines on
AAPD, RCV1 and Reuters-21578. It can be observed that CFTC outperformed all
other baselines in most metrics, and the results confirmed the effectiveness
of CFTC.

#### Results on AAPD

As shown in Table 3, our CFTC outperformed all other baselines by a
significant margin on AAPD dataset. In particular, CFTC with BERT performed
best: Hamming Loss is 0.0222, which is a 3.5% improvement over the best result
in baselines, and Micro-$F_{1}$ also received a 1.2% (absolute) improvement.
In addition, our CFTC with two text encoders (BiLSTM, BERT) outperformed the
corresponding encoder baselines, the results showed that our CFTC is effective
for different text encoders.

#### Results on RCV1

Compared to AAPD dataset, the performance difference between models on RCV1
was reduced, but our CFTC continued to perform better than other baselines on
main metrics. As shown in Table 4, CFTC with BiLSTM exceeded the baseline
BiLSTM by 6.3% on Hamming Loss and by 1.0% (absolute) on Micro-$F_{1}$, and
CFTC with BERT beat the baseline BERT by 4.2% on Hamming Loss and by 0.4%
(absolute) on Micro-$F_{1}$.

#### Results on Reuters-21578

As shown in Table 5, our CFTC achieved better performance on Reuters-21578
dataset. Compared with the baseline BiLSTM, CFTC with BiLSTM achieved a 7.5%
improvement on Hamming Loss and a 2.2% (absolute) improvement on
Micro-$F_{1}$. Also, CFTC with BERT also defeated the baseline BERT, leading
by 6.5% on Hamming Loss and by 0.8% (absolute) on Micro-$F_{1}$.

###  6.2 Ablation Study

Table 6: Ablation analysis on main mechanisms of our framework on AAPD and
RCV1. $\backslash$ denotes the removing operation. MM and CD denote the Mask
Mechanism and Counterfactual De-bias, respectively

.  Models AAPD RCV1 Hamming Loss $\downarrow$ Micro-$\bm{F_{1}}$ $\uparrow$
Hamming Loss $\downarrow$ Micro-$\bm{F_{1}}$ $\uparrow$ $\text{CFTC}_{BiLSTM}$
0.0237 71.4 0.0074 88.0 BiLSTM 0.0254 70.3 0.0079 87.0 $\backslash$ MM 0.0241
71.1 0.0076 87.6 $\backslash$ CD 0.0242 70.9 0.0077 87.4 $\backslash$ MM&CD
0.0249 70.8 0.0079 87.1

We investigated the independent impact of each module in our proposed CFTC.
The results are reported in Table 6. When we only employed the predict-then-
modify backbone (`\`MM&CD), the model performed slightly better than BiLSTM,
which illustrated the role of the precise LI. When we removed the Mask
Mechanism module (`\`MM) and Counterfactual De-bias module (`\`CD), the model
performance decreased significantly compared to $\text{CFTC}_{BiLSTM}$. This
showed the effectiveness of both modules in the counterfactual de-bias part.

Further, CFTC was more effective on AAPD dataset compared to RCV1 dataset.
This result implied that the LD-based model was more susceptible to the
correlation shortcut on smaller datasets, and as the size of the datasets
increased, the model became more precise in mining the underlying mechanisms
of the task, thus reducing the reliance on the correlation shortcut.

###  6.3 Label Co-occurrence Frequency

Figure 4: Co-occurrence frequency of some labels on AAPD dataset. The first
column is the true co-occurrence frequency in the dataset, and the second and
third columns are the co-occurrence frequencies of the predictions before and
after Counterfactual De-bias (CD). Significant differences are boxed in red.

A notable manifestation of the label correlation bias is the LD-based model’s
tendency to generate high-frequency label combinations and ignore low-
frequency label combinations. Therefore, the label co-occurrence frequency
could, to some extent, reflect the effect of the label correlation bias.

To demonstrate the effectiveness of the counterfactual de-bias technique in
CFTC, we compared the co-occurrence frequencies of some labels (‘cs.ai’,
‘cs.ds’, ‘cs.ne’) before and after counterfactual de-bias in Fig. 4. The first
column showed the true co-occurrence frequency in AAPD. The second and third
columns showed the co-occurrence frequencies of the predictions before and
after counterfactual de-bias. It can be seen that the predictions before de-
bias tended to generate high-frequency label combinations, and a large number
of low-frequency label combinations were not captured. This reflected that the
LD-based model was heavily influenced by the label correlation bias. In
contrast, the predictions after counterfactual de-bias showed higher
similarity to the true co-occurrence frequency, which reflected the effect of
the counterfactual de-bias technique in eliminating the label correlation
bias.

Table 7: The predictions of CFTC with different LI. We obtain different LI by
changing ‘Given Labels’. ①, ②, ③, ④ refer to cs.lg, st.ml, cs.it, ma.it,
respectively. $\varnothing$ means empty set.  Ground Truth: ①, ② $Y_{T}$: ①,
②, ③, ④
---
Given Labels | $\bm{Y}_{T^{*}+LI}$ | $\bm{Y}_{T+LI}$ | $\bm{Y}_{cd}$
①, ②, ③, ④ |  ①, ③, ④ |  ①, ②, ③, ④ |  ①, ②
①, ② |  ①, ② |  ①, ② |  ①, ②
①, ④ | ① |  ①, ② |  ①, ②
$\varnothing$ | $\varnothing$ | ② |  ①, ②
③, ④ |  ③, ④ |  ③, ④ |  ①, ②, ③, ④
Figure 5: The visualization of the normalized label co-occurrence matrix of
the four mentioned labels. The lighter the color, the higher the frequency of
co-occurrence between labels

###  6.4 Case Study

####  6.4.1 Different Label Information

To demonstrate that CFTC is able to use LI accurately, we selected a specific
text and compared predictions in the case of different LI in Table 7. The text
can be found in Appendix A. We obtained different LI by changing the given
labels fed to the LI extractor (Although the given labels in training process
are $Y_{T}$, CFTC supports arbitrarily changing the given labels in testing
process).

Row 1 was the prediction without changing LI (Given Labels was $Y_{T}$),
compared to ground truth (①, ②), $Y_{T}$ had two more error labels (③, ④), and
the error labels were retained in $Y_{T+LI}$, which reflected the effect of LI
on $Y_{T+LI}$. However, after counterfactual de-bias, the error labels were
corrected in $Y_{cd}$. To compare the effect of LI on predictions, we changed
LI by selecting a portion of the labels in $Y_{T}$ as Given Labels and
compared the three predictions $Y_{T^{*}+LI}$, $Y_{T+LI}$, $Y_{cd}$. To help
readers understand the correlation between these labels, we visualized the
normalized label co-occurrence matrix of these four labels in Fig. 5. When
feeding the correct LI to CFTC (row 2), all predictions were correct, which
was in line with our expectations. Further, when LI was partially correct (row
3), $Y_{T+LI}$ and $Y_{cd}$ were both correct, while $Y_{T^{*}+LI}$ was
incorrect, which indicated that the text was useful for predicting $Y_{T+LI}$
and $Y_{cd}$. However, when LI was incorrect, $Y_{cd}$ would be much more
accurate than $Y_{T+LI}$. More specifically, missing LI (row 4) caused
$Y_{T+LI}$ to be incomplete, while $Y_{cd}$ was correct, and when LI was
completely incorrect (row 5), $Y_{T+LI}$ got the completely incorrect
prediction consistent with Given Labels, while $Y_{cd}$ contained both the
correct labels and Given Labels. This result illustrated that $Y_{T+LI}$
tended to make predictions dominated by LI, while $Y_{cd}$ would fuse text and
LI to make causality-based predictions. In fact, this is evidence that CFTC
has succeeded in blocking the label correlation shortcut.

####  6.4.2 Different Models

Table 8: Four examples of labels generated by different models on AAPD
dataset. The same/different colored labels indicate that they are highly
correlated/uncorrelated.

# | Ground Truth | $\bm{Y}_{BiLSTM}$ | $\bm{Y}_{SGM}$ | $\bm{Y}_{CFTC}$
---|---|---|---|---
1 |  cs.it, ma.it |  cs.it, ma.it, cs.ni |  cs.it, ma.it |  cs.it, ma.it
2 |  cs.ai, cs.pl |  cs.ai, cs.pl |  cs.ai, cs.lo, cs.pl |  cs.ai, cs.pl
3 |  cs.it, ma.it, ma.oc, ma.pr |  cs.it, ma.it, ma.oc, ma.pr |  cs.it, ma.it |  cs.it, ma.it, ma.oc, ma.pr
4 |  cs.cl, cs.ir |  cs.cl, cs.ir, cmp-lg |  cs.cl, cmp-lg |  cs.cl, cs.ir

We selected several examples and showed the prediction results of different
models for comparison in Table 8, where BiLSTM did not make use of label
information and SGM did. All texts can be found in Appendix A. The
experimental results illustrated that CFTC outperformed other two models in
these cases. Meanwhile, we could find the effect of the label correlation bias
on the LD-based models. As shown in Table 8, in most cases, ground truths were
correlated with each other, and more accurate predictions can often be
obtained by the LD-based model (row 1). However, when ground truths were
uncorrelated (rows 2, 3, 4), the LD-based model tended to predict the
incorrect labels with high correlation ($Y_{SGM}$ in rows 2, 4) or miss
correct labels with low correlation ($Y_{SGM}$ in rows 3, 4) due to the
presence of the label correlation shortcut, while the model without the label
dependency performed relatively better ($Y_{BiLSTM}$ in rows 2, 3, 4). Our
CFTC could use the label dependency accurately, so the performance of CFTC
outperformed the other two baselines in all cases.

##  7 Conclusion

In this paper, we attributed the bias of LD-based models to the label
correlation shortcut. To avoid this unwanted bias, we designed a
counterfactual framework with the predict-then-modify backbone named CFTC to
obtain causality-based predictions. The experimental results showed that CFTC
effectively blocked the label correlation shortcut and achieved competitive
performance. In the future, we hope that our proposed CFTC can be applied in
more NLP scenarios and help deep learning models to better grasp the
underlying mechanisms of NLP tasks.

#### Competing interests

This work was supported by the Shanghai Municipal Science and Technology Major
Project (2021SHZDZX0102), and the Shanghai Science and Technology Innovation
Action Plan (20511102600).

##  Appendix A Text Contents in Analysis

In this paper, we selected several representative samples from AAPD (Yang et
al, 2018) to analyze the model performance. We showed these contents in Table
9 & 10. AAPD contained the abstract and the corresponding subjects of 55840
papers in the computer science field from the website.

By comparing the predictions obtained from these samples, we verified that
there was unwanted bias in the LD-based model due to the label correlation
shortcut, and our CFTC could alleviate this bias and obtain causality-based
predictions.

Table 9: The text content of the samples selected in Section 6.4.1.  # |  Text
---|---
1 |  This paper describes a simple framework for structured sparse recovery based on convex optimization. We show that many structured sparsity models can be naturally represented by linear matrix inequalities on the support of the unknown parameters, where the constraint matrix has a totally unimodular (TU) structure. For such structured models, tight convex relaxations can be obtained in polynomial time via linear programming. Our modeling framework unifies the prevalent structured sparsity norms in the literature, introduces new interesting ones, and renders their tightness and tractability arguments transparent.
Table 10: The text content of the samples selected in Section 6.4.2.  # |  Text
---|---
1 |  We consider a scenario where a monitor is interested in being up to date with respect to the status of some system which is not directly accessible to this monitor. However, we assume a source node has access to the status and can send status updates as packets to the monitor through a communication system. We also assume that the status updates are generated randomly as a Poisson process. The source node can manage the packet transmission to minimize the age of information at the destination node, which is defined as the time elapsed since the last successfully transmitted update was generated at the source. We use queuing theory to model the source-destination link and we assume that the time to successfully transmit a packet is a gamma distributed service time. We consider two packet management schemes: LCFS (Last Come First Served) with preemption and LCFS without preemption. We compute and analyze the average age and the average peak age of information under these assumptions. Moreover, we extend these results to the case where the service time is deterministic.
2 |  The most advanced implementation of adaptive constraint processing with Constraint Handling Rules (CHR) allows the application of intelligent search strategies to solve Constraint Satisfaction Problems (CSP). This presentation compares an improved version of conflict-directed backjumping and two variants of dynamic backtracking with respect to chronological backtracking on some of the AIM instances which are a benchmark set of random 3-SAT problems. A CHR implementation of a Boolean constraint solver combined with these different search strategies in Java is thus being compared with a CHR implementation of the same Boolean constraint solver combined with chronological backtracking in SICStus Prolog. This comparison shows that the addition of “intelligence” to the search process may reduce the number of search steps dramatically. Furthermore, the runtime of their Java implementations is in most cases faster than the implementations of chronological backtracking. More specifically, conflict-directed backjumping is even faster than the SICStus Prolog implementation of chronological backtracking, although our Java implementation of CHR lacks the optimisations made in the SICStus Prolog system. To appear in Theory and Practice of Logic Programming (TPLP).
3 |  In this paper we look at isometry properties of random matrices. During the last decade these properties gained a lot attention in a field called compressed sensing in first place due to their initial use in [7, 8]. Namely, in [7, 8] these quantities were used as a critical tool in providing a rigorous analysis of $l_{1}$ optimization’s ability to solve an under-determined system of linear equations with sparse solutions. In such a framework a particular type of isometry, called restricted isometry, plays a key role. One then typically introduces a couple of quantities, called upper and lower restricted isometry constants to characterize the isometry properties of random matrices. Those constants are then usually viewed as mathematical objects of interest and their a precise characterization is desirable. The first estimates of these quantities within compressed sensing were given in [7, 8]. As the need for precisely estimating them grew further a finer improvements of these initial estimates were obtained in e.g. [2, 4]. These are typically obtained through a combination of union-bounding strategy and powerful tail estimates of extreme eigenvalues of Wishart (Gaussian) matrices (see, e.g. [19]). In this paper we attempt to circumvent such an approach and provide an alternative way to obtain similar estimates.
4 |  This article evaluates the performance of two techniques for query reformulation in a system for information retrieval, namely, the concept based and the pseudo relevance feedback reformulation. The experiments performed on a corpus of Arabic text have allowed us to compare the contribution of these two reformulation techniques in improving the performance of an information retrieval system for Arabic texts.

##  Appendix B Label Co-occurrence Matrix

To capture the implied interactions of labels, we employed the label co-
occurrence matrix (Ma et al, 2021; Liu et al, 2021a) as prior knowledge and
applied a graph neural network to extract deeper label information. The label
co-occurrence matrix $A\in\mathbb{R}^{|L|\times|L|}$ is the statistic of co-
occurrence between labels, where $A_{ij}$ denotes the conditional probability
of a text belonging to label $L_{i}$ when it belongs to label $L_{j}$. We
counted the label co-occurrence matrices of AAPD, RCV1 and Reuters-21578, and
visualized them as in Fig. 6.

(a) AAPD

(b) RCV1

(c) Reuters-21578

Figure 6: The label co-occurrence matrix of AAPD, RCV1 and Reuters-21578.

The results showed the existence of sparse co-occurrence relationships between
the labels, and this particular relationship can provide additional
information to the model in Multi-Label Text Classification tasks.

## References

  *   * Alswaidan and Menai (2020) Alswaidan N, Menai MEB (2020) A survey of state-of-the-art approaches for emotion recognition in text. Knowl Inf Syst 62(8):2937–2987. 10.1007/s10115-020-01449-0, URL https://doi.org/10.1007/s10115-020-01449-0
  * Boutell et al (2004) Boutell MR, Luo J, Shen X, et al (2004) Learning multi-label scene classification. Pattern Recognit 37(9):1757–1771. 10.1016/j.patcog.2004.03.009, URL https://doi.org/10.1016/j.patcog.2004.03.009
  * Chen et al (2020) Chen B, Huang X, Xiao L, et al (2020) Hyperbolic capsule networks for multi-label classification. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, pp 3115–3124, 10.18653/v1/2020.acl-main.283, URL https://doi.org/10.18653/v1/2020.acl-main.283
  * Chen et al (2017) Chen G, Ye D, Xing Z, et al (2017) Ensemble application of convolutional and recurrent neural networks for multi-label text categorization. In: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017. IEEE, pp 2377–2383, 10.1109/IJCNN.2017.7966144, URL https://doi.org/10.1109/IJCNN.2017.7966144
  * Chen et al (2021a) Chen H, Ma Q, Lin Z, et al (2021a) Hierarchy-aware label semantics matching network for hierarchical text classification. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. Association for Computational Linguistics, pp 4370–4379, 10.18653/v1/2021.acl-long.337, URL https://doi.org/10.18653/v1/2021.acl-long.337
  * Chen et al (2021b) Chen H, Xia R, Yu J (2021b) Reinforced counterfactual data augmentation for dual sentiment classification. In: Moens M, Huang X, Specia L, et al (eds) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computational Linguistics, pp 269–278, 10.18653/v1/2021.emnlp-main.24, URL https://doi.org/10.18653/v1/2021.emnlp-main.24
  * Chen and Ren (2021) Chen Z, Ren J (2021) Multi-label text classification with latent word-wise label information. Applied Intelligence 51(2):966–979. 10.1007/s10489-020-01838-6, URL https://doi.org/10.1007/s10489-020-01838-6
  * Chen et al (2019) Chen Z, Wei X, Wang P, et al (2019) Multi-label image recognition with graph convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, pp 5177–5186, 10.1109/CVPR.2019.00532, URL http://openaccess.thecvf.com/content_CVPR_2019/html/Chen_Multi-Label_Image_Recognition_With_Graph_Convolutional_Networks_CVPR_2019_paper.html
  * Cheng et al (2022) Cheng D, Li J, Liu L, et al (2022) Sufficient dimension reduction for average causal effect estimation. Data Min Knowl Discov 36(3):1174–1196. 10.1007/s10618-022-00832-5, URL https://doi.org/10.1007/s10618-022-00832-5
  * Devlin et al (2019) Devlin J, Chang M, Lee K, et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp 4171–4186, 10.18653/v1/n19-1423, URL https://doi.org/10.18653/v1/n19-1423
  * Du et al (2021) Du M, Manjunatha V, Jain R, et al (2021) Towards interpreting and mitigating shortcut learning behavior of NLU models. In: Toutanova K, Rumshisky A, Zettlemoyer L, et al (eds) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. Association for Computational Linguistics, pp 915–929, 10.18653/v1/2021.naacl-main.71, URL https://doi.org/10.18653/v1/2021.naacl-main.71
  * Feder et al (2021) Feder A, Oved N, Shalit U, et al (2021) Causalm: Causal model explanation through counterfactual language models. Comput Linguistics 47(2):333–386. 10.1162/coli_a_00404, URL https://doi.org/10.1162/coli_a_00404
  * Gonçalves and Quaresma (2003) Gonçalves T, Quaresma P (2003) A preliminary approach to the multilabel classification problem of portuguese juridical documents. In: Moura-Pires F, Abreu S (eds) Progress in Artificial Intelligence, 11th Protuguese Conference on Artificial Intelligence, EPIA 2003, Beja, Portugal, December 4-7, 2003, Proceedings, Lecture Notes in Computer Science, vol 2902. Springer, pp 435–444, 10.1007/978-3-540-24580-3_50, URL https://doi.org/10.1007/978-3-540-24580-3_50
  * Hochreiter and Schmidhuber (1997) Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. 10.1162/neco.1997.9.8.1735, URL https://doi.org/10.1162/neco.1997.9.8.1735
  * Jang et al (2017) Jang E, Gu S, Poole B (2017) Categorical reparameterization with gumbel-softmax. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, pp 5434–5445, URL https://openreview.net/forum?id=rkE3y85ee
  * Kim (2014) Kim Y (2014) Convolutional neural networks for sentence classification. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, pp 1746–1751, 10.3115/v1/d14-1181, URL https://doi.org/10.3115/v1/d14-1181
  * Kipf and Welling (2017) Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, pp 1746–1751, URL https://openreview.net/forum?id=SJU4ayYgl
  * Lewis (1997) Lewis DD (1997) Reuters-21578 text categorization test collection, distribution 1.0. In: Reuters Ltd.
  * Lewis et al (2004) Lewis DD, Yang Y, Rose TG, et al (2004) RCV1: A new benchmark collection for text categorization research. J Mach Learn Res 5:361–397. URL http://jmlr.org/papers/volume5/lewis04a/lewis04a.pdf
  * Li and Yue (2020) Li L, Yue W (2020) Dynamic uncertain causality graph based on intuitionistic fuzzy sets and its application to root cause analysis. Applied Intelligence 50(1):241–255. 10.1007/s10489-019-01520-6, URL https://doi.org/10.1007/s10489-019-01520-6
  * Liu et al (2021a) Liu H, Chen G, Li P, et al (2021a) Multi-label text classification via joint learning from label embedding and label correlation. Neurocomputing 460:385–398. 10.1016/j.neucom.2021.07.031, URL https://doi.org/10.1016/j.neucom.2021.07.031
  * Liu et al (2021b) Liu N, Wang Q, Ren J (2021b) Label-embedding bi-directional attentive model for multi-label text classification. Neural Process Lett 53(1):375–389. 10.1007/s11063-020-10411-8, URL https://doi.org/10.1007/s11063-020-10411-8
  * Luo et al (2019) Luo G, Zhao B, Du S (2019) Causal inference and bayesian network structure learning from nominal data. Applied Intelligence 49(1):253–264. 10.1007/s10489-018-1274-3, URL https://doi.org/10.1007/s10489-018-1274-3
  * Ma et al (2021) Ma Q, Yuan C, Zhou W, et al (2021) Label-specific dual graph neural network for multi-label text classification. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. Association for Computational Linguistics, pp 3855–3864, 10.18653/v1/2021.acl-long.298, URL https://doi.org/10.18653/v1/2021.acl-long.298
  * Nam et al (2017) Nam J, Mencía EL, Kim HJ, et al (2017) Maximizing subset accuracy with recurrent neural networks in multi-label classification. In: Guyon I, von Luxburg U, Bengio S, et al (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 5413–5423, URL https://proceedings.neurips.cc/paper/2017/hash/2eb5657d37f474e4c4cf01e4882b8962-Abstract.html
  * Niu et al (2021) Niu Y, Tang K, Zhang H, et al (2021) Counterfactual VQA: A cause-effect look at language bias. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, pp 12,700–12,710, 10.1109/CVPR46437.2021.01251, URL https://openaccess.thecvf.com/content/CVPR2021/html/Niu_Counterfactual_VQA_A_Cause-Effect_Look_at_Language_Bias_CVPR_2021_paper.html
  * Niven and Kao (2019) Niven T, Kao H (2019) Probing neural network comprehension of natural language arguments. In: Korhonen A, Traum DR, Màrquez L (eds) Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguistics, pp 4658–4664, 10.18653/v1/p19-1459, URL https://doi.org/10.18653/v1/p19-1459
  * Ozmen et al (2022) Ozmen M, Zhang H, Wang P, et al (2022) Multi-relation message passing for multi-label text classification. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022. IEEE, pp 3583–3587, 10.1109/ICASSP43922.2022.9747225, URL https://doi.org/10.1109/ICASSP43922.2022.9747225
  * Paranjape et al (2022) Paranjape B, Lamm M, Tenney I (2022) Retrieval-guided counterfactual generation for QA. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, pp 1670–1686, 10.18653/v1/2022.acl-long.117, URL https://doi.org/10.18653/v1/2022.acl-long.117
  * Qian et al (2021) Qian C, Feng F, Wen L, et al (2021) Counterfactual inference for text classification debiasing. In: Zong C, Xia F, Li W, et al (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. Association for Computational Linguistics, pp 5434–5445, 10.18653/v1/2021.acl-long.422, URL https://doi.org/10.18653/v1/2021.acl-long.422
  * Read et al (2009) Read J, Pfahringer B, Holmes G, et al (2009) Classifier chains for multi-label classification. In: Buntine WL, Grobelnik M, Mladenic D, et al (eds) Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Proceedings, Part II, Lecture Notes in Computer Science, vol 5782. Springer, pp 254–269, 10.1007/978-3-642-04174-7_17, URL https://doi.org/10.1007/978-3-642-04174-7_17
  * Schapire and Singer (1998) Schapire RE, Singer Y (1998) Improved boosting algorithms using confidence-rated predictions. In: Bartlett PL, Mansour Y (eds) Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT 1998, Madison, Wisconsin, USA, July 24-26, 1998. ACM, pp 80–91, 10.1145/279943.279960, URL https://doi.org/10.1145/279943.279960
  * Shah et al (2020) Shah H, Tamuly K, Raghunathan A, et al (2020) The pitfalls of simplicity bias in neural networks. In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates, Inc., pp 9573–9585, URL https://proceedings.neurips.cc/paper/2020/file/6cfe0e6127fa25df2a0ef2ae1067d915-Paper.pdf
  * Tang et al (2020) Tang K, Huang J, Zhang H (2020) Long-tailed classification by keeping the good and removing the bad momentum causal effect. In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, URL https://proceedings.neurips.cc/paper/2020/hash/1091660f3dff84fd648efe31391c5524-Abstract.html
  * Tsai and Lee (2020) Tsai C, Lee H (2020) Order-free learning alleviating exposure bias in multi-label classification. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, pp 6038–6045, URL https://ojs.aaai.org/index.php/AAAI/article/view/6066
  * Tsoumakas and Katakis (2007) Tsoumakas G, Katakis I (2007) Multi-label classification: An overview. Int J Data Warehous Min 3(3):1–13. 10.4018/jdwm.2007070101, URL https://doi.org/10.4018/jdwm.2007070101
  * Vu et al (2022) Vu HT, Nguyen MT, Nguyen VC, et al (2022) Label-representative graph convolutional network for multi-label text classification. Applied Intelligence https://doi.org/10.1007/s10489-022-04106-x
  * Wang et al (2022) Wang C, Liu L, Sun S, et al (2022) Rethinking the framework constructed by counterfactual functional model. Applied Intelligence 52(11):12,957–12,974. 10.1007/s10489-022-03161-8, URL https://doi.org/10.1007/s10489-022-03161-8
  * Wang et al (2021a) Wang R, Ridley R, Su X, et al (2021a) A novel reasoning mechanism for multi-label text classification. Inf Process Manag 58(2):102,441. 10.1016/j.ipm.2020.102441, URL https://doi.org/10.1016/j.ipm.2020.102441
  * Wang et al (2019) Wang S, Cai J, Lin Q, et al (2019) An overview of unsupervised deep feature representation for text categorization. IEEE Trans Comput Soc Syst 6(3):504–517. 10.1109/TCSS.2019.2910599, URL https://doi.org/10.1109/TCSS.2019.2910599
  * Wang et al (2020) Wang T, Liu L, Liu N, et al (2020) A multi-label text classification method via dynamic semantic representation model and deep neural network. Applied Intelligence 50(8):2339–2351. 10.1007/s10489-020-01680-w, URL https://doi.org/10.1007/s10489-020-01680-w
  * Wang et al (2021b) Wang W, Feng F, He X, et al (2021b) Deconfounded recommendation for alleviating bias amplification. In: Zhu F, Ooi BC, Miao C (eds) KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021. ACM, pp 1717–1725, 10.1145/3447548.3467249, URL https://doi.org/10.1145/3447548.3467249
  * Wang et al (2021c) Wang W, Feng F, He X, et al (2021c) Clicks can be cheating: Counterfactual recommendation for mitigating clickbait issue. In: Diaz F, Shah C, Suel T, et al (eds) SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. ACM, pp 1288–1297, 10.1145/3404835.3462962, URL https://doi.org/10.1145/3404835.3462962
  * Wang and Culotta (2021) Wang Z, Culotta A (2021) Robustness to spurious correlations in text classification via automatically generated counterfactuals. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, pp 14,024–14,031, URL https://ojs.aaai.org/index.php/AAAI/article/view/17651
  * Wankhade et al (2022) Wankhade M, Rao ACS, Kulkarni C (2022) A survey on sentiment analysis methods, applications, and challenges. Artif Intell Rev 55(7):5731–5780. 10.1007/s10462-022-10144-1, URL https://doi.org/10.1007/s10462-022-10144-1
  * Xiao et al (2019) Xiao L, Huang X, Chen B, et al (2019) Label-specific document representation for multi-label text classification. In: Inui K, Jiang J, Ng V, et al (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics, pp 466–475, 10.18653/v1/D19-1044, URL https://doi.org/10.18653/v1/D19-1044
  * Xun et al (2020) Xun G, Jha K, Sun J, et al (2020) Correlation networks for extreme multi-label text classification. In: Gupta R, Liu Y, Tang J, et al (eds) KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020. ACM, pp 1074–1082, 10.1145/3394486.3403151, URL https://doi.org/10.1145/3394486.3403151
  * Yang et al (2018) Yang P, Sun X, Li W, et al (2018) SGM: sequence generation model for multi-label classification. In: Bender EM, Derczynski L, Isabelle P (eds) Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018. Association for Computational Linguistics, pp 3915–3926, URL https://aclanthology.org/C18-1330/
  * Yang et al (2019) Yang P, Luo F, Ma S, et al (2019) A deep reinforced sequence-to-set model for multi-label classification. In: Korhonen A, Traum DR, Màrquez L (eds) Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguistics, pp 5252–5258, 10.18653/v1/p19-1518, URL https://doi.org/10.18653/v1/p19-1518
  * Yang et al (2021) Yang X, Zhang H, Qi G, et al (2021) Causal attention for vision-language tasks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, pp 9847–9857, 10.1109/CVPR46437.2021.00972, URL https://openaccess.thecvf.com/content/CVPR2021/html/Yang_Causal_Attention_for_Vision-Language_Tasks_CVPR_2021_paper.html
  * Yao et al (2021) Yao L, Chu Z, Li S, et al (2021) A survey on causal inference. ACM Trans Knowl Discov Data 15(5):74:1–74:46. 10.1145/3444944, URL https://doi.org/10.1145/3444944
  * Yue et al (2021) Yue Z, Wang T, Sun Q, et al (2021) Counterfactual zero-shot and open-set visual recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, pp 15,404–15,414, 10.1109/CVPR46437.2021.01515, URL https://openaccess.thecvf.com/content/CVPR2021/html/Yue_Counterfactual_Zero-Shot_and_Open-Set_Visual_Recognition_CVPR_2021_paper.html
  * Zhang et al (2021) Zhang X, Zhang Q, Yan Z, et al (2021) Enhancing label correlation feedback in multi-label text classification via multi-task learning. In: Zong C, Xia F, Li W, et al (eds) Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, Findings of ACL, vol ACL/IJCNLP 2021. Association for Computational Linguistics, pp 1190–1200, 10.18653/v1/2021.findings-acl.101, URL https://doi.org/10.18653/v1/2021.findings-acl.101