:
204 2023 Conformal and Probabilistic Prediction with Applications
PMLRProceedings of Machine Learning Research Harris Papadopoulos, Khuong An
Nguyen, Henrik Boström and Lars Carlsson

# Conformal Regression in Calorie Prediction for Team Jumbo-Visma

Kristian van Kuijk<EMAIL_ADDRESS>
Department of Advanced Computing Sciences, Maastricht University, The
Netherlands
Visma Connect, The Hague, The Netherlands
Mark Dirksen<EMAIL_ADDRESS>
Visma Connect, The Hague, The Netherlands
Christof Seiler<EMAIL_ADDRESS>
Department of Advanced Computing Sciences, Maastricht University, The
Netherlands
Mathematics Centre Maastricht, Maastricht University, The Netherlands

###### Abstract

UCI WorldTour races, the premier men’s elite road cycling tour, are grueling
events that put physical fitness and endurance of riders to the test. The
coaches of Team Jumbo-Visma have long been responsible for predicting the
energy needs of each rider of the Dutch team for every race on the calendar.
Those must be estimated to ensure riders have the energy and resources
necessary to maintain a high level of performance throughout a race. This
task, however, is both time-consuming and challenging, as it requires precise
estimates of race speed and power output. Traditionally, the approach to
predicting energy needs has relied on judgement and experience of coaches, but
this method has its limitations and often leads to inaccurate predictions. In
this paper, we propose a new, more effective approach to predicting energy
needs for cycling races. By predicting the speed and power with regression
models, we provide the coaches with calorie needs estimates for each
individual rider per stage instantly. In addition, we compare methods to
quantify uncertainty using conformal prediction. The empirical analysis of the
jackknife+, jackknife-minmax, jackknife-minmax-after-bootstrap, CV+, CV-
minmax, conformalized quantile regression, and inductive conformal prediction
methods in conformal prediction reveals that all methods achieve valid
prediction intervals. All but minmax-based methods also produce sufficiently
narrow prediction intervals for decision-making. Furthermore, methods
computing prediction intervals of fixed size produce tighter intervals for low
significance values. Among the methods computing intervals of varying length
across the input space, inductive conformal prediction computes narrower
prediction intervals at larger significance level.

###### keywords:

Sports Analytics, Road Cycling, Sports Nutrition, Conformal Prediction, Data
Mining

##  1 Introduction

Nutrition is a key part of the performance of a rider. Until 2020, the Dutch
cycling team Jumbo-Visma, winner of the Tour de France 2022, would start
preparing their calorie estimates up to three weeks in advance to ensure they
had adequate estimates per cyclist and per stage. This is a time-consuming
task. To improve team performance, we built regression models to predict the
calories burned by a rider without needing any human computation. The models
use information like the stage profile, the body mass index of a cyclist, or
the race tactics, but also unforeseen factors such as the weather conditions.
Following our forecasts, the nutritionists and cooks prepare meals for each
rider per race day using the Jumbo Foodcoach app. This automated process
ensures riders are provided with their exact nutrition needs, leading to a
considerable advantage on race days.
Despite a significant improvement in calorie prediction from the manual
predictions of coaches ($R^{2}$ score of 0.55 for the prediction by coaches to
0.82 for the regression models), coaches still tune the output predictions.
This means coaches tend to increase or decrease the models’ outputs based on
knowledge and previous experiences for specific races. Given this tendency for
coaches to adjust the model predictions, instead of predicting a single
outcome, it would be more beneficial to predict a range of possibilities. This
can be achieved through prediction intervals. These intervals are calibrated
based on the probability of encompassing the true output. By quantifying the
reliability of the model predictions in estimating the speed and power of Team
Jumbo-Visma riders, coaches can adapt predictions based on the uncertainty of
the forecasts. To achieve this, we employ methods from the conformal
prediction framework introduced by [Vovk et al.(2005)Vovk, Gammerman, and
Shafer], providing valid and efficient prediction intervals. Each interval is
computed given a significance value $\alpha$. This means if we take for
instance $100$ Tour de France races and predict the calorie intake for a
specific rider per race, in the long run, the true value will be outside the
prediction bounds on average for only $\alpha$ races or less. Figure 1
illustrates our approach. After we compute prediction intervals for both the
race speed and the rider’s power output for a specific race, coaches combine
both to obtain an energy forecast. As a concrete example, for one of the races
of the 2022 season, the long-term power forecast bounds were $[213,265]$ for a
specific rider, with a predicted power of $245.17$ (true value of $238.13$).
Given the planned tactic and the previous experience of coaches with this
race, the coach decided to round the power to $250$ watts. Combined with the
predicted race time of $384$ minutes, computed from the speed forecast, this
resulted in a calorie forecast of $5760$ kilocalories.

speedpower$\text{energy}=\text{power}\cdot\frac{\text{race
distance}}{\text{speed}}$ machine learning coach Figure 1: Energy forecast
procedure for Team Jumbo-Visma coaches. Machine learning provides prediction
intervals. Coaches pick a value from the speed and power intervals and
forecast energy consumption.

In 2018, [Hilmkil et al.(2018)Hilmkil, Ivarsson, Johansson, Kuylenstierna, and
van Erp] predicted the heart-rate response of a cyclist during a training
session. The promising results led to a number of papers to predict the power
performance of professional riders at the Tour de France ([Kataoka and
Gray(2019)]), to predict the winner of the Tour de France ([Hobson and
Goff(2017)]), to identify the next top cyclist ([Janssens et
al.(2022)Janssens, Bogaert, and Maton]), and to athlete monitoring ([Leeuw et
al.(2022)Leeuw, Oberkofler, Heijboer, and Knobbe]). Nevertheless, none of
those methods quantify uncertainty in their predictions. The data science team
of Visma Connect started working with Team Jumbo-Visma coaches and
nutritionists in 2020 to improve the performance of the team using machine
learning and mathematical methods. Previously, the calorie intakes were
computed manually by the coaches using only domain knowledge from previous
similar races and experience. But out on the track, unforeseen factors impact
how much energy the cyclists burn. The weather, for instance, can cause
cyclists to exert themselves more, or perhaps the tactics of the team needs to
change due to other circumstances. This means coaches would often have to
review their estimates several times before each stage of the race, a time-
consuming exercise that had to be done for each rider for all races of the
season.
We present the data that we received from Team Jumba-Visma in Section 2. We
introduce our baseline prediction model and review current conformal
prediction methods in Section 3. We benchmark conformal methods on data from
the Giro d’Italia and the Tour de France in Section 4. Finally, we interpret
our findings and give recommendations on how coaches can fine-tune conformal
methods in practice in Section 5.

##  2 Data

The dataset for this paper consists of 1446 instances, all from the Team
Jumbo-Visma men’s team. The data is provided by Team Jumbo-Visma through
Smartabase Human Performance Platform and Athlete Management System
(Smartabase), a data management platform for professional sport organizations
and collected through a Garmin device and a crank based power meter. This
allows to record duration, heart rate, speed, distance, elevation gains,
calories, power, and other variables, of all race and training sessions. The
https://www.procyclingstats.com/index.phpProCyclingStats API, a database that
references all professional cycling races, allows to filter the training
instances from the race data and provides more accurate race information such
as the actual distance of the race, since it can happen that a rider forgets
to turn off his Garmin devices at the end of a stage. For this project, we
only retrieve the race name, race date, distance, race type (whether it is a
one day race, stage race or Grand Tour) and name of the rider. Lastly, the
https://darksky.net/DarkSky API provides all the weather information for each
race, from the temperature to the wind effect at every race kilometer.
The speed and power datasets use $8$ and $10$ features, respectively, ranging
from the race type (one day race, stage race, or Grand Tour), the stage
profile with the ascent/descent and distance, the weather conditions with the
temperature, humidity, negative wind-effect and rainfall, attributes of the
riders (body mass index (BMI)), and the tactics with the role of each rider
(helper, climber or leader). The BMI of the rider and race strategy are only
taken into account for the power dataset.
To take into account the steepness of a stage, we compute an ascent relation
variable by dividing the ascent with the descent coefficient. Concerning the
weather data, we obtain the negative wind effect by computing the mean of the
wind speed when the degree of the cyclist is against the wind. This is done
with the GPS files provided by Team Jumbo-Visma and by inspecting the
direction the rider faces every three seconds. Lastly, we compute the rainfall
by the product of the precipitation intensity times the precipitation
probability.

##  3 Methods

###  3.1 Prediction Model

The time and the power are the two predicted response variables we provide to
Team Jumbo-Visma. We train a random forest model as the underlying regressor
as it performed best for our dataset in terms of $R^{2}$ and root-mean-square
error. The energy $E$ is then computed mathematically from the time $t$ and
the power $p$: $E=t\cdot p=\frac{d}{s}\cdot p$, with $d$ the race distance and
$s$ the speed. This allows the Team Jumbo-Visma coaches to better understand
our predictions and tweak them according to what they believe could be the
actual power of a specific rider and stage. Predicting the energy directly
makes it harder for the coaches to understand the logic underneath a certain
prediction. In fact, even before we started working with the Dutch cycling
team, the coaches first predicted the speed and power to finally obtain the
needed calories intake of a specific rider. The predicted speed is the same
for all the riders for each stage, while the power differs per rider. Hence,
we use no information concerning individual riders for the speed estimator.
Considering the very small finish time difference among riders compared to the
overall length of a stage, we focus on predicting the average race speed.
In practice, we perform forecasts daily for Team Jumbo-Visma. Naturally,
considering only one weather forecast for a race that takes place in more than
10 days is suboptimal. To solve this uncertainty, we assign weights for short-
term forecasting based on how many days in advance the forecast is produced.
Thus, the weather has a more important influence in the predictions closer to
the race day. For example, five days prior to a race, we assign a weight of
0.9 to the model with weather features, and 0.1 to the one without weather
features. For a forecast 10 days preceding the start of the race, the weights
are of (0.5, 0.5), respectively. The pair of weights always sum to $1$ and are
based on research from the NASA Space Place team at NASA’s Jet Propulsion
Laboratory ([NASA(2022)]).

###  3.2 Conformal Prediction Methods

In this section, we introduce the conformal prediction methods that we
benchmark for Team Jumbo-Visma. A theoretical description of the methods
mentioned can be found in [Barber et al.(2021)Barber, Candès, Ramdas, and
Tibshirani, Linusson(2021), Romano et al.(2019)Romano, Patterson, and Candes,
Vovk et al.(2005)Vovk, Gammerman, and Shafer].
Conformal prediction, introduced by Vovk, Gammerman and Shafer in 2005, has
been proven to provide valid output ([Vovk et al.(2005)Vovk, Gammerman, and
Shafer]), i.e. predicted sets for any fixed confidence level $1-\alpha$ will
not cover the true response with frequency at most $\alpha$. Since 2005, the
conformal prediction framework has been applied to many learning algorithms,
from support vector machines ([Forreryd et al.(2018)Forreryd, Norinder,
Lindberg, and Lindstedt]), $k$-nearest neighbors ([Wang et al.(2017)Wang, Sun,
Miao, Wang, Luo, and Li]), to ridge regression ([Burnaev and Vovk(2014)]),
among others. In addition to providing uncertainties quantification for
prediction, one of the main research focuses recently has been on providing a
coverage of $\geq 1-\alpha$ while keeping a low computational training cost.
For instance, the jackknife+ method has a training cost equal to the training
set size $n$. They train $n$ leave-one-out models and provide rigorous
coverage guarantees regardless of the distribution of the data entries for any
algorithm that treats the training points symmetrically ([Barber et
al.(2021)Barber, Candès, Ramdas, and Tibshirani]).
We investigate the following conformal regression methods: jackknife and its
variations (jackknife+, jackknife-minmax, jackknife+-after- bootstrap, and
jackknife-afterbootstrap-minmax), cross-validation (CV) and its variations
(CV+ and CV-minmax), conformalized quantile regression (CQR) and inductive
conformal prediction (ICP).
The different jackknife methods are based on the creation of $n$ set of leave-
one-out models, with $n$ the size of our training set. This means we train $n$
models omitting each time one entry on which we test the performance of the
model. The prediction intervals are then constructed from the $(1-\alpha)n$th
smallest value of the empirical distribution of the absolute residuals of
these omitted data entries. For better coverage, the jackknife can be adapted
to the jackknife-minmax using simply the minimal and maximal value of the
leave-one-out predictions. While the different jackknife methods do not suffer
from overfitting, all have a lage training time. This does not pose a problem
in our case since the dataset used is relatively small.
Nevertheless, for larger datasets the same methods can be applied using cross-
validation, the CV method, instead of leave-one-out. This means we calibrate
the model on a proportion of data. The refined method of the jackknife can
also be implemented to the base CV method. To reduce the training time, the
jackknife+ method can also be adapted operating a bootstrap approach, using
only the available bootstrapped samples. This results in reduced computational
time and is usually as robust as the jackknife+ method.
All prediction intervals outputted by the jackknife and CV methods (and their
respected refined methods) have constant width for all features across the
input space. This behaviour is suboptimal since we desire the conformal
predictor to reflect the certainty at a given feature value
([Linusson(2021)]). This is the case of the CQR and ICP methods. [Romano et
al.(2019)Romano, Patterson, and Candes] introduced the CQR method combining
conformal prediction and classical quantile regression, inheriting the
advantages of both. CQR uses quantile regressors to estimate the prediction
bounds.
Lastly, the ICP method computes a nonconformity score indicating how different
the instance is compared to other instances of the training set, and produces
a corresponding $p$-value ([Vovk et al.(2005)Vovk, Gammerman, and Shafer]).
Those $p$-values express the proportion of the same or less conforming
instances than the considered example. The nonconformity also takes into
account the difficulty of predicting the response based on how different the
instance is from other instances in the training set. We compute it by
training a $k$-nearest neighbours regressor. This regressor returns the
difficulty of predicting the outcome by considering the error of the
underlying model. In this paper, we use as nonconformity score the absolute
difference between the expected value and the value observed from the
underlying regression model.

##  4 Experiments & Results

For all the experiments, we preprocess the data and perform feature
engineering (described in Section 2). We repeat five-fold cross-validation
five times and report the average. We compare the error rate and interval
width of all methods as a function of the significance $\alpha$ (Figures 2 and
3) of the CV+, CV-minmax, jackknife+-after-bootstrap, jackknife-minmax-after-
bootstrap, jackknife-minmax, jackknife+, CQR, and ICP methods for the speed
and power response variables. To differentiate constant and non-constant
interval size prediction intervals methods, the two methods computing non-
constant interval size prediction intervals (CQR and ICP methods) are depicted
by dashed lines. As significance levels larger than $0.20$ are very unusual,
since the error rate becomes too large for the prediction intervals to be used
in practice, all figures only include $\alpha\leq 0.20$. All experiments are
performed on an Intel i7 with 8 CPU cores at 3GHz and 16GB of RAM.
The jackknife-minmax, jackknife-minmax-after-bootstrap and CV-minmax’s error
rates are considerably lower than the target value (Figure 2). The error rate
at $\alpha=0.20$ is approximately $0.10$ for the three methods for the speed
estimator. The jackknife+, jackknife+-after-bootstrap, CV+, ICP, and CQR
methods compute prediction intervals close to the target value, particularly
for the power response variable.
Concerning the interval widths (Figure 3), both response variables have a
similar trend. The jackknife+, CV+ and jackknife+-after-bootstrap method
produce the tightest intervals, particularly for the power response variable,
followed by the minmax methods. The CQR and ICP methods produce considerably
wider intervals for low $\alpha\leq 0.05$. Nevertheless, as $\alpha$
increases, the CQR and ICP methods intervals are comparable to the other
methods.
To illustrate the benefits of our approach, we compare in Figure 4 the manual
predictions of coaches for two Grand Tour races of 2019 (Grand Tours are
considered the most prestigious races of the season, typically spanning over
three weeks) with our own forecasts, both single-point and prediction
intervals, and the true response in predicting the race speed. An identical
comparison for the power output can be found in Figure 5. We predict the race
speed and the power output to obtain the energy needs per rider. The
prediction of coaches tend to often fall outside the prediction intervals,
while the error rate of our prediction interval is close to the target value.
Furthermore, our model predictions demonstrate a greater degree of accuracy in
predicting the true output compared to the predictions from coaches. The mean
absolute error in Figure 4 is $2.46$ km/h for the coaches as opposed to $1.73$
km/h for the model ($\approx 30\%$ lower). For the power response, the mean
absolute errors in Figure 5 are $23.10$ watts and $12.68$ watts for the
coaches and the model respectively ($\approx 45\%$ lower). We improve the
$R^{2}$ for calorie prediction compared to manual predictions by $49\%$.

Figure 2: Error rate for the speed (left panel) and power (right panel)
response variables (methods computing intervals of varying length across the
input space are depicted by dashed lines). Figure 3: Interval width for the
speed (left panel) and power (right panel) response variables (methods
computing intervals of varying length across the input space are depicted by
dashed lines). Figure 4: Speed forecasts comparison between coaches’ manual
predictions, regression models and prediction intervals (significance
$\alpha=0.10$) to the true value for the Giro d’Italia 2019 and the Tour de
France 2019 excluding time trials. Figure 5: Power forecasts comparison
between manual predictions of coaches, regression models and prediction
intervals (significance $\alpha=0.10$) to the true value for the Giro d’Italia
2019 (for $3$ riders) and the Tour de France 2019 (for $6$ riders) excluding
time trials (names of riders A-I are anonymized).

##  5 Discussion

The jackknife+, jackknife+-after-bootstrap, and CV+ methods produce both valid
and efficient prediction intervals for any $\alpha\leq 0.20$, with tight
enough intervals to be useful in decision-making and the rate of instances
outside of prediction intervals not exceeding $\alpha$.
The jackknife-minmax, jackknife-minmax-after-bootstrap and CV-minmax produce
intervals with a lower error rate than the target value. This results from the
commonality of those three methods using the minimal and maximal value of the
leave-one-out or folds ([Barber et al.(2021)Barber, Candès, Ramdas, and
Tibshirani]). This behaviour is conservative considering we want the
significance $\alpha$ to reflect the empirical error rate. The ICP and CQR
methods are the only methods to reflect the certainty at a given feature
value. They do not produce narrow enough intervals to be useful in decision-
making for low $\alpha\leq 0.05$. Nevertheless, for larger $\alpha$, the
produced intervals by the ICP method are comparable to the other methods.
The CQR performs worst in terms of width of prediction intervals for the speed
response variable, leading to wider intervals compared to other methods and an
error rate under the target value $\alpha$. Our models are trained using a
random forest model as underlying regressor. [Meinshausen(2006)] showed that
quantile regression forests are frequently excessively conservative, resulting
in unnecessarily wide prediction intervals. ICP is computationally efficient
as we only need to fit a single regression function. In contrast, we must run
the regression function repeatedly when using the jackknife and CV approaches.
These advantages come at a statistical price. If the training set size is much
smaller than $n$, the size of the dataset, then the fitted model may be a poor
fit, leading to wide prediction intervals. If instead, the training set size
is close to $n$, then the calibration set is very small, leading to high
variability ([Barber et al.(2021)Barber, Candès, Ramdas, and Tibshirani]). ICP
sacrifices different parts of the training set at different stages of
prediction affecting its informational efficiency ([Vovk(2015)]). This may
result in more conservative prediction intervals for our small dataset.
The minor differences in performance between the jackknife, CV (and their
refined methods excluding minmax based methods), and ICP methods are not
noticeable in decision-making according to Team Jumbo-Visma coaches. All
methods compute valid prediction intervals that can be considered narrow
enough by coaches to be useful. The ICP prediction intervals are comparable to
the jackknife+ method for $\alpha\geq 0.05$ for the speed and $\alpha\geq
0.10$ for the power response variable. Most importantly, the latter produces
prediction intervals of varying lengths across the input space. This means the
ICP method reflects the certainty of the model for a given feature value.
Considering our small dataset, the heavy training cost of the different
jackknife methods does not cause an issue ($1314$ seconds for the power
response variable).
The power and speed data are different. We have power data for each rider, a
total of $1446$ training instances. The power tends to differ greatly among
riders in the same race. In contrast, we have speed aggregated data per race,
a total of $436$ training instances. The smaller sample size for the speed
response variable results in larger error rates. We believe that this affects
the separation in error rate between minmax methods and other methods for the
speed data. This effect is weaker for the power data due to the difference in
the number of training instances.
Choosing a significance value $\alpha$ is an important part in the process of
generating confidence intervals. The lower $\alpha$, the larger the prediction
intervals. If the prediction sets are too wide, we risk that they are not
useful anymore in decision-making. Low confidence results in high uncertainty
about the true value. To pick the $\alpha$, we recommend a similar method to
the elbow method applied to clustering (tracing back to [Thorndike(1953)]).
Figure 3 suggests that both $\alpha=0.04$ or $\alpha=0.06$ are reasonable
choices for the ICP method.

##  6 Conclusion

This paper introduces the calorie prediction project we started for Team
Jumbo-Visma. Our energy forecasts are used daily by the coaches and
nutritionists of the team. We provide Team Jumbo-Visma with prediction
intervals that are narrow enough to be useful in practice. The ICP method
performs best for our dataset. In future research, we plan to provide
predictions for the women’s team. We also plan to include domain knowledge
through a Bayesian model, and to conformalize the posterior predictive
intervals.

## References

  * [Barber et al.(2021)Barber, Candès, Ramdas, and Tibshirani] Rina Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani.  Predictive inference with the jackknife+.  _Annals of Statistics_ , 49:486–507, 02 2021.  10.1214/20-AOS1965.
  * [Burnaev and Vovk(2014)] Evgeny Burnaev and Vladimir Vovk.  Efficiency of conformalized ridge regression.  In _Conference on Learning Theory_ , pages 605–622. PMLR, 2014.
  * [Forreryd et al.(2018)Forreryd, Norinder, Lindberg, and Lindstedt] Andy Forreryd, Ulf Norinder, Tim Lindberg, and Malin Lindstedt.  Predicting skin sensitizers with confidence—using conformal prediction to determine applicability domain of gard.  _Toxicology in Vitro_ , 48:179–187, 2018.
  * [Hilmkil et al.(2018)Hilmkil, Ivarsson, Johansson, Kuylenstierna, and van Erp] Agrin Hilmkil, Oscar Ivarsson, Moa Johansson, Dan Kuylenstierna, and Teun van Erp.  Towards machine learning on data from professional cyclists.  _12th World Congress on Performance Analysis of Sports_ , 2018.
  * [Hobson and Goff(2017)] Chad Michael Hobson and John Eric Goff.  Using the 2011-16 Tours de France to refine prediction model and elicit racing strategies.  _Proceedings of the Institution of Mechanical Engineers, Part P: Journal of Sports Engineering and Technology_ , 231:232 – 240, 2017\.
  * [Janssens et al.(2022)Janssens, Bogaert, and Maton] Bram Janssens, Matthias Bogaert, and Mathijs Maton.  Predicting the next Pogačar: a data analytical approach to detect young professional cycling talents.  _Annals of Operations Research_ , pages 1–32, 01 2022.  10.1007/s10479-021-04476-4.
  * [Kataoka and Gray(2019)] Yasuyuki Kataoka and Peter Gray.  Real-time power performance prediction in Tour de France.  In Ulf Brefeld, Jesse Davis, Jan Van Haaren, and Albrecht Zimmermann, editors, _Machine Learning and Data Mining for Sports Analytics_ , pages 121–130, Cham, 2019. Springer International Publishing.
  * [Leeuw et al.(2022)Leeuw, Oberkofler, Heijboer, and Knobbe] Arie-Willem Leeuw, Tobias Oberkofler, Mathieu Heijboer, and Arno Knobbe.  Athlete monitoring in professional road cycling using similarity search on time series data.  In _Machine Learning and Data Mining for Sports Analytics_ , 09 2022\.
  * [Linusson(2021)] Henrik Linusson.  _Nonconformity Measures and Ensemble Strategies : An Analysis of Conformal Predictor Efficiency and Validity_.  PhD thesis, Stockholm University, Department of Computer and Systems Sciences, 2021.
  * [Meinshausen(2006)] Nicolai Meinshausen.  Quantile regression forests.  _Journal of Machine Learning Research_ , 7(35):983–999, 2006.  URL http://jmlr.org/papers/v7/meinshausen06a.html.
  * [NASA(2022)] NASA.  How reliable are weather forecasts?, Jun 2022.  URL https://scijinks.gov/forecast-reliability/.
  * [Romano et al.(2019)Romano, Patterson, and Candes] Yaniv Romano, Evan Patterson, and Emmanuel Candes.  Conformalized quantile regression.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * [Thorndike(1953)] Robert L. Thorndike.  Who belongs in the family?  _Psychometrika_ , 18:267–276, 1953.
  * [Vovk(2015)] Vladimir Vovk.  Cross-conformal predictors.  _Annals of Mathematics and Artificial Intelligence_ , 74(1):9–28, 2015.  10.1007/s10472-013-9368-4.  URL https://doi.org/10.1007/s10472-013-9368-4.
  * [Vovk et al.(2005)Vovk, Gammerman, and Shafer] Vladimir Vovk, Alex Gammerman, and Glenn Shafer.  _Algorithmic Learning in a Random World_.  01 2005.  10.1007/b106715.
  * [Wang et al.(2017)Wang, Sun, Miao, Wang, Luo, and Li] Zhan Wang, Xiyang Sun, Jiacheng Miao, You Wang, Zhiyuan Luo, and Guang Li.  Conformal prediction based on k-nearest neighbors for discrimination of ginsengs by a home-made electronic nose.  _Sensors_ , 17(8), 2017.  ISSN 1424-8220.  10.3390/s17081869.


# Label Mask for Multi-Label Text Classification

Rui Song<EMAIL_ADDRESS>Xingbing Chen<EMAIL_ADDRESS>Zelong
Liu<EMAIL_ADDRESS>Haining An<EMAIL_ADDRESS>Zhiqi Zhang
<EMAIL_ADDRESS>Xiaoguang Wang<EMAIL_ADDRESS>Hao Xu
<EMAIL_ADDRESS>School of Artificial Intelligence, Jilin University  Public
Computer Education and Research Center, Jilin University  College of Computer
Science and Technology, Key Laboratory of Symbolic Computing and Knowledge
Engineering of Ministry of Education, Jilin University  College of Electronic
Science and Engineering, Jilin University  College of Construction
Engineering, Jilin University  College of Sotfware, Jilin University

###### Abstract

One of the key problems in multi-label text classification is how to take
advantage of the correlation among labels. However, it is very challenging to
directly model the correlations among labels in a complex and unknown label
space. In this paper, we propose a Label Mask multi-label text classification
model (LM-MTC), which is inspired by the idea of cloze questions of language
model. LM-MTC is able to capture implicit relationships among labels through
the powerful ability of pre-train language models. On the basis, we assign a
different token to each potential label, and randomly mask the token with a
certain probability to build a label based Masked Language Model (MLM). We
train the MTC and MLM together, further improving the generalization ability
of the model. A large number of experiments on multiple datasets demonstrate
the effectiveness of our method.

###### keywords:

Multi-label Text Classification, Bert, Cloze Questions, Masked Language Model

††journal: Journal of LaTeX Templates

##  1 Introduction

Text categorization is a basic task in natural language processing, Multi-
label text classification assigns multiple different labels to a document,
rather than a document corresponding to a single category. In recent years,
multi-label text classification has been widely used in sentiment analysis
[1], topic classification [2], information retrieval [3], label recommendation
[4]. How to fully capture semantic patterns from original documents, how to
extract discriminant information related to corresponding labels from each
document, and how to accurately mine the correlation between labels are the
three aspects that researchers pay attention to [5]. Among them, the relevance
modeling of complex and unknown label systems has always been the focus of
scholars’ efforts.

Figure 1: Pearson correlation coefficient between all label pairs in AAPD
dataset. The lighter the color, the more relevant the label pairs are.

One of the most direct methods to solve multi-label text classification is to
transform the multi-label text classification task into several binary
classification tasks [6], however, this tends to ignore the relationship
between multiple labels. Similarly, some deep learning approaches, such as CNN
[7] and attention mechanism [2], can model the documents effectively, but they
still ignores the relationship between labels. As shown in Figure 1, there are
specific correlations between different label pairs in the dataset AAPD
calculated by Pearson correlation coefficient. For labels 0 and 1, the
correlation is 1, which means that both appear together in all instances.
Therefore, for some datasets with less label information or severe long-tail
distribution, the association between labels can provide more important
information [8].

The emergence of large-scale pre-trained language models, such as Bert
(Bidirectional Encoder Representations from Transformers) [9] have made
knowledge transfer in the field of natural language processing easier. Studies
have confirmed that the intermediate layer of Bert encodes a wealth of
linguistic information [10]. Inspired by Cloze Questions (CQ) methods based
large-scale pre-trained language models, we propose a Label Mask multi-label
text classification model (LM-MTC) to capture the potential semantic and
association relations among labels [11, 12]. Specifically, we map different
labels to different tokens and build a set of token prefix templates. During
training, we spliced the token template with the sentences to be classified
and input them to Bert. When predicting, we mask all the label tokens and
predict them. The advantages of Bert can help the model adaptively capture the
semantic relationship between labels and documents, as well as the association
relationships. In addition, to make better use of the predictive ability of
Bert, we constructed a multi-task framework, randomly masked label tokens, and
used the Mask Language Model (MLM) to predict masked tokens to assist in
optimizing the multi-label text classification learning task. Our
contributions are as follows:

  * 1.

We propose a Label Mask Multi-label Text Classification model (LM-MTC), which
converts multi-label text classification into a Cloze Questions (CQ) task and
captures the potential relationship between labels with the help of a pre-
trained language model. We introduced MLM with LM-MTC for joint training which
further improved the performance of the model.

  * 2.

Through attention analysis, we confirm the LM-MTC’s ability to capture
potential associations between labels.

  * 3.

We carry out lots of experiments on different types of multi-label text
classification tasks to prove the effectiveness of the proposed model.

##  2 Related Work

###  2.1 Multi-label Text Classification

Multi-label text classification is a basic task in NLP. There are some methods
to solve it by transforming the multi-label text classification task into
several binary classification tasks [6, 7, 2] Some approaches take advantage
of pair associations or mutexes between labels. Pairwise comparison (RPC)
induces a binary preference relation using a natural extension of pairwise
classification, which transforms the multi-label learning task into a label
ranking task [13].

However, it is more efficient to assume that a label can be related to
multiple labels and take advantage of the higher-order dependencies of the
label. Classifier chains (CC) transforms the task of MTC into a chain of
binary classification tasks [14]. K-labelsets (RAkEL) constructs small random
subsets of labels and transforms MTC into a single-label classification task
of random subsets [15]. With the development of deep learning in recent years,
some studies have resorted to sequence learning models to solve MTC such as
[16], and Sequence Generation Model (SGM) [17]. They generate a possible label
sequence through the RNN decoder. However, sequence model requires searching
for the optimal solution in the potential space, which is too time-consuming
when there are too many labels.

Some approaches model the joint probability distribution of labels, rather
than associations for specific labels, such as Bayesian networks [18, 19] and
undirected graph models [20]. [21] strengthen the similarity between the joint
distribution of multi-labels and the predicted multi-labels by means of
adverse learning framework. In recent years, due to the effectiveness of Graph
Neural Network (GNN) [22] in modeling non-Euclidean spatial data, some methods
use GNN to capture the correlation of labels. Label-Specific Attention Network
(LSAN) proposes a Label Attention Network model that considers both document
content and Label text, and uses self-attention mechanism to measure the
contribution of each word to each Label [23]. MAGNET uses a feature matrix and
a correlation matrix to capture and explore key dependencies between labels
[24].

###  2.2 Cloze Questions

A natural way to gain knowledge from a pre-trained MLM is to set the task as
fill-in-the-blanks. AUTOPROMPT creates prompts for a diverse set of tasks
automatically and shows the inherent ability of MLM to perform emotion
analysis and natural language reasoning [25]. [11] converts the input text
into a cloze problem containing the task description combined with gradient-
based optimization and proves that ’green’ LM like Bert can still have
competitive performance compared with GPT-3 [26]. Furthermore, PTE has
successfully solved the problems of text classification and natural language
reasoning in small samples by using cloze questions method [12]. In addition,
[27] propose a similar P-tuning method to improve GPT’s ability in natural
language understanding tasks.

##  3 Preliminaries

First, we give some notations and describe the MTC task. For the given label
space $\mathbf{L}=\\{l_{1},l_{2},...,l_{L}\\}$ and a text
$x=\\{w_{1},w_{2},...,w_{m}\\}$, the MTC task aims to learn a a mapping
function $\chi:x\to\hat{L}$, where $\hat{L}\subseteq L$ and $|\hat{L}|\geq 1$.
Unlike single-label tasks, a text can belong to several different categories.
For ease of calculation, the output space of the labels is defined as a vector
$L_{O}\in\mathbf{R}^{|L|}$. For example, when $L=3$ and
$\hat{L}=\\{l_{1},l_{2}\\}$, then $L_{O}$ is $[1,1,0]$. Then we can rewrite
the objective function as $\chi^{\prime}:x\to L_{O}$. In addition, we define 1
as a positive label and 0 as a negative label.

Given a movie emotion dichotomy sentence, ”The movie was so touching!”, CQ
usually generates a new sentence for input by a prefix/suffix template $\tau$:
”The movie was so touching! I $\sharp$ it!”. $\sharp$ can be either ”like” or
”hate”, indicating positive or negative emotions, respectively. The new input
with a prefix template can be expressed as:

$x^{\prime}=\tau||x$ (1)

where $||$ denotes concatenation.

Consider an MLM $M$ with a vocabulary $V$ and a sentence $x$ with mask $m$,
the task of $M$ is to predict the probability of the original word in the mask
$p_{M}^{m}(w|x)$ where $w\in V$. The goal of CQ is to predict the true value
of $\sharp$ by MLM, that is $p_{M}^{m}(\sharp|x^{\prime})$. So the
classification task can be transformed into the prediction task of MLM with
the help of templates.

##  4 LM-MTC

In this section, we describe the proposed model in detail. The specific
execution process is shown in Algorithm 1.

Figure 2: LM-MTC model structure.

###  4.1 Cloze Patterns

For cloze tasks, although some studies have demonstrated the advantages of a
template approach, it is not clear whether the same template will work for
every model, or what kind of template will fit the model better [25]. For MTC,
because different documents contain different numbers of true labels and the
output is mapped to a single token, it is not possible to build a specific
template for each label. To this end, we build a template system for the
entire label space. First of all, considering the different states of labels,
the labels in each position should have three different states: 0,1 and
$mask$. We emphasize the order of the different labels, which is very
important for label prediction [17]. In addition, we also introduce a
location-based prompt to allow Bert to clearly recognize where the current
Label is located. For label sequence with mask $[1,0,mask]$, we generate the
template as follows:

$\begin{split}\color[rgb]{0,0,0}{[LS-1][YES-1][LE-1]}\\\
\color[rgb]{0,0,0}{[LS-2][NO-2][LE-2]}\\\
\color[rgb]{0,0,0}{[LS-3][MASK-3][LE-3]}\end{split}$ (2)

where $LS$ denotes label start and $LE$ denotes label end.

###  4.2 Training and Inference

The forward propagation. After the template is generated, we treat it as a
prefix to the original sentence and input $x^{\prime}$ to the pre-training
model together. The training process has two main objectives: to predict the
probability distribution of multiple labels in the label space, and to predict
the mask by MLM. Assume that the output of Bert is
$O\in\mathbf{R}^{|x^{\prime}|_{max}*768}$, then the prediction of the
distribution of the labels and the prediction of the mask can be obtained by
using one layer of full connection:

$\begin{split}O_{l}=O*W_{l}+b_{l}\\\ O_{m}=O*W_{m}+b_{m}\end{split}$ (3)

where $|x^{\prime}|_{max}$ denotes max token length,
$W_{l}\in\mathbf{R}^{768*|L|}$. In order for MLM to predict the mask, we need
to extend $V$ to $V^{\prime}$. $V^{\prime}$ depends on the size of the label
space, so that $W_{m}\in\mathbf{R}^{768*|V^{\prime}|}$.

Joint loss. We use the Binary Cross Entropy (BCE) as the loss function for MTC
and the Cross Entropy as the loss function for MLM. BCE loss can be written as
follows:

$\begin{split}\mathcal{L}_{mtc}=\frac{1}{|L|}\sum_{i=1}^{|L|}(y_{ti}log(\sigma(y_{pi}))+\\\
(1-y_{ti})log(1-\sigma(y_{pi}))\end{split}$ (4)

where $\sigma$ denotes sigmoid activation function, $y_{t}$ is the ground
truth label and $y_{p}$ denotes the predicted results. The final joint loss
function is:

$\mathcal{L}=\mathcal{L}_{mtc}+\lambda\mathcal{L}_{mlm}$ (5)

Inference. When inferring, mask all the labels and calculate the probability
of all the masked labels. As with training, we express the output of labels as
$O_{l}$ and use the logistic sigmoid function for probability normalization:

$P_{l}=\sigma(O_{l})$ (6)

Then, all probability values greater than 0.5 are predicted to be positive
labels, otherwise predicted to be negative labels.

Algorithm 1 Label Mask Multi-label Text Classification

Input: The original sentence $X$ corresponds to the label $Y_{t}$.
Output: Predicted labels $Y_{p}$ and a trained model $M$.

1: if train then

2: generate templates T as Eq 2 and input tokens with masks $X^{\prime}$ as Eq
1.

3: train $M$ by optimizing the joint loss function as Eq 5.

4: else

5: generate templates T by masking all the labels and input tokens
$X^{\prime}$ as Eq 1.

6: calculate output $O_{l}$ by feeding $X^{\prime}$ into $M$.

7: predict $Y_{p}$ by probability distribution as Eq 6.

8: end if

##  5 Experiments

###  5.1 Datasets

In view of the wide application of multi-label text classification, we applied
our method on different types of data sets to verify the effectiveness of LM-
MTC. The statistics of the dataset are shown in Table 1.

Dataset | Label Sets | Train Size | Test Size
---|---|---|---
GAIC | 17 | 24000 | 3000
AAPD | 54 | 53840 | 1000
Reuters-21578 | 90 | 8630 | 2158
RMSC | 22 | 5020 | 646
Emotion | 28 | 43410 | 5427
Toxic | 6 | 11357 | 4868
Table 1: Datasets statistics.

  * 1.

GAIC111https://tianchi.aliyun.com/competition/entrance/531852. A competition
dataset of desensitized medical texts which all words are replaced by numbers.
We combined the dataset of the preliminary match and the semifinal match
together, removed the 12 labels added in the semifinal match, and finally
constituted a 17-label classified dataset of 30,000 samples. Then, we divide
the training set, development set and test set according to the ratio of
8:1:1. We pre-trained a Bert model based on desensitization data and used the
parameters of this Bert in the MTC task.

  * 2.

AAPD [17]. A widely used large-scale classification dataset for
multidisciplinary academic papers. The goal is to predict the subject by
abstracts.

  * 3.

Reuters-21578 [28]. Reuters news text dataset created in 1987 which has been a
standard benchmark for MTC. We follow the classification criteria of [24] and
use 90 categories.

  * 4.

RMSC [29]. Collected from a Chinese popular music
website222https://music.douban.com. The goal is to distinguish musical styles
based on different reviews. 22 styles are defined. We divide the dataset into
training/validation/test sets as described in the original paper. Note that
since it is a Chinese dataset, we use the Chinese pre-trained Bert
model333https://huggingface.co/bert-base-chinese. In the process of data
preprocessing, we remove non-Chinese and non-English special symbols.

  * 5.

Emotion [30]. A largest manually annotated dataset of 58k English Reddit
comments for fine-grained sentiment classification labeled for 27 emotion
categories and neutral, 28 categories totally.

  * 6.

Toxic Comments444https://www.kaggle.com/c/jigsaw-toxic-comment-classification-
challenge. A dataset from Toxic Comment Classification Challenge Competition
contains text that may be considered profane, vulgar, or offensive. We remove
comments that don’t carry any negative sentiment and keep only 16,225 tagged
records as our dataset. We split the train/test set randomly in a 7:3 ratio.

###  5.2 Evalution Metrics

The same as [15, 17], Hamming Loss and micro-F1 Score are used for the main
evaluation metrics. Besides, we also use Accuracy and Micro-Jaccard for
further evaluation.

  * 1.

Accuracy. The strict accuracy of multi-label classification. For the given
prediction result $Y_{p}\in\mathbf{R}^{|\Gamma|*|L|}$ and ground truth
$Y_{t}\in\mathbf{R}^{|\Gamma|*|L|}$, Accuracy is calculated as follows:

$Accuracy=\sum_{i}^{|\Gamma|}{\frac{\Xi(Y_{ti},Y_{pi})}{|\Gamma|}}$ (7)

where $|\Gamma|$ is the test set size, $\Xi(\cdot)$ is an indicator function.
If the corresponding elements at all positions are equal in $Y_{ti}$ and
$Y_{pi}$, $\Xi(Y_{ti},Y_{pi})=1$, else $\Xi(Y_{ti},Y_{pi})=0$.

  * 2.

Micro-F1 [17]. It can be interpreted as a weighted average of accuracy and
recall. It calculates indicators globally by calculating total true positives,
false negatives and false positives.

  * 3.

Micro-Jaccard [31]. The Jaccard similarity coefficient is defined as the size
of the intersection divided by the size of the union of two label sets.

  * 4.

Hamming loss (HL) [32]. It directly calculates the proportion of misclassified
labels. A value of 0 means that all labels for each sample have been assigned
the correct label.

###  5.3 Baselines

We compare LM-MTC with the widely available baselines:

  * 1.

Binary Relevance (BR) [6]. Categorize each label separately, regardless of the
correlation between the labels.

  * 2.

Classifier Chains (CC) [14]. Consider the high order correlation between
labels and convert it into a binary classification chain.

  * 3.

CNN [33]. Convolutional neural network is used to extract text features and
outputs the distribution of labels in the label space.

  * 4.

CNN-RNN [16]. CNN and RNN are combined for local and global text modeling.

  * 5.

Hierarchical Attention Network (HAN) [2]. Model text with hierarchical
attention to words and sentences.

  * 6.

HAN+LG [29]. Introduce Label Graph (LG) on the basis of HAN, and the soft
association relationship between labels is trained by LG.

  * 7.

SGM [17]. View the MCT as a sequence generation problem, and apply a sequence
generation model with a novel decoder structure to solve it.

  * 8.

Bert [9]. Self-attention based pretrained language model. Make different fine-
tuning for different downstream tasks.

  * 9.

Bert+MLM. On the basic Bert classification, additional MLM tasks are added.

  * 10.

MEGNET [24]. A graph attention network-based model to capture the attentive
dependency structure among the labels.

  * 11.

Label-Wise (LW) LSTM with PT and FT [34]. A document representation with
label-aware information is obtained through a pre-training model and fine-
tuned for different downstream tasks. PT denotes the pre-training method. FT
denotes the fine-tuned method on downstream task.

(a) Top11.

(b) 2nd attention.

(c) 6th attention.

(d) 12th attention.

Figure 3: Spearman correlation coefficient of top11 labels and attention visualization of different Bert layers of AAPD test set. Dataset | Model | Accuracy(+,%) | Micro-F1(+,%) | Micro-Jaccard(+,%) | HL(-)
---|---|---|---|---|---
| Bert | 89 | 92 | 86 | 0.02
| Bert+MLM | 87 | 91 | 84 | 0.02
| LM | 9 | 93 | 86 | 0.02
# GAIC | LM+MLM | 92 | 95 | 9 | 0.01
| Bert | 42.2 | 72.57 | 56.95 | 0.0226
| Bert+MLM | 40.5 | 72.8 | 57.23 | 0.0227
| ML | 41.3 | 73.73 | 58.39 | 0.0228
# AAPD | LM+MLM | 42.3 | 73.61 | 58.23 | 0.0224
| Bert | 85.25 | 89.09 | 80.33 | 0.0029
| Bert+MLM | 85.03 | 89.4 | 80.83 | 0.00274
| LM | 84.15 | 89.6 | 81.15 | 0.0028
# Reuters-21578 | LM+MLM | 85.59 | 89.97 | 81.77 | 0.00264
| Bert | 37.46 | 72.21 | 56.51 | 0.0507
| Bert+MLM | 39.63 | 73.18 | 57.71 | 0.0499
| LM | 36.53 | 72.81 | 57.24 | 0.0499
# RMSC | LM+MLM | 38.39 | 73.8 | 58.47 | 0.0517
| Bert | 47.08 | 57.55 | 40.4 | 0.031
| Bert+MLM | 43.97 | 54.7 | 37.7 | 0.033
| LM | 45.35 | 58.02 | 40.86 | 0.0306
# Emotion | LM+MLM | 47.63 | 59.07 | 41.92 | 0.0304
| Bert | 52.67 | 85.76 | 75.07 | 0.1043
| Bert+MLM | 51.97 | 84.96 | 73.86 | 0.1087
| LM | 52.71 | 85.86 | 75.23 | 0.1048
# Toxic Comments | LM+MLM | 53.53 | 85.77 | 75.08 | 0.1041
Table 2: Performance over different datasets. The bold represents the optimal results.  | AAPD | Reuters-21578 | RMSC
---|---|---|---
Model | Micro-F1(+,%) | HL(-) | Micro-F1(+,%) | HL(-) | Micro-F1(+) | HL(-)
BR | 64.6 | 0.0316 | 87.8 | 0.0316 | 41.8 | 0.083
CC | 65.4 | 0.0306 | 87.9 | 0.0306 | 44.3 | 0.107
CNN | 65 | 0.0264 | 86.3 | 0.0287 | 59.1 | 0.0702
CNN-RNN | 66.4 | 0.0278 | 85.5 | 0.0282 | - | -
HAN | 70.81 | 0.0236 | - | - | 66.75 | 0.059
HAN+LG | 71.19 | 0.0235 | - | - | 68.21 | 0.058
SGM | 71 | 0.0245 | - | - | - | -
Bert | 72.57 | 0.0226 | 89.09 | 0.0029 | 72.21 | 0.0507
Bert+MLM | 72.23 | 0.0231 | 89.4 | 0.00274 | 73.18 | 0.0499
MEGNET | 69.6 | 0.0252 | 89.9 | 0.0252 | - | -
LW-LSTM+PT | 71.21 | 0.0239 | - | - | 67.04 | 0.0583
LW-LSTM+FT | 71.31 | 0.0241 | - | - | 72.18 | 0.0537
LM | 73.73 | 0.0228 | 89.6 | 0.0028 | 72.81 | 0.0499
LM+MLM | 73.61 | 0.0224 | 89.97 | 0.00264 | 73.8 | 0.0517
Table 3: Comparison with other baseline methods. The bold represents the
optimal results.

###  5.4 Details

We set the learning rate as 5e-5, the batch size as 16, and the running epoch
as 40. We set the warm up epoches ratio to 0.1, set the mask probability of
MLM to 0.15. We use AdamW as the optimizer [35]. All of the code is written
using PyTorch and runs on NVIDIA RTX 3090.

###  5.5 Overall Results

We report the experimental results of our method on all datasets in Table 2
and compare them with Bert and Bert+MLM. We calculate the Accuracy, Micro-F1,
Micro-Jaccard and Hamming-Loss. Compared with BERT, in most cases, LM has
significant performance improvement in all evaluation metrics, which indicates
that the model performance can be effectively improved after transforming MTC
into template populated tasks. We also notice that adding MLM can further
improve the performance of ML and Bert, which illustrates the effectiveness of
joint training.

We explain this phenomenon from the essence of Bert. When we input the label
and the original sentence together into Bert, it is equivalent to constructing
the context for the label, and self-attention can sensitively capture such
context relations that do not exist in the original sentence. In this way, we
introduce the association among labels which can increase the model’s ability
to understand the label context. In addition, since Bert is essentially an
MLM, allowing Bert to continue learning the mask for different downstream
tasks can improve its performance.

We also compared LM-MTC with some other commonly used baseline models. The
results are shown in Table 3. We notice that approaches with labels
relationships is often superior to methods that do not consider the
relationship between labels. For example, CC is better than BR, and HAN+LG is
better than HAN. This demonstrates the necessary of label relevance. We also
notice that our method outperforms all baselines, because our approach takes
advantage of Bert’s powerful context-capture capabilities to capture semantic
associations between labels. This approach is more efficient than some methods
of explicit label association modeling, such as graph-based or chain-based
methods.

###  5.6 Analysis

####  5.6.1 Attention Visualization

The middle layer of Bert has been shown to adequately capture semantic
relationships between words [10]. In LM-MTC, each potential label can be
treated as a word, so we verify how the LM-MTC captures the correlation of
labels by visualizing the attention of each layer.

Figure 3(a) shows the Spearman correlation between different labels of AAPD
testset. To facilitate the observation, we chose the top11 labels with high
relevance. Similar to Figure 1, test labels have a similar correlation
distribution with train dataset.

After that, we take the attention output parameters of different Bert layers.
We average all the attention heads and select the attention scores between all
the label pairs. We add up all the batch data to get a global score matrix of
attention on testset. We select the attention matrix of the 2nd layer (Figure
3(b)), the 6th layer (Figure 3(c)) and the last layer (Figure 3(d)) for
visualization. Bert in the shallow layer learns some rough information, the
6th layer pays more attention to the local correlation, and the attention of
the last layer is closer to the original label correlation distribution. This
shows that deep Bert can capture the correlation between labels, which also
provides a valid explanation for the advantages of LM-MTC.

####  5.6.2 MLM Loss Ratio

For multi-tasking learning, different task Loss should be of similar magnitude
[36]. According to the changes in MTC and MLM loss as shown in Figure 4, we
select different $\lambda$ to investigate the effect of MLM task weight on
model performance.

Figure 4: Changes of every 10 batch loss function during training.

As is shown in Figure 5(b), for two different datasets, RMSC and Emotion, the
best performance occurs at $\lambda=0.05$, $\lambda$ is too large or too small
will have a negative impact on the model. Therefore, So to keep the MTC task
dominant, and to make the MLM task have enough impact, we set $\lambda=0.05$
for all datasets.

(a) RMSC.

(b) Emotion.

Figure 5: Model performance under different $\lambda$.

####  5.6.3 Different Mask Strategies

In fact, different templates may have different effects on the same task [25].
As a comparison, we assign the same token to each label in different
positions, that is, we modify the template in Eq 2 as follows:

$\begin{split}\color[rgb]{0,0,0}{[LS][YES][LE]}\\\
\color[rgb]{0,0,0}{[LS][NO][LE]}\\\
\color[rgb]{0,0,0}{[LS][MASK][LE]}\end{split}$ (8)

| RMSC | Emotion
---|---|---
| Diff | Same | Diff | Same
Accuracy | 38.39 | 21.67 | 47.63 | 37.31
Micro-F1 | 73.8 | 60.63 | 59.07 | 51.38
Micro-Jaccard | 58.47 | 43.5 | 41.92 | 34.57
HL | 0.0517 | 0.065 | 0.0304 | 0.0323
Table 4: The influence of different mask strategies on the results.

We repeat the experiment on RMSC and Emotion, and the results are shown in
Table 4. When all labels use the same templates, there is a significant
performance drop. This means that different label templates can better provide
identification information for Bert, but same template does not.

##  6 Conclusion

In this paper, we propose LM-MTC model for multi-label text classification. We
build prefix templates for multiple labels, transform MTC into cloze task, and
combine training with MLM to improve the performance of the model under a
variety of evaluation metrics. Further, we explain the ability of LM-MTC to
capture the potential association between labels and LM-MTC can perform well
in tests against multiple types of datasets. In future work, we will explore
the ability of different templates to combine with different pre-trained
language models.

##  7 Acknowledgements

This work was supported by National Natural Science Foundation of China
(NSFC), ”From Learning Outcome to Proactive Learning: Towards a Humancentered
AI Based Approach to Intervention on Learning Motivation” (No.62077027).

## References

  * [1] E. Cambria, D. Olsher, D. Rajagopal, Senticnet 3: A common and common-sense knowledge base for cognition-driven sentiment analysis, AAAI (2014) 1515–1521.
  * [2] Z. Yang, D. Yang, C. Dyer, X. He, J. A. Smola, H. E. Hovy, Hierarchical attention networks for document classification, HLT-NAACL (2016) 1480–1489.
  * [3] S. Gopal, Y. Yang, Multilabel classification with meta-level features, SIGIR (2010) 315–322.
  * [4] I. Katakis, I. Vlahavas, G. Tsoumakas, Multilabel text classification for automated tag suggestion, european conference on principles of data mining and knowledge discovery.
  * [5] L. Xiao, X. Huang, B. Chen, L. Jing, Label-specific document representation for multi-label text classification, EMNLP/IJCNLP (1) (2019) 466–475.
  * [6] R. M. Boutell, J. Luo, X. Shen, M. C. Brown, Learning multi-label scene classification, Pattern Recognition (2004) 1757–1771.
  * [7] J. Liu, W.-C. Chang, Y. Wu, Y. Yang, Deep learning for extreme multi-label text classification, SIGIR (2017) 115–124.
  * [8] L. Xiao, X. Zhang, C. Huang, M. Song, L. Jing, Does head label help for long-tailed multi-label text classification, AAAI 2021.
  * [9] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, north american chapter of the association for computational linguistics.
  * [10] G. Jawahar, B. Sagot, D. Seddah, What does bert learn about the structure of language, ACL (1) (2019) 3651–3657.
  * [11] T. Schick, H. Schütze, It’s not just size that matters: Small language models are also few-shot learners, arxiv.
  * [12] T. Schick, H. Schütze, Exploiting cloze-questions for few-shot text classification and natural language inference, EACL (2021) 255–269.
  * [13] E. Hüllermeier, J. Fürnkranz, W. Cheng, K. Brinker, Label ranking by learning pairwise preferences, Artif. Intell. (2008) 1897–1916.
  * [14] J. Read, B. Pfahringer, G. Holmes, E. Frank, Classifier chains for multi-label classification, Machine Learning (2011) 333–359.
  * [15] G. Tsoumakas, I. Vlahavas, Random k-labelsets: An ensemble method for multilabel classification, ECML (2007) 406–417.
  * [16] G. Chen, D. Ye, Z. Xing, J. Chen, E. Cambria, Ensemble application of convolutional and recurrent neural networks for multi-label text categorization, IJCNN (2017) 2377–2383.
  * [17] P. Yang, X. Sun, W. Li, S. Ma, W. Wu, H. Wang, Sgm: Sequence generation model for multi-label classification, COLING (2018) 3915–3926.
  * [18] Z. Barutcuoglu, E. R. Schapire, G. O. Troyanskaya, Hierarchical multi-label prediction of gene function, Bioinformatics (2006) 830–836.
  * [19] M.-L. Zhang, K. Zhang, Multi-label learning by exploiting label dependency, KDD (2010) 999–1008.
  * [20] S. Wang, J. Wang, Z. Wang, Q. Ji, Multiple emotion tagging for multimedia data by exploiting high-order dependencies among emotions, IEEE Trans. Multimedia (2015) 2185–2197.
  * [21] S. Wang, G. Peng, Z. Zheng, Capturing joint label distribution for multi-label classification through adversarial learning, IEEE Trans. Knowl. Data Eng. (2020) 2310–2321.
  * [22] F. Scarselli, M. Gori, C. A. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural network model, IEEE Transactions on Neural Networks (2009) 61–80.
  * [23] L. Xiao, X. Huang, B. Chen, L. Jing, Label-specific document representation for multi-label text classification, EMNLP/IJCNLP (1) (2019) 466–475.
  * [24] P. Ankit, S. Muru, S. Malaikannan, Multi-label text classification using attention-based graph neural network, ICAART: PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON AGENTS AND ARTIFICIAL INTELLIGENCE, VOL 2 (2020) 494–505.
  * [25] T. Shin, Y. Razeghi, L. L. R. IV, E. Wallace, S. Singh, Autoprompt: Eliciting knowledge from language models with automatically generated prompts, empirical methods in natural language processing (2020) 4222–4235.
  * [26] T. B. B., M. Benjamin, R. Nick, S. Melanie, K. Jared, D. Prafulla, N. Arvind, S. Pranav, S. Girish, A. Amanda, A. Sandhini, H.-V. Ariel, K. Gretchen, H. Tom, C. Rewon, R. Aditya, D. Z. M., W. Jeffrey, W. Clemens, H. Christopher, C. Mark, S. Eric, L. Mateusz, G. Scott, C. Benjamin, C. Jack, B. Christopher, M. Sam, R. Alec, S. Ilya, A. Dario, Language models are few-shot learners, NIPS 2020.
  * [27] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, J. Tang, Gpt understands, too.
  * [28] F. Debole, F. Sebastiani, An analysis of the relative hardness of reuters-21578 subsets: Research articles, Journal of the American Society for Information Science and Technology (2005) 584–596.
  * [29] G. Zhao, J. Xu, Q. Zeng, X. Ren, Review-driven multi-label music style classification by exploiting style correlations, arXiv: Computation and Language.
  * [30] D. Dorottya, M.-A. Dana, K. Jeongwoo, C. Alan, N. Gaurav, R. Sujith, Goemotions: A dataset of fine-grained emotions, ACL (2020) 4040–4054.
  * [31] P. Baldi, S. Brunak, Y. Chauvin, A. C. Andersen, H. Nielsen, Assessing the accuracy of prediction algorithms for classification: an overview, BIOINFORMATICS (2000) 412–424.
  * [32] E. R. Schapire, Y. Singer, Improved boosting algorithms using confidence-rated predictions, Machine Learning (1998) 80–91.
  * [33] Y. Kim, Convolutional neural networks for sentence classification, EMNLP (2014) 1746–1751.
  * [34] H. Liu, C. Yuan, X. Wang, Label-wise document pre-training for multi-label text classification, international conference natural language processing (2020) 641–653.
  * [35] I. Loshchilov, F. Hutter, Fixing weight decay regularization in adam, arXiv: Learning.
  * [36] Z. Chen, V. Badrinarayanan, C.-Y. Lee, A. Rabinovich, Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, international conference on machine learning.