# MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees

ZhenZhe Ying
Ant Group
HangZhou, Zhejiang
<EMAIL_ADDRESS>
&Zhuoer Xu
Nanjing University
Nanjing, Jiangsu
<EMAIL_ADDRESS>
&Zhifeng Li
Ant Group
HangZhou, Zhejiang
<EMAIL_ADDRESS>
&Weiqiang Wang
Ant Group
HangZhou, Zhejiang
<EMAIL_ADDRESS>
&Changhua Meng
Ant Group
HangZhou, Zhejiang
<EMAIL_ADDRESS>

###### Abstract

Despite the success of deep learning in computer vision and natural language
processing, Gradient Boosted Decision Tree (GBDT) is yet one of the most
powerful tools for applications with tabular data such as e-commerce and
FinTech. However, applying GBDT to multi-task learning is still a challenge.
Unlike deep models that can jointly learn a shared latent representation
across multiple tasks, GBDT can hardly learn a shared tree structure.

In this paper, we propose Multi-task Gradient Boosting Machine (MT-GBM), a
GBDT-based method for multi-task learning. The MT-GBM can find the shared tree
structures and split branches according to multi-task losses. First, it
assigns multiple outputs to each leaf node. Next, it computes the gradient
corresponding to each output (task). Then, we also propose an algorithm to
combine the gradients of all tasks and update the tree. Finally, we apply MT-
GBM to LightGBM. Experiments show that our MT-GBM improves the performance of
the main task significantly, which means the proposed MT-GBM is efficient and
effective.

##  1 Introduction

Gradient boosting is a powerful machine-learning technique that achieves
state-of-the-art results in a variety of applications. It is a process of
constructing a strong predictor by ensembling weak predictors and performing
gradient descent in a functional space. Gradient boosting has achieved
tremendous success in a wide range of machine learning tasks. It is popular in
applications with heterogeneous features, noisy tabular data, and complex
dependencies, such as recommender systems (Wang et al., 2016; Wen et al.,
2019), web search (Agrawal & Shanahan, 2010), FinTech (Wang et al., 2018; Mei
& Li-ren, 2015; Yu-Zhuo & Xiao-Ni, 2006), online advertising, etc (Roe et al.,
2004; Zhang & Haghani, ). Among all gradient boosting algorithms, XGBoost and
LightGBM are the two most famous algorithms. They have contributed to
countless top solutions in machine learning and data mining competitions on
tabular data.

Despite the considerable success of gradient boosting, applying GBDT models to
multi-task learning is still challenging. The GBDT model aims to train a high-
performance discriminative model towards a single loss function of a single
task. Therefore, it cannot learn a shared latent feature representation, which
is essential for multi-task learning.

We list three things that limit the applicability of GBDT in multi-task
learning.

First and foremost, the core of efficient multi-task learning is to extract
valuable hidden representation that shares across all tasks, which is non-
trivial for tree-based ensemble learning methods like GBDT. For example, in
news recommendation, user response prediction tasks include the prediction of
click-through rate, dwell-time, thumbs, reviews, shares, and so on. These
objects correspond to the same user; thus an informative representation of the
user is the key to such a multi-task learning problem. The recent development
of deep learning can achieve this naturally using hidden embeddings and layers
in neural networks (Ruder, 2017; Huang et al., 2018; Gao et al., 2019), thus
has shown its dominance for multi-task learning in computer vision and natural
language processing (Su et al., 2015; Radford et al., 2019; Liu et al., 2019).
However, it is known that neural networks sometimes fail in applications with
tabular data.

Second, the splitting mechanism for decision trees aims to separate the input
space as well as the training samples according to a single loss function,
which could not be applied directly to multi-task learning where there are
multiple losses. When optimizing end-to-end neural networks, it is natural to
sum up the gradients from all losses. However, for tree-based models, the
splits that have maximal information gain for the losses can differ a lot,
especially when training classification and regression tasks at the same time.
Therefore, it is challenging to find tree splits that are suitable for all
tasks.

Third, in multi-task learning, the model should keep similar paces for each
task for stable learning. For neural networks, current gradient descent
methods with adaptive step size can adjust the learning rate for different
tasks. However, the learning speed of GBDT is related to the residual errors.
Therefore, when there are multiple tasks, if a fixed learning rate is applied
to all tasks, the model tends to overfit certain tasks while underfit the
others. Therefore, how to adjust learning rates for multiple tasks remains a
challenge.

In this paper, we propose a new algorithmic technique for multi-task learning
based on GBDT, namely multi-task Gradient Boosting Machine (MT-GBM). The key
novelty of MT-GBM is that it learns shared decision tree structures for
multiple tasks, which can be understood as the hidden knowledge across all
tasks. As the decision trees are shared, it requires almost the same
computational overhead comparing to a single task GBDT. Therefore, MT-GBM can
be efficiently trained. We verified MT-GBM on two financial datasets.
Experimental results show that MT-GBM can achieve significantly higher
performance than existing state-of-the-art gradient boosted decision trees
methods, XGBoost (Roe et al., 2004), LightGBM (Ke et al., 2017) and Catboost
(Prokhorenkova et al., 2018) .

Our contribution can be summarized as follows:

  * •

We propose MT-GBM that learns an ensemble of shared decision trees learned
from multiple losses simultaneously, which significantly improves the
efficiency and effectiveness of GBDTs for multi-task learning.

  * •

We propose a new learning algorithm that adaptively adjusts the learning rate
for each task by taking the scale of residual errors into consideration. It
stabilizes training on arbitrarily complex problems.

  * •

Our method is easy to implement upon existing GBDT models such as XGBoost and
LightGBM. We tested our method by modifying LightGBM and showed significant
performance improvements.

##  2 Preliminaries

###  2.1 Multi-Task learning with decision trees

It is widely known that multi-task learning could improve the performance of
every single task by sharing knowledge from all tasks. When using max-margin
machine learning (i.e., SVMs) or deep learning (i.e., feed-forward neural
networks), multi-task learning has achieved significant success (Caruana,
1997; Evgeniou & Pontil, 2004; Bingel & Søgaard, 2017). However, with
decision-tree-based models, it remains unclear how to learn shared knowledge
from multiple tasks.

There are many potential reasons why adding extra outputs to a model might
improve generalization performance. For example, to the extent that tasks are
uncorrelated, their contribution to the aggregated gradient, which guides the
node splitting in GBDT, can appear as noise to other tasks. Thus uncorrelated
tasks might improve generalization by acting as a source of the noise. Figure
1 (a) shows a situation of data samples with a noisy label. One single sample
contains two rectangles while the left represents label 1 and the right
represents label 2. We can assume that task 1 clicks behavior and task 2 is
shares behavior in the recommendation system. It makes it clear that two tasks
fix the splitting line from dotted line to solid line. Figure 1 (b) shows that
Sub tasks also help the sample splitting when dataset has the class imbalance
problem.

Figure 1: (a) Samples with noisy tasks. Each sample contains two rectangles as
tasks and the color means whether it is positive or not. There are some noises
in each task in the middle of the figure. Take two tasks into consideration is
better than any one of them;(b) Samples with sub tasks. Task 2 is a subclass
of task 1, and it helps the model to split the small group from the big group
which is not accurate enough. Finally model will predict a high probability
for sample in small group and a medium in big group.

###  2.2 Problem Definition

GBDT is an ensemble model of decision trees, which are trained in sequence. In
each iteration, GBDT learns a new decision tree by minimizing the negative
gradients (also known as the residual errors) of previously trained trees.
Specifically, to construct the decision trees, the algorithm finds the split
points in the feature space according to the residual errors.

Assume we have $n$ learning tasks $t\in\\{1,2,...,n\\}$, each with a set of
supervised data $\\{(x_{1},y_{1}),...,(x_{m},y_{m})\\}$. For simplicity, we
assume that the number of data points $m$ in each task is the same, and all
the tasks share the same input space. For the $t$-th task, there is a
predefined learning objective $L_{t}$. In regression, we might consider the
mean squared error:

$L_{t}=\frac{1}{m}\sum_{i=1}^{m}{(y_{i}-p_{i})^{2}}$ (1)

And we use the negative log-likelihood as the classification loss:

$L_{t}=-\frac{1}{m}\sum_{i=1}^{m}{(y_{i}\log p_{i}+(1-y_{i})\log(1-p_{i}))}$
(2)

Finally, we assume the main task objective function $L_{1}$ as the metric to
evaluate the model.

###  2.3 Related Work

In this section, we ignore the related work of neural network because most of
neural network work well in specific tabular dataset (Zhang & Yang, 2018;
Ruder, 2017) .

To achieve a multi-task tree, there were some works about multi-classifier
tree on Adaboost (Faddoul et al., 2012) . This paper revises the information
gain rule for learning decision trees and modified MT-Adaboost to combine
multi-task Decision Trees as weak learners. Since its algorithm depends on
their new equation of Information Gain, the model does not support doing the
regression task yet. Obviously, it fails when it comes to residual errors in
GBDT cause the loss is float number even in classifier tasks.

Another approach to multi-task learning is the work sharing tree structure for
different tasks (Chapelle et al., 2011). This paper presents a general
solution of boosting for different task types by minimizing the sum of losses
$\sum_{i=1}^{n}{L_{i}}$. It uses a relative value to control the strength of
the connection between the tasks. We approve of the way about exploring the
general solution. But the part of optimization in which splitting the node
would be hard and slow when consider minimizing all loss functions. It seems
to be a bit complex and obsolete and is beaten by the existing implementations
of gradient boosted decision trees such as XGBoost and LightGBM.

However, the GBDTs has some tricks to deal with multi-class task (Fan et al.,
2017), i.e. training _n_ trees in one iteration for _n_ classes. When
inferring, the trees belong to the specified class will be activated.

Algorithm 1 Traditional Multi-Class GBDT

0: $n$ the number of tasks, $m$ the number of samples. Accumulative prediction
of previous rounds:$\\{p_{1,t},\dots,p_{m,t}\\}_{t=1}^{n}$; Goundtruth
label:$\\{y_{1,t},\dots,y_{m,t}\\}_{t=1}^{n}$

0: New decision trees $T_{1},T_{2},...,T_{n}$ in this iteration.

1: for $i=1,\dots,m$ do

2: $p_{i,t}$ = $\frac{\exp(p_{i,t})}{\sum_{t=1}^{n}{\exp(p_{i,t})}}$,
$t=1,\dots,n$

3: for $t=1,\dots,n$ do

4: // J is the number of nodes in the tree

5: // N is the number of samples in the node j

6: $\tilde{y}_{j,t}=y_{j,t}-p_{j,t}$, $j=1,\dots,N$

7: // $R_{jti}$ means that the $i$th sample belongs to the $j$ node at the
$t$th task

8: // $\gamma_{jti}$ means the prediction of $i$ sample at the $t$th task

9: $\\{R_{jti}\\}_{j=1}^{J}$ =
$J-\text{terminalNodeTree}(\\{\tilde{y_{lt}},x_{l}\\}_{l}^{N})$

10: $\gamma_{jti}$ =
$\frac{K-1}{K}\frac{\sum{\tilde{y}_{lt}}}{\sum{|\tilde{y}_{lt}|(1-|\tilde{y}_{lt}|)}}$,
$j=1,\dots,J$

11: end for

12: end for

But the trees for different classes have no interaction when training except
the last softmax transformation. The trees for the label with few positive
samples would have a terrible training, since it can not learn things from the
other labels.

Moreover, it has a big problem that it costs _n_ times of resources when
training and inferring. There were papers that implied the multi-class by
solving the sparse outputs problem (Si et al., 2017) and achieving an order of
magnitude improvements in model size and prediction time over existing
methods. But it still doesn’t seem to be a multi-task solution, while
realistic datasets above always have dense outputs and multi-class which can
be positive at the same time. Therefore we call it a multi-class solution, not
multi-task.

To address the limitations of previous works, we propose two new novel
techniques called multi-output tree and multi-task splitting.

##  3 Multi-Output tree

The multi-task demands a new structure called multi-output tree to save the
intermediate results. The new model should manage to specify weights for each
label, and update leaf outputs with specified rates, which make it necessary
to multiply gradients of the samples with the number of labels.

In this section, we propose a new structure tree by modifying the traditional
decision tree, which almost has no disruptive innovation to the GBDT training
procedure. Multi-output Tree shares same structure and same leaf instead of
sharing other parameters. And we enrich the node and leaf information which
contains Gradients, Hessians, mean value of the data at the node and etc.
There will be a discussion about them below:

Firstly, each sample in the dataset should have _n_ predictions, _n_ Gradients
and _n_ Hessians, where the _n_ is the number of labels. They should be
calculated at the beginning of each iteration like old GBDT. And it causes a
problem that the histogram object(Ke et al., 2017) which is used to speed up
splitting could only be calculated by one Gradient and one Hessian for each
sample.

Then, we separate the concept of Gradients and Hessians. When training the
tree, old Gradients and Hessians were designed both for splitting the node and
updating the output. In MT-GBM, we denote the Gradients and Hessians used for
updating as _${\bm{G}}_{u}$_ and _${\bm{H}}_{u}$_ , respectively. The
Gradients and Hessians used for splitting the node are called ensemble _G e_
and _H e_, while it has one value for each sample, which determines how the
tree splits. And more discussion about the ensemble _G e_ & _H e_ and updating
_G u_ & _H u_ will be shown in the next sections.

Figure 2: (a) Old structure of gradients in the GBDTs. Each rectangle is the
$G$ of one sample. Assume the $H$ equals constant in the square loss and model
splits samples by the $G$; (b) New structure of gradients in the MT-GBM. The
arrows in horizontal direction mean the transformation generating updating $G$
and the arrows in vertical direction mean the transformation generating
ensemble $G$.

At last, we only need to save the mean of residual errors of all the data at
each leaf for each label, which was used to calculated residual errors in
iterations below. It’s easy to support the model to save part of trees for
every single label. Therefore the dump model file will be the same size as the
traditional GBDT and single label can be inferred as the same speed.

##  4 Multi-Task Splitting

In this section, we define how to calculate the two types of _G_ and _H_ and
their usages in MT-GBM.

XGBoost and LightGBM have done a lot of work to make the GBDT efficient by
proposing new loss function and parallel computing when splitting. Their loss
functions were based on Gradients and Hessians of the data:

$L^{*}=-\frac{1}{2}\sum_{j=1}^{T}\frac{G^{2}_{j}}{H_{j}+\lambda}+\gamma T$ (3)

The loss serves as the most principle of the splitting selection, although
XGBoost and LightGBM both have some techniques modify or simplify them in
computational procedure. At each leafs, the _H_ and _G_ means the sum of the
Gradients and Hessians of all the data on it.

###  4.1 Gradients for Splitting

The tree splits depending on the loss function, while the sum of two leafs’
losses should be minimum at the split position. It would be a large amount of
computation if model try to minimize all the losses. Therefore we propose a
new Ensemble Gradient _G e_ and a new Ensemble Hessian _H e_ in the new
equation:

$L^{*}=-\frac{1}{2}\sum_{j=1}^{T}\frac{G^{2}_{ej}}{H_{ej}+\lambda}+\gamma T$
(4)

To consider all the _G_ and _H_ of all type of labels, there would be two
functions that calculate the _G e_ and _H e_: _G e_ =
_$f_{G}$(G1,G2,G3,…,Gn)_, _H e_ = _$f_{H}$(H1,H2,H3,…,Hn_).

Different tasks may bring different distributions of gradients to model. Set a
normalization factor $w$ to normalize all the losses of tasks would be
beneficial to the ensemble. We normalize them to a distribution with 0.05 mean
and 0.01 standard deviation.

We realizes the _$f_{G}$_ and _$f_{H}$_ have to be designed to outstand only
one type of label or several relative labels with positive correlation in one
iteration by trials. That is because different labels will be learned or
fitted at different rates and the _G_ and _H_ will have a completely different
order of magnitude, which is hard to ensemble. Thus we define the _$f_{G}$_
end _$f_{H}$_ by sum them with weights which will randomly concentrate on one
of few tasks. The probability of each task be chosen can be assigned which
represents the importance of tasks.

Algorithm 2 The calculations of Ge and He for one sample before an iteration

0: predication of trees above of each task:_p 1,p2,p3,…,pn_;true label of each
task:_l 1,l2,l3,…,ln_

0:  _G e,He_

1: $n$ = number of tasks

2: for $i=1,\dots,n$ do

3: $G_{i}=\frac{d}{dp}L(l_{i},p_{i})$

4: $H_{i}=\frac{d^{2}}{dp^{2}}L(l_{i},p_{i})$

5: end for

6: Randomly initialize
${{\textnormal{w}}_{1},{\textnormal{w}}_{2},{\textnormal{w}}_{3},...,{\textnormal{w}}_{n}}$
to make ${\textnormal{w}}_{i}*G_{i}$ with $0.05$ mean and standard deviation
$0.01$

7: Randomly choice one or more indexes of tasks from $1$ to $n$ as $k_{1}$ to
$k_{m}$

8: for $k$ = $k_{1}$ , …, $k_{m}$ do

9: $w_{k}=\gamma{\textnormal{w}}_{k}$, which $\gamma$ is a parameter from $10$
to $100$

10: end for

11: $G_{e}=\sum_{j=1}^{n}({\textnormal{w}}_{j}\cdot G_{j})$

12: Randomly initialize
${{\textnormal{v}}_{1},{\textnormal{v}}_{2},{\textnormal{v}}_{3},...,{\textnormal{v}}_{n}}$
to make $v_{i}\cdot H_{i}$ with $1.0$ mean and standard deviation $0.1$

13: $H_{e}=\sum_{j=1}^{n}({\textnormal{v}}_{j}\cdot H_{j})$

14: return $G_{e},H_{e}$

The parameter $\gamma$ is used as a factor to outstanding the chosen tasks. It
could be verified from 10 to 100 by experiments from case by case. The smaller
correlation, which means that there are strong conflicts between tasks, the
better result with big $\gamma$. From the algorithm we could find it is
complex to keep different $G$ and $H$ ensemble balance. We propose a common
one above, and believe there might be other better ensemble algorithms for
particular problems and tasks.

When it comes to a problem with main task and sub tasks, $k_{1}$ to $k_{m}$
should not be selected randomly. We suggest that to always choice the main
task label as the $k$. And there is a helpful method to improve the learning
process, which is to calculate or evaluate the correlation of the main task
with others and give a bigger w and smaller v for the task of positive
correlation.

###  4.2 Gradients for updating leaf value

After defined the $G_{e}$ and $H_{e}$, the GBDT tree learner could build trees
for each iteration. The length of $G_{e}$ and $H_{e}$ should be same as the
length of the number of samples, while the length of Updating Gradient _G un_
and Updating Hessian _H un_ same as the length of the number of samples
multiplying the number of labels.

It is acceptable for the MT-GBM to set the same rate for each label with
traditional GBDT. However, we still have some explorations about the _G un_
and _H un_, and propose the algorithm below with some tricks.

Algorithm 3 The calculations of Gun and Hun for one sample before an iteration

0: predication of trees above of each task :_p 1,p2,p3,…,pn_;true label of
each task:_l 1,l2,l3,…,ln_

0:  _G u1,Gu2,Gu3,…,Gun_,_H u1,Hu2,Hu3,…,Hun_

1: $n$ = number of tasks

2: for $i=1,\dots,n$ do

3: $G_{ui}=\frac{d}{dp}L(l_{i},p_{i})$

4: $H_{ui}=\frac{d^{2}}{dp^{2}}L(l_{i},p_{i})$

5: concatenate the $G_{ui}$ of all the samples as vector $G_{sui}$

6: end for

7: calculate the correlation $corr$ with $n$ vectors
($G_{su1},G_{su2},G_{su3},...,G_{sun}$): $corr$ =
$f(G_{su1},G_{su2},G_{su3},...,G_{sun}$)

8: for $i=1,\dots,n$ do

9: $G_{ui}*=clip(corr,0.5,1)$

10: end for

11: return _G u1,Gu2,Gu3,…,Gun_,_H u1,Hu2,Hu3,…,Hun_

The function $f$ we defined is used to calculate correlation of the gradients
vector of each label in current iteration, which means the updating speed
should be fast if all the gradients have the similar distribution. Actually
whether the calculation of correlation works or not depends on many things
such as type of tasks and meaning of tasks. Therefore set the $corr$ as a
constant equal 1.0 might be better on some complex datasets, which means the
updating rate is same as the traditional single task GBDT. And noticed that we
multiply the same $corr$ for each $G_{ui}$ in one iteration, it is because
that the GBDT learns things from residual errors. Trees get inconsistent paces
if we set different rates for different tasks, which leads to bad results.

##  5 Experiments

###  5.1 Experiment setup

Through experiments, we seek to answer the following questions:

1\. Can MT-GBM achieve better performance state-of-the-art implementations of
gradient boosted decision trees (GBDTs) XGBoost, LightGBM and Catboost?

2\. Does neural networks of multi-task achieves same performance as MT-GBM?

In this section, we report the experimental results regarding our proposed MT-
GBM algorithm which was developed on the LightGBM(The
code111https://github.com/Microsoft/LightGBMhttps://github.com/Microsoft/LightGBM
is available at GitHub). Particularly, the details of experimental setup,
including data description, baseline models, and some specific experiments
settings, will be presented below. All experiments are done with the same
features in GBDTs, Neural Network, and MT-GBM, and the sub tasks in MT-GBM are
derived from the main task. Our experimental environment is a Linux server
with two i9-7980 CPUs (in total 36 cores) and 128GB memories.

###  5.2 Datasets and baselines

We tested the methods on two different real-world datasets. The first one is
part of China Foreign Currency Volume from 2017 to 2019. The dataset is about
the currency exchange volume between one country and China by common people.
The target is simple: use the historic data to predict next day volume and we
make the last 30 days value as the testset. It is a time-series problem, and
it only contains one amount in every workday from 2017 to 2019 and has no
features. Rmse and mape are used to evaluate the models.

GBDT is proved as a great model in time-series cases and features are often be
constructed by sliding window (Lee et al., 2018; Liu et al., 2018; Xia & Chen,
2017). We count the minimum, maximum, average, and variance of the last three
days, one week, two weeks, and one month as the sliding window features. With
these, a baseline is finished and we run XGBoost and LightGBM on it.

And the second dataset is Kaggle Competition:IEEE-CIS Fraud Detection
(222https://www.kaggle.com/c/ieee-fraud-detection), which is a credit fraud
detection case from Vesta Corporation. The target of model is to predict
whether this transaction is fraud or not. There is about 3.5% fraud
transactions in the dataset while ROC-AUC is a good metric for the imbalance
data. There are 600,000 samples in the trainset and 500,000 in the testset.
Since we can not get the true label of testset, we cut the last 20% samples in
the trainset by datetime as the validation data, exactly same as the reality.

About 400+ original features, contains user information, shop information and
transactions details and anonymous features, are provided by the official. As
a supplement, 100+ features are mined from raw features in a baseline solution
of the golden section. We ensure that the baseline is a top single model by
the leaderboard. We run XGBoost, LightGBM, CatBoost and MT-GBM with the
setting copied from the open source code.

###  5.3 Results

Table 1: Results of models for experiment 1 Models | best mape | mape of iteration 150 | best rmse | rmse of iteration 150
---|---|---|---|---
XGBoost | 0.0396 | 0.0475 | 316.43 | 362.50
LightGBM | 0.0438 | 0.0452 | 400.32 | 403.47
NN | 0.0652 | - | 453.87 | -
MT-GBM | 0.0387 | 0.0423 | 308.21 | 336.82

In the dataset of Currency Volume, the trend of amount may have some regular
patterns. Only the label of amount itself is not enough. We propose that a sub
label is the volume of next day divided by current day, while the main label
is the volume of next day. Two labels have different orders of magnitude while
the sub label is a number floating around 1.0 and the main label is great than
$10^{8}$.

All tree models in experiment 1 perform not badly as Table 1 shown. XGBoost is
better than LightGBM at the same parameters on both metrics.

The result of first experiment shows the effects of ensemble numerical labels
with different orders of magnitude. We compare the LightGBM and MT-GBM as the
improvement of multi-task model, as we add the multi-task algorithm on the
LightGBM repository. From Table 1, the best mape is reduced from 4.3% to 3.9%
and the best rmse is reduced from 400.32 to 308.21.

We conclude the results that multi-task teach model not only the volume of
currency but also the rise and fall which make model robust at small dataset
by splitting at a better position, meanwhile slowing down convergence speed of
the model.

Table 2: Details of sub tasks for experiment 2 Task index |  Definition |  Correlation with main task
---|---|---
1 |  whether this card is fraud with same amounts in two days |  0.418
2 |  whether this ip is fraud in two days |  0.309
3 |  whether this card is fraud in same shop |  0.415
4 |  what percentage of the transactions of same amounts is fraud in two days |  0.461
Table 3: Results of models for experiment 2 Models | sub task list | 3-folds’ ROC-AUC | fold 1 | fold 2 | fold 3
---|---|---|---|---|---
XGBoost | - | 0.9443 | 0.9393 | 0.9417 | 0.9364
CatBoost | - | 0.9371 | 0.9302 | 0.9309 | 0.9311
LightGBM | - | 0.9442 | 0.9383 | 0.9408 | 0.9369
NN-4 Tasks | [1,2,3] | 0.8410 | 0.8235 | 0.8346 | 0.8273
MT-GBM-2 Tasks | [1] | 0.9447 | 0.9402 | 0.9410 | 0.9368
MT-GBM-2 Tasks | [2] | 0.9441 | 0.9384 | 0.9402 | 0.9374
MT-GBM-2 Tasks | [3] | 0.9446 | 0.9397 | 0.9405 | 0.9380
MT-GBM-3 Tasks | [1,2] | 0.9441 | 0.9391 | 0.9388 | 0.9370
MT-GBM-3 Tasks | [2,3] | 0.9448 | 0.9397 | 0.9405 | 0.9374
MT-GBM-3 Tasks | [1,3] | 0.9441 | 0.9395 | 0.9396 | 0.9359
MT-GBM-4 Tasks | [1,2,3] | 0.9454 | 0.9402 | 0.9407 | 0.9383
MT-GBM-5 Tasks | [1,2,3,4] | 0.9443 | 0.9393 | 0.9400 | 0.9372

However, the second dataset has not provided multi labels too. In the risk
management, fraud transaction is often divided into many classes by the type
of fraud. Table 2 shows the details of sub tasks.

The dataset is divided into 3 folds in order to reduce the influence of
randomness. The final result contains the average of 3 folds and single fold
versions. Since we have 4 sub tasks in the case, some combinations have been
tested. From Table 3, most of the multi-task experiments behave better than a
single task on the ROC-AUC of the main task. We can find that 2-tasks models
with sub task 1 or 3 have presentable results and the 4-tasks model has the
best performance through comparative analyses between LightGBM and MT-GBM. Low
correlation task means irrelevant task which is helpless to the main task and
high correlation seem to have less supplement for the main task, so the task 1
and 3 are the best for this case. And the Catboost and XGBoost with single
task can’t reach the metric of MT-GBM. It is worthwhile mentioning that this
experiment is training as the outstanding parameter $\gamma$ equals 50, cause
the correlation between sub task and main task seems not so strong. And the
improvement will be descended when $\gamma$ equals 10. We regard this case as
an example that our algorithm makes a balance on the conflicting tasks.

Somehow the more tasks do not bring the better result. We concluded it the
ensemble method of Gradients and Hessians is not perfect enough, failing to
express too many tasks.

###  5.4 Comparison with neural networks

We also try neural networks because it is a natural multi-task model. There
are lots of big numerical numbers in the features disturbing the parameters in
the neural cells. Therefore we have to transform them through $F(x)$:

$F(x)=log_{10}(x+1)$ (5)

On the other hand, we fill the NaN value in the features with 0. And after
this data preprocessing, we create a simple Multi Layer Perceptron network
structure with 2 hidden layers and GELUs activation function (Hendrycks &
Gimpel, 2016) to fit the dataset. The result is shown in Tables 1 and 3.

The neural network performs badly in both experiments as we explain before. It
is not an unexpected result while most of these problems are solved with GBDT
in the industry.

##  6 Conclusion

In this paper, we have proposed a new general algorithm, multi-task gradient
boosted tree, basing on gradient boosted decision tree, which provides
powerful features to the tree model. We have performed both theoretical
analysis and experimental studies on the implementation. The experimental
results are consistent with the theory and show that with the help of multi-
task, MT-GBM can significantly outperform traditional GDBTs. And the MT-GBM
has great extensibility as it didn’t place restrictions on the datasets and
the type of task.

## References

  * Agrawal & Shanahan (2010) Ritesh J Agrawal and James G Shanahan.  Location disambiguation in local searches using gradient boosted decision trees.  In _Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , pp. 129–136, 2010.
  * Bingel & Søgaard (2017) Joachim Bingel and Anders Søgaard.  Identifying beneficial task relations for multi-task learning in deep neural networks.  _arXiv preprint arXiv:1702.08303_ , 2017.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Chapelle et al. (2011) Olivier Chapelle, Pannagadatta Shivaswamy, Srinivas Vadrevu, Kilian Weinberger, Ya Zhang, and Belle Tseng.  Boosted multi-task learning.  _Machine learning_ , 85(1-2):149–173, 2011.
  * Evgeniou & Pontil (2004) Theodoros Evgeniou and Massimiliano Pontil.  Regularized multi–task learning.  In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 109–117, 2004.
  * Faddoul et al. (2012) Jean Baptiste Faddoul, Boris Chidlovskii, Rémi Gilleron, and Fabien Torre.  Learning multiple tasks with boosted decision trees.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pp. 681–696. Springer, 2012.
  * Fan et al. (2017) Cun-qun Fan, Xian-gang Zhao, Man-yun Lin, Li-zi Xie, You Ma, and Xiao-hu Feng.  A multitasking run time prediction method based on gbdt in satellite ground application system.  In _Current Trends in Computer Science and Mechanical Automation Vol. 2_ , pp. 145–150. Sciendo Migration, 2017.
  * Gao et al. (2019) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin.  Neural multi-task recommendation from multi-behavior data.  In _2019 IEEE 35th International Conference on Data Engineering (ICDE)_ , pp. 1554–1557. IEEE, 2019.
  * Hendrycks & Gimpel (2016) Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Huang et al. (2018) Jizhou Huang, Wei Zhang, Yaming Sun, Haifeng Wang, and Ting Liu.  Improving entity recommendation with search log and multi-task learning.  In _IJCAI_ , pp. 4107–4114, 2018.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  Lightgbm: A highly efficient gradient boosting decision tree.  In _Advances in neural information processing systems_ , pp. 3146–3154, 2017.
  * Lee et al. (2018) Ritchie Lee, Mykel J Kochenderfer, Ole J Mengshoel, and Joshua Silbermann.  Interpretable categorization of heterogeneous time series data.  In _Proceedings of the 2018 SIAM International Conference on Data Mining_ , pp. 216–224. SIAM, 2018.
  * Liu et al. (2018) Song Liu, Yaming Cui, Yaze Ma, and Peng Liu.  Short-term load forecasting based on gbdt combinatorial optimization.  In _2018 2nd IEEE Conference on Energy Internet and Energy System Integration (EI2)_ , pp. 1–5. IEEE, 2018.
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_ , 2019.
  * Mei & Li-ren (2015) SHAO Mei and JIANG Li-ren.  The application of decision tree in logistics finance credit risk.  _Logistics Sci-Tech_ , 2015.
  * Prokhorenkova et al. (2018) Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin.  Catboost: unbiased boosting with categorical features.  In _Advances in neural information processing systems_ , pp. 6638–6648, 2018.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Roe et al. (2004) Byron P. Roe, Hai Jun Yang, Ji Zhu, Yong Liu, and Gordon Mcgregor.  Boosted decision trees as an alternative to artificial neural networks for particle identification.  _Nuclear Instruments & Methods in Physics Research_, 543(2–3), 2004.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Si et al. (2017) Si Si, Huan Zhang, S Sathiya Keerthi, Dhruv Mahajan, Inderjit S Dhillon, and Cho-Jui Hsieh.  Gradient boosted decision trees for high dimensional sparse output.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pp. 3182–3190. JMLR. org, 2017.
  * Su et al. (2015) Chi Su, Fan Yang, Shiliang Zhang, Qi Tian, Larry S Davis, and Wen Gao.  Multi-task learning with low rank attribute embedding for person re-identification.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3739–3747, 2015.
  * Wang et al. (2018) Maoguang Wang, Jiayu Yu, and Zijian Ji.  Credit fraud risk detection based on xgboost-lr hybrid model.  2018\.
  * Wang et al. (2016) Yaozheng Wang, Dawei Feng, Dongsheng Li, Xinyuan Chen, Yunxiang Zhao, and Xin Niu.  A mobile recommendation system based on logistic regression and gradient boosting decision trees.  In _2016 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1896–1902. IEEE, 2016.
  * Wen et al. (2019) Hong Wen, Jing Zhang, Quan Lin, Keping Yang, and Pipei Huang.  Multi-level deep cascade trees for conversion rate prediction in recommendation system.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 338–345, 2019.
  * Xia & Chen (2017) Ying Xia and Jungang Chen.  Traffic flow forecasting method based on gradient boosting decision tree.  In _2017 5th International Conference on Frontiers of Manufacturing Science and Measuring Technology (FMSMT 2017)_. Atlantis Press, 2017\.
  * Yu-Zhuo & Xiao-Ni (2006) A. N. Yu-Zhuo and L. U. Xiao-Ni.  Enterprise credit risk evaluation based on the decision tree.  _Journal of Hebei University of Technology_ , 35(6):64–68, 2006.
  * (27) Yanru Zhang and Ali Haghani.  A gradient boosting method to improve travel time prediction.  _Transportation Research Part C Emerging Technologies_ , pp. S0968090X15000741.
  * Zhang & Yang (2018) Yu Zhang and Qiang Yang.  An overview of multi-task learning.  _National Science Review_ , 5(1):30–43, 2018\.

##  Appendix A Implementation Details

Codebase. For the Ieee-Fraud dataset, we use the codebase from
333https://www.kaggle.com/whitebird/ieee-internal-blend. The repository has
done feature engineering work for the dataset from a realistic scene. We use
it directly. We use the cross-validation evaluation method and divide the
dataset into four folds as 3-folds train and 1 hold-out validation. For each
fold, we use a hold-out set as the validation set and train a machine to learn
the model from scratch using the remaining data. We set the same random seed
so that the result of each fold is comparable.

Custom Object Function. The _G e_, _G u_, _H e_, and _H u_ can be implied in
the $fobj$ function in the Train API. We build the Multi-Task framework for
GBDTs and leave the custom function because we believe the high-level user
would find better ensemble methods for Gradients and Hessians in different
scenes.

Hyperparameters. For the Currency dataset, we tuning the best hyperparameters
for all models. We run XGBoost, LightGBM, and MT-GBM with the setting: a small
tree depth as 6, learning rate as 0.03, and lambda $L1$ as 0.1 since time-
series cases is easy to overfit.

For the Ieee-Fraud dataset, we copy the hyperparameters from Github: max depth
as 16, learning rate as 0.03, lambda $L1$ as 0.1, and max leaves as 500.
Settings of XGBoost and Catboost have small differences to get better results.

Comparison with neural networks. Note that the neural network in our paper is
the MLP, since other networks did not fit for tabular data. The network
hyperparameters are the best in the two datasets while we get them by grip
search.

##  Appendix B Additional experiment results

For clearness, we only show the final metric in the main text. There are some
visualizations of intermediate process here.

Figure 3: Train and Valid MAPE. Figure 4: Train and Valid MSE.

Figures 3 and 4 show the metric during training in the Currency dataset. The
lines of single task(blue and yellow) are declined fast and overfit at about
iteration 150. The lines of Multi-Task(green and red) behaves better at both
metrics. However, the valid metric of Multi-Task overfits finally too, as a
result of a normal phenomenon in the tree model.

Figure 5: Roc-auc curves.

In Ieee-Fraud experiment we have defined 4 sub tasks for the main task. The
first three of them are the different types of fraud extracted from the main
task. Take the label of main task and first three tasks as $y_{gt1}$ to
$y_{gt4}$, the predictions of single task LGB model as $y_{single}$, and the
predictions of MT-GBM as $y_{mt1}$ to $y_{mt4}$. Figure 5 shows the Roc-auc
curves of different outputs.


# DataOps for Societal Intelligence: a Data Pipeline for Labor Market Skills
Extraction and Matching

Damian A. Tamburri  Technical University Eindhoven
Jheronimus Academy of Data Science
s’Hertogenbosch, The Netherlands
<EMAIL_ADDRESS>Willem-Jan Van Den Heuvel  Tilburg University
Jheronimus Academy of Data Science
s’Hertogenbosch, The Netherlands
<EMAIL_ADDRESS>Martin Garriga  Tilburg University
Jheronimus Academy of Data Science (JADS)
s’Hertogenbosch, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Big Data analytics supported by AI algorithms can support skills localization
and retrieval in the context of a labor market intelligence problem. We
formulate and solve this problem through specific DataOps models, blending
data sources from administrative and technical partners in several countries
into cooperation, creating shared knowledge to support policy and decision-
making. We then focus on the critical task of skills extraction from resumes
and vacancies featuring state-of-the-art machine learning models. We showcase
preliminary results with applied machine learning on real data from the
employment agencies of the Netherlands and the Flemish region in Belgium. The
final goal is to match these skills to standard ontologies of skills, jobs and
occupations.

EDICS Category: 3-BBND

##  I Introduction

One of the major downsides of our digital economy is the increasing
predominance with which technology has substituted human labour and skills. On
one hand, computing skills are increasing in demand and low in supply. On the
other hand, formerly needed skills and labour areas often become subject to
crisis. We argue that these are _*societal*_ shortcomings and reflect just one
example of the relative lack of _societal intelligence_ ; that is, the ability
of our society at large to harness the digital ecosystem and turn it into an
opportunity. In particular, Big Data analytics supported by DataOps methods
and AI algorithms now offer the potential to make the act of finding the right
skills into a labor market intelligence problem. The challenges we face in
this context are manifold:

(i) Data Integration of data sources from administrative and technical
partners (job seekers, employers, trainers, public authorities, etc) in
several countries into cooperation, creating shared knowledge to support
policy and decision-making;

(ii) Extracting available/required skills and employment options, specially in
low-compliance areas (e.g., borderlands); and,

(iii) Matching skills from resumes with vacancies automatically, leveraging
structured information about Skills, Competences and Opccupations from
ontologies and taxonomies such as the EU standards
ESCO111https://ec.europa.eu/esco/ and ISCO [1].

(iv) Assess, Improve and Extend the existing ontologies and taxonomies with
the descriptions of novel, arising job profiles and skills.

In this context, the main contributions of this paper target the following
research questions:

RQ1. How to model a DataOps Pipeline to enable Societal and Labour Market
Intelligence in the context of skills extraction and matching?

We design such pipeline addressing the aforementioned challenges, to move from
Big Data to Knowledge. Then we illustrate how to automate and scale the
analysis of job vacancies and resumes, and then matching against
representative ontologies (such as ESCO/ISCO).

RQ2. How to extract and match the skills (from vacancies and resumes) in an
effective and efficient way?

We instantiate a pipeline with hundreds of thousands vacancies stemming from
the Dutch-Flemish labour market and then solve the skills extraction problem
featuring state-of-the-art machine learning models.

The rest of this paper is structured as follows. Section II presents related
work and background definitions. Section III describes the DataOps design for
Societal and Labour Market Intelligence. Section IV showcases how to apply the
DataOps pipeline for the skills extraction and matching problem. Section V
presents the validation with real data of the Dutch-Flemish cross-border labor
market. Section VI discusses main findings and limitations. Finally, Section
VII concludes the paper.

##  II State of the Art

###  II-A Societal and Labor Market Intelligence

Societal intelligence is the family of approaches, services, and platforms
that adopt Big data analytics, machine-learning or other technologies specific
to business intelligence over social, organisational, and societal data for
the purpose of a more instrumented, self-sustainable society. Furthermore,
Labor market intelligence (LMI) refers to the design and use of Big Data and
AI algorithms and frameworks to analyze labor market information for
supporting policy and decision-making [2, 3, 4]. Labor market information
encompasses skills, competencies, qualifications, and occupations, including
the ICT techniques and services to manage such information, in particular,
mobility-related services. Key to LMI is the identification and adoption of
standard taxonomies for occupations and skills, to foster the circulation of
information in a multi-language job market like the European one.

###  II-B Labor Market Ontologies

The standard ontology to classify occupations for the international labour
market is ISCO [5] [1], a four-level hierarchy; the interested reader may
consider [5] for more details. ISCO is one of the most adopted taxonomies in
Europe and it is a reference worldwide, while the United States developed its
own, namely O*NET222https://www.onetonline.org/.

For skills description one can find broad regional classifications such as
ESCO333https://ec.europa.eu/esco/resources/data/static/model/html/model.xhtml.
ESCO is a multilingual taxonomy of European Skills, Competences,
Qualifications and Occupations that extends ISCO through a further level of
fine-grained occupation descriptions, and a taxonomy of skills, competencies
and qualifications. One can navigate the ISCO-ESCO cross-linkage to access the
list of occupation examples containing a set of skills/competences and
qualifications requested by the corresponding occupation.

Specific classifications can be found for each country or cross-border region.
As a relevant example within this work, think of
Competent444https://competent.vdab.be/competent/, developed in Belgium and
recently extended to the Netherlands as Competent-
NL555http://production.competent.be/competent-nl/main.html. Competent and
other fine-grained classifications can be easily linked to ISCO and ESCO
through mapping tables [6].

###  II-C DataOps

Like DevOps, DataOps aims to combine the production, operation and delivery
(of data) into a single, agile practice that directly supports specific
business functions to improve quality, speed, and collaboration and promote a
culture of continuous improvement [7]. A DataOps methodology combines and
interconnects data engineering, data integration, data quality, and data
security/privacy [8] to deliver data from its source to the person, system, or
application that can turn it into business value [9].

##  III Big data for Societal Intelligence: A proof-of-concept for the Labor
Market

Figure 1: Big Data enables Societal and Labor Market Intelligence. The process
specifies which Vs of Big Data are involved in each step (adapted from [3])

The extraction of knowledge from (Big) labor market data has been addressed
using the knowledge discovery in databases (KDD) approach as baseline [10] –
as shown in Figure 1, together with the span of the Vs of Big Data throughout
the process.

#### Step 1: Select Data Sources

Each Web source is evaluated and ranked to guarantee its reliability – e.g.,
vacancy publication date, update frequency, (un)structured data, privacy
restrictions, etc. Heterogeneous Data sources are considered here, including
but not limited to, governmental employment offices (from different
regions/countries), private Websites (such as indeed.com) and social networks
(such as Linkedin). At the end of the process, the data sources are selected.
This step deals with the 4 Vs of Big Data including veracity, intended as the
biases, noise and/or abnormalities in data.

#### Step 2 Pre-processing Selected Data

This includes data cleansing tasks to remove noise from the data or
inappropriate outliers (if any), deciding how to handle missing data, as well
as removing duplicated entries. Data quality and cleaning tasks are indeed
mandatory steps of any data-driven decision-making approach for guaranteeing
the trust on the overall process, intended as the ”the extent to which data is
accepted or regarded as true, real and credible” [3]. This step reduces the
complexity of the Big Data scenario, by addressing the variety dimension.

#### Step 3: Transformation

This step transforms data into a unified model, which depends on the goal of
the process, by means of data reduction and projection techniques. It usually
involves ETL (Extract, Transform, Load) techniques typical of Big Data
Integration scenarios [11]. Around 70% of the effort of the overall process is
spent on Data Selection, Pre-processing and Transformation, to provide input
data in the amount, structure and format that suit each Data Mining task
perfectly [12].

A crucial transformation for Societal and Labour Market Intelligence consists
of extracting the important information – i.e., the skills (hard and soft) in
the job vacancy text. More formally, the transformation function takes a job
vacancy $j_{k}\in J$ as input and extracts a set of skills
$S_{k}=\\{s_{1},...,s_{n}\\}$ where $S_{k}\in P(S)$, the powerset of skills
$S$. Note that the skills set $S$ evolves throughout time as novel skills
arise.

#### Step 4: Data Mining and Machine Learning

This step requires to identify proper AI algorithms (e.g., classification,
prediction, regression, clustering), searching for patterns of interest. In
our context, it usually involves text classification algorithms to map
vacancies/resumes into one of several predefined classes in a taxonomy or
ontology of job profiles and skills, such as ISCO, ESCO and/or Competent. More
formally, categorization assigns a Boolean value to each pair
$(j_{k},c_{i})\in JxC$ – where J are job vacancies and C the predefined
categories (of a given taxonomy) – true if $j_{k}$ belongs to $c_{i}$ and
false otherwise. This categorization task can be solved through machine
learning: let $J={J_{1},...,J_{n}}$ be a set of job vacancies and the
classification of $J$ under the taxonomy consists of $|O|$ independent
problems of classifying each job vacancy $J_{i}\in J$ under a given taxonomy
occupation code $o_{i}$ for $i=1...|O|$. Then, a classifier is a function
$\phi:JxO\xrightarrow{}\\{0,1\\}$.

#### Step 5: Interpretation and Feedback

Finally, this step employs visual paradigms to represent the resulting
knowledge, according to the ultimate goal. In our context, this step concerns
the users’ ability to understand the data. For example, public administrations
might be interested in identifying the most requested occupations in a cross-
border area; companies might focus on monitoring skills trend and novel,
arising skills and occupations, so that they can design training paths.

##  IV A DataOps Processing Solution for Skills Extraction and Matching

From the steps described in the previous section, we focused our prototype on
the critical task of skills extraction and matching. This entails a DataOps
pipeline combining Big Data and Machine Learning models to process text in
vacancies and resumes, and then extract the sentences containing skills.
Figure 2 shows the Model Training and Prediction pipelines, detailed below.

#### Data Pre-processing

Input data must be provided in the amount, structure and format that suit each
Data Mining task perfectly. Low quality data will lead to low quality results:
garbage in, garbage out [13]. Data Pre-processing may feature several tasks,
typically including Data Cleaning, Transformation, Integration, Normalization,
Missing Data Imputation and Noise Identification [12].

Given the characteristics of our input (vacancies text), several text
preprocessing steps are necessary, mainly comprising (but not limited to):

  * •

Distinguish end-of-sentence periods ( . ) from others (e.g. abbreviations,
acronyms, websites, etc.).

  * •

Distinguish end-of-sentence line breaks (\n) from spurious ones.

  * •

Trim leading/trailing spacing/tabbing.

  * •

Identify and replace special characters used as bullet points (such as
hyphens, dashes or even ’o’).

  * •

Filter stopwords with little lexical meaning, such as pronouns and articles.

The output of this step is a list of processable sentences, ready to be used
by the Machine Learning algorithm.

Figure 2: Model training and prediction pipelines for skills extraction and
matching.

#### Sentences Annotation for Learning

Each sentence in the training set must be annotated by domain experts: the
ones that contain a skill are annotated with a 1, otherwise with a 0.

In the case of ambiguous sentences, we created a codebook with help of the
domain experts as an aid to identify and correctly annotate the sentences,
with the following, previously agreed definitions:

Hard skill: Is a professional skill, activity or knowledge. These are related
to functions, processes and roles in organizations and are needed in practice
to successfully complete a specific task. For example:

Your work consists of checking and controlling cleaning activities of the
chambermaids, opening and / or closing your service, communication with the
reception and / or guests and light administrative work.

An example of vacancy sentences without hardskill:

It is an international company that provides management, advisory and
engineering services on socially relevant issues related to mobility,
environment & space, buildings and water.

Soft skill: Is a personal, emotional, social or intellectual skill, also
called behavioral skill or competence. Soft skills are often seen as good-to-
have in addition to the hard skills. For example:

You are able to maintain customer relationships, identify opportunities and
utilize and expand your network;

#### Model Training

First, the model is trained on unlabeled data over different pre-training
tasks. This reduces the need for many heavily-engineered task-specific
architectures. Pre-trained models can be fine-tuned with just one additional
output layer to create state-of-the-art models for a wide range of tasks, such
as question answering and language inference, without substantial task-
specific architecture modifications [14].

#### Supervised Training (fine-tunning)

Then a training set is given to form a description that can be used to predict
unseen examples. This training set (’train’ data) is typically a set of
labeled instances [12]. This is pre-processed (as described above), tokenized
and converted into the format that the model understands – typically in the
form of csv or tsv files, containing the columns with the annotations, the
vacancies/resumes and skills ids for traceability, and the actual sentence
text.

At this point the model can be actually trained for the needed purposes. Early
stopping with a configurable number of epochs is recommended, as the training
theoretically should finish when the model achieves the optimal state under
the setup conditions, avoiding overfitting.

Afterwards, the accuracy can be verified using the ’dev’ validation dataset, a
labeled set not directly used for training. Note that currently a small set of
manually annotated data is used to train the model. Increasing the training
set even by 500-600 examples can boost predictions accuracy.

All in all, the output of this process is the trained model which can be used
for further predictions. Note that, compared to pre-training, fine-tuning is
relatively inexpensive. All of the results in the following section can be
replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU,
starting from the exact same pre-trained model.

#### Predictions with the Pre-trained ML Model

At this point the model is ready to make predictions, according to the
pipeline depicted in Figure 2. Recall that the goal of the pipeline is to
classify (predict) sentences describing skills from a vacancy text.

Pre-processing is equal to the one performed for training. Then the sentences
extracted from the processed vacancies/resumes text (in the proper tsv/csv
format) are given as input to the model, pre-trained and fine-tuned during the
training pipeline. The output contains the predictions for each sample, with
the original text, ids and class probabilities (whether the sentence contains
a skill or not).

##  V Validation: The Case on the cross-border Dutch-Flemish labor market

###  V-A Context

The case-study in question exemplifies the applicability and impact of the
proposed DataOps pipeline, featuring the economy and the labor market in Dutch
border regions which are malfunctioning w.r.t. other regions in the
Netherlands. At the same time, there are opportunities on the other side of
the border (i.e., Belgium, Germany) that are not or insufficiently utilized.
Many factors can impede cross-border work: differences in language and
culture, institutional and administrative differences, lack of cross-border
infrastructure, psychological factors, economic differences and information
delays (source: CPB666Netherlands Bureau for Economic Policy Analysis –
https://www.cpb.nl/en).

Werkinzicht777An EU Interreg project – https://werkinzicht.eu/, the umbrella
project for this work, is working to provide a clear picture of cross-border
VL-NL labor market information for job seekers, employers, educational
institutions, governments and intermediaries. Cross-border job boards are
available from different organizations. For example, UWV and VDAB – Dutch and
Flemish employment agencies respectively – publish vacancies from both sides
of the border.

To date, the baseline attained in previous projects – e.g., during the ESCO
skills mapping pilot [6] – shows that the automated transformation using
mapping tables introduces a high level of noise (between 81/92%), meaning only
one/two out of every ten skills represented a correctly transferred skill.
Besides, the manual effort is cumbersome: a sample of 610 different skills
took, on average 60 hours [6]. This can be certainly improved through our
semi-supervised approach.

###  V-B Execution and Results

Our prototype pipelines focus on skills extraction, as shown in Figure 2. We
used three alternatives for the underlying Machine Learning Model. First, a
Simple Neural Network888Source and documentation: https://tinyurl.com/y86rerbw
as baseline. Then, two alternative deployments of the best model to date for
this task: BERT (Bidirectional Encoder Representations from Transformers)
[14], namely: BERT in the
Cloud999Sourceanddocumentationat:https://tinyurl.com/ybtygt52, running on
Google Cloud with a single TPU device with 4 TPU chips and 8 cores; and BERT
Local deployment101010Source code and documentation at:
https://tinyurl.com/ycks8sp4 running on on-premise hardware.

The input dataset was provided by the Dutch employment agency (UWV) –
consisting of millions of vacancies from year 2014 onwards. For the training
and validation activities, the domain experts curated a total of 10K vacancies
that became the gold-standard. After pre-processing it we obtained a total of
300K sentences.

#### Annotation

Six domain experts from UWV annotated a total of 3K sentences – around 500
sentences per person. Each sentence is annotated with a ’1’ if contains a
skill, and as ’0’ if it does not. The annotation process was aided by the
Codebook111111https://tinyurl.com/y8k34xqw (in dutch) discussed in Section
III, describing what is intended as a Skill, and how to annotate the dataset.
We performed two rounds of annotations plus feedback, and also cross-checked
the annotations made by the experts with the ones made independently by three
researchers (authors of the present paper). We calculated the Inter-rater
reliability assessment for the annotation process: the annotations by the two
groups (i.e., the domain experts and the researchers) were then compared in
order to appraise each others annotations and comments, and to reach unanimity
on the skills definition and identification. The final result for the widely
known K$\alpha$ [15] coefficient that measures the agreement amounted to 0.86,
higher than the typical reference score of 0.80 (i.e., 80% agreement).

#### Model Training

We trained the models according to the steps discussed in Section IV. The
baseline pre-trained model used both for BERT Cloud and BERT Local options is
the BERT Multilingual Cased Model121212https://github.com/google-
research/bert/blob/master/multilingual.md. From the 3K annotated sentences we
adopted a 75/25 splitting for training (train set) and validation (dev set)
respectively.

#### Results

The accuracy results are shown in Figure 3. The best performing model in this
case is BERT Cloud, with an accuracy of 0.816 after 210 training steps
(epochs), followed by BERT Local (0.8) and Simple-NN (0.78). All three models
show high and competitive accuracy, although it could be improved by means of
training with more annotated sentences, fine-tunning hyperparameters, etc., as
future work.

Finally, Table I shows the confusion matrix for the best performing model –
i.e., BERT Cloud. Note the high recall (0.94) of annotated skills correctly
predicted. Moreover, as stated by the experts from UWV, the worst case
scenario are false negatives, i.e., skills that are incorrectly classified as
not skills. The model has a very low false negative rate of 0.05.

Figure 3: Accuracy results (more is better) after two-steps training for the three evaluated models: BERT Cloud, BERT Local and Simple-NN TABLE I: Confusion Matrix for Prediction Results using BERT Cloud |  | Annotation (Actual Class)
---|---|---
|  | Skill | Not Skill
Prediction | Predicted Skill | 0.85 (TP) | 0.30 (FP)
Predicted Not Skill | 0.05 (FN) | 0.79 (TN)
Accuracy | 0.816
Recall | 0.945

##  VI Discussion and Limitations

RQ1. How to model a DataOps Pipeline to enable Societal and Labour Market
Intelligence in the context of skills extraction and matching?

To answer RQ1 on the DataOps pipeline for Labour Market Intelligence, Section
III presented our design to move from from big data to knowledge, providing
societal and labour administrators with Big Data analytics and business
intelligence. This allows employment agencies to have evidence of the
competitive advantage that the automated analysis of Web job vacancy enables
with respect to e.g., the classical survey-based analyses, in terms of (i)
near-real-time labor market information; (ii) reduced time-to-market (iii)
fine-grained territorial analysis, from countries down to municipalities [3].

To answer RQ2 regarding the job vacancies and resumes analysis and matching,
Section LABEL:analytic illustrated shown how to scale the identification of
the most requested skills and then matching against the most representative
ontologies such as ISCO/ESCO and cross-border ones as Competent.

RQ2. How to extract and match the skills (from vacancies and resumes) in an
effective and efficient way?

To answer RQ3 regarding the skills extraction problem, we instantiated our
pipeline within the prototype, featuring state-of-the-art machine learning
models with hundreds of thousands vacancies stemming from the Dutch-Flemish
labour market. In terms of accuracy, the best performing model is BERT Cloud.
It also features the fastest execution time, as it leverages on-demand TPU
(TensorFlow Processing Units) infrastructure from Google Cloud, resulting in
speedups of up to 10x for training and predictions – from hours/days to
minutes. That should be taken into account when deploying this prototype in
production, given the scalability and performance issues that may arise.

###  VI-A Limitations

A domain-specific limitation is that our models have been trained on UWV data
and encompasses Dutch vacancies and resumes. This can bring along cross-border
applicability issues as e.g., Flemish and Dutch languages may differ. However,
the baseline BERT multilingual models have been used in different contexts
(e.g, training in one language and predicting in another) with different
languages and outstanding results [16]. Another limitation is that the amount
of annotated data is still small, and involving more experts in annotating
data for training (both in Dutch and Flemish) is mandatory – although the
results already show high accuracy.

Another concern to be addressed is the data privacy and governance: data
should travel back and fort to/from the cloud during the training and
prediction processes. This may pose an issue according to GDPR regulations
given the sensitivity of the data. As the Werkinzicht project is subject to
such regulations, the local solutions may overcome the cloud ones because of
privacy and governance, and in spite of performance and scalability.

##  VII Conclusions

This article introduces and explores the design for a DataOps intelligence
analytics platform to support a more sustainable labour market. The proposed
platform falls into the architecture landscape of what we defined as societal
intelligence, that is, the ability of society to become reflective, using big
open data analytics to instrument that reflection.

Our proposed design was partially validated against a real-life case-study
from the Dutch borderlands labour acceleration initiative. Moreover, we
implemented a prototype for the key tasks of skills extraction from vacancies
and resumes, checked against real data from the Dutch employment agency UWV.
Results not only show that our appraoch is implementable, but also that our
prototype achieves high results in terms of accuracy and recall, being
actually usable implementable in practice.

Meantime, we observed that several technical challenges exist in our proposed
design and prototype, that deserve further attention. Conversely, we also
highlighted the potential impact behind our proposed design and how it may
actually instrument societal intelligence in the Netherlands - an impact that
can potentially increase the NL national wealth by up to 5% (projected).

In the future we plan to: (a) further refine the social, organizational, and
technical requirements behind the proposed design; (b) gather hard data over
the data-areas present in our case-study to refine and bring to production our
proof-of-concept implementation of the design proposal; (c) Integrate our
prototype in the overall pipeline for skills matching against existing
ontologies, and (d) Refine such ontologies to reflect the highly dynamic
labour market.

## References

  * [1] E. Hoffmann, “International statistical comparisons of occupational and social structures,” in _Advances in cross-national comparison_. Springer, 2003, pp. 137–158.
  * [2] R. Boselli, M. Cesarini, S. Marrara, F. Mercorio, M. Mezzanzanica, G. Pasi, and M. Viviani, “Wolmis: a labor market intelligence system for classifying web job vacancies,” _Journal of Intelligent Information Systems_ , vol. 51, no. 3, pp. 477–502, 2018.
  * [3] M. Mezzanzanica and F. Mercorio, “Big data for labour market intelligence: An introductory guide,” _Eur. Training Found., Torino, Italy, Tech. Rep_ , 2019\.
  * [4] UK Commission for Employment and Skills, “The importance of labour market intelligence.” UK Government, Tech. Rep., 2015, https://goo.gl/TtRwvS.
  * [5] ILO International Labour Office, “Isco-08 international standard classification of occupations.” International Labour Office, Tech. Rep., 2012, https://www.ilo.org/wcmsp5/groups/public/---dgreports/---dcomm/---publ/documents/publication/wcms_172572.pdf.
  * [6] ESCO European Commission on Skills, Competences, Qualifications and Occupations, “Final report on the esco mapping pilot.” European Commission, Tech. Rep., 2016, https://tinyurl.com/yczlrhs2.
  * [7] J. Ereth, “Dataops-towards a definition.” _Proceedings of LWDA (Lernen, Wissen, Daten, Analysen) Conference_ , pp. 104–112, 2018.
  * [8] A. Palmer, “From devops to dataops,” Tamr Inc, 2015, https://www.tamr.com/blog/from-devops-to-dataops-by-andy-palmer/.
  * [9] J. Euston, “What is dataops?: Platform for the machine learning age,” Jul 2018\. [Online]. Available: https://www.nexla.com/define-dataops
  * [10] U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth, “The kdd process for extracting useful knowledge from volumes of data,” _Communications of the ACM_ , vol. 39, no. 11, pp. 27–34, 1996.
  * [11] P. Vassiliadis, A. Simitsis, and S. Skiadopoulos, “Conceptual modeling for etl processes,” in _Proceedings of the 5th ACM international workshop on Data Warehousing and OLAP_ , 2002, pp. 14–21.
  * [12] F. H. a. Salvador García, Julián Luengo, _Data Preprocessing in Data Mining_ , 1st ed., ser. Intelligent Systems Reference Library 72. Springer International Publishing, 2015. [Online]. Available: http://gen.lib.rus.ec/book/index.php?md5=F91A1699CE0E23DE40B59A8F0F8797EC
  * [13] I. Mierswa, M. Wurst, R. Klinkenberg, M. Scholz, and T. Euler, “Yale: Rapid prototyping for complex data mining tasks,” in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2006, pp. 935–940.
  * [14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [15] K. Krippendorff, _Content analysis: An introduction to its methodology_. Sage publications, 2018.
  * [16] T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual bert?” _arXiv preprint arXiv:1906.01502_ , 2019.