# Efficient Attention Branch Network with Combined Loss Function for Automatic
Speaker Verification Spoof Detection

###### Abstract

Many endeavors have sought to develop countermeasure techniques as
enhancements on Automatic Speaker Verification (ASV) systems, in order to make
them more robust against spoof attacks. As evidenced by the latest ASVspoof
2019 countermeasure challenge, models currently deployed for the task of ASV
are, at their best, devoid of suitable degrees of generalization to unseen
attacks. Upon further investigation of the proposed methods, it appears that a
broader three-tiered view of the proposed systems; comprised of the
classifier, feature extraction phase, and model loss function, may to some
extent lessen the problem. Accordingly, the present study proposes the
Efficient Attention Branch Network (EABN) modular architecture with a combined
loss function to address the generalization problem. The EABN architecture is
based on attention and perception branches; the purpose of the attention
branch—also interpretable from a human’s point of view—is to produce an
attention mask meant to improve classification performance. The perception
branch, on the other hand, is used for the primary purpose of the problem at
hand, that is, spoof detection. The new EfficientNet-A0 architecture was
employed for the perception branch, with nearly ten times fewer parameters and
approximately seven times fewer floating-point operations than the top
performing SE-Res2Net50 network. The final evaluation results on ASVspoof 2019
dataset suggest an EER = 0.86% and t-DCF = 0.0239 in the Physical Access (PA)
scenario using the logPowSpec input feature, the EfficientNet-A0 for the
perception branch, and the combined loss function. Furthermore, using the LFCC
input feature, the SE-Res2Net50 for the perception branch, and the combined
loss function, the proposed model performed at figures of EER = 1.89% and
t-DCF = 0.507 in the Logical Access (LA) scenario, which to the best of our
knowledge, is the best single system ASV spoofing countermeasure.

Index Terms: Automatic speaker verification, Spoof Detection, ASVspoof,
Efficient Attention Branch Network, Combined Loss Function, EfficientNet-A0

##  1 Introduction

Remote authentication has excited great interests in various academic circles
and otherwise, given the increasing reliance on online applications as well as
the onset of certain conditions such as the Covid-19 pandemic. Such
circumstances call for an easy-to-use, accurate, and efficient authentication
system. Along this thread, Automatic Speaker Verification (ASV) system and
other biometric systems such as face recognition, electronic signatures, iris-
based, and hybrid methods have been proposed as a means to satisfy user needs
[1, 2]. Nevertheless, virtually all of these systems are vulnerable to spoof
attacks (i.e., spoofable). A system based on face recognition for example, may
be spoofed by simply displaying a person’s image (photo) to the system [3].
Likewise, a fingerprint system can be spoofed by copying a fingerprint. In
particular, ASV systems are also vulnerable in the face of four types of
attacks, counting on recording and replaying the voice of the authorized
person (replay attack), text-to-speech systems that are trained with the voice
of the targeted person, voice conversion systems, and speaker imitation [4].
The threats facing ASV systems in terms of spoof attacks are potentially high
and may amass to serious implications [5]. In consequence, since 2015,
ASVspoof challenges were held for research communities worldwide to try and
enhance ASV systems so as to make them robust against spoofing attacks.

A total of three ASVspoof challenges have thus far been held, with the first
instance in 2015, covering only speech synthesis and voice conversion (also
called logical access scenario) attacks [6]. A variety of methods and systems
were proposed and implemented by ASV organizers to produce spoof samples,
exciting the interest of many researchers intrigued by both the challenge and
the dataset provided therein. The second ASVspoof challenge held in 2017,
focused more on the replay attack (also called physical access scenario) [7,
8]. In order to be able to test the performance of countermeasure systems in
real conditions, the organizers produced the dataset in different
environmental conditions and using different devices. Further comprehensive
conditions were investigated in 2019 to account for all three attacks
considered in previous challenges [9], ushering in the development of an
extensive dataset using state-of-the-art voice conversion and speech synthesis
systems. Spoofing samples in this challenge were more realistic and
challenging in view of the improvements made to the spoof systems in previous
years. For replay attacks, in particular, samples were produced with greater
degrees of control, and a tandem detection cost function (t-DCF) metric was
used as the primary metric to assess the efficiency of integrating
countermeasures with ASV systems.

Models proposed to assess the ASVspoof 2019 dataset can be categorized into
two main classes: methods based on extraction and engineering of features and
methods based on classifier architecture. Methods of the first category
incorporate features such as Mel-filter frequency Cepstral coefficients
(MFCC), Inverted Mel-Filter Frequency Cepstral coefficients (IMFFC), Constant
Q Cepstral Coefficients (CQCC), Group Delay (GD) gram, Instantaneous Amplitude
(IA), Instantaneous Frequency (IF), X-vectors, and features from deep learning
models [10, 11, 12, 13, 14, 15, 16, 17]. Some methods also use raw signals to
extract features using methods such as SincNet [18] or Variational Auto
Encoder (VAE) [19]. The second category deals with a variety of classifiers
such as Neural network-based methods including VGG [18], Squeeze-Excitation
(SE), Residual network, Siamese networks [20], and recurrent networks [21], as
well as other traditional GMM-based methods. Certain methods have also used
end-to-end structures for this purpose.

Inquiries made into the 2019 ASVspoof dataset results are suggestive of two
primary drawbacks of the proposed methods. The first points to a lack of
generalization and high error rate against unseen attacks, which is clearly
observed given the difference between errors obtained for the training,
development, and evaluation sets. In addressing this lack of generalization,
numerous studies have tried to improve generalization by means of fusing
several models (ensemble models) [22]. Such fusion and ensemble models and
methods that use deep neural networks have led to considerable increases in
model parameters as well as the necessary floating-point operations (FLOPS).
Under such circumstances, it would be infeasible to use the proposed models in
specific applications. This provides the required grounds for the integration
of simple yet efficient countermeasure techniques with ASV systems to make ASV
robust. Moreover, the proposed body of research fails to provide a detailed
understanding for how models detect spoof attacks or handle generalization
issue. This ambiguity can be interpreted in terms of the incapacity of humans
or rather human-oriented decision making to differentiate between the spoofed
and the bonafide samples detected by the final system. A detailed examination
of this issue can provide further insight into the development of better
system.

The primary purpose of this work is to provide a model for detecting spoofing
attacks on ASV systems. An interpretable attention mask in a new modular
architecture was used for this purpose via the introduction of perception and
attention branches. Furthermore, for the first time in this domain, the
EfficientNet-A0 [23] architecture was employed to achieve a system with low
number of parameters and FLOPS. The proposed architecture along with the newly
combined loss function and masks that provide a more human-oriented
perspective, was used to obtain comparable and, in some cases, top-performing
results in these spoofing attacks.

The following section provides a brief review of relevant studies conducted in
recent years. The proposed countermeasures and the loss function are
introduced in Section 3 and Section 4 calls attention to the general
configuration used for experiments. Section 5 gives the analysis results along
with a summarization of the work. The study is finally concluded in Section 6.

##  2 Related work

This section reviews some of the research carried out on spoof attack
detection, taking a look on the best-performing methods, as per results
obtained on the ASVspoof 2019 dataset. Similar models and tasks were also
investigated inclusive of new architecture and the application of attention
mechanisms and attitude in the loss function.

Cheng-I Lai et al. proposed a deep model to obtain discriminative features in
both time and frequency domains [24]. The proposed design includes a filter-
based attention mechanism used to improve or ignore commonly extracted
features implemented in the ResNet architecture to classify attended input
maps. The reserved classifier used in their study (Residual Network) consists
of a convolution layer equipped with dilated mechanism instead of a fully
connected layer, which runs as an attentive filtering network; i.e., masks
input features. The obtained results were suggestive of the relatively high
performance of the model given the use of an attention mechanism to produce
attention masks as well as an appropriate classifier.

X. Li et al. attempted to use the Res2Net architecture, which has achieved
significant results in various computer vision tasks [25]. They proposed a new
Res2Net architecture by revisioning ResNet blocks to allow for multi-scale
features. In a Res2Net architecture, input feature maps of a block are divided
into several groups of channels with a similar residual structure to the
original ResNet. Using channels, feature map sizes can be different,
increasing the covered area, and thereby yielding features with different
scales. This modification improves system performance and the model’s
generalization against unseen attacks. In addition, using this architecture
could reduce the size or number of model parameters relative to the original
ResNet structure while improving model performance. The obtained results show
that the Res2Net50 model performs better than the ResNet34 and ResNet50 models
in both physical and logical access scenarios. They also showed that
integrating the block with Squeeze-and-excitation (SE), which produces SE-
Res2Net blocks, leads to better performance. Figure 1 illustrates the
architecture and structure of these blocks. Significant results were also
obtained in both scenarios for the proposed SE-Res2Net50 network based on SE-
Res2Net blocks and CQT feature. The network proposed in this work has nearly
0.9 million parameters, which is relatively small compared to other
architectures. However, the main drawback to the model is the high number of
FLOPS, which leads to increased runtime in the inference phase due to the
multiplicity of blocks and the structure of SE-Res2Net.

(a) ResNet

(b) Res2Net

(c) SE-Res2Net

Figure 1: ResNet, Res2Net, and SE-Res2Net blocks [25].

Zhang et al. focused on logical attacks in their work [26], explaining the
lack of model generalization against unseen attacks as caused by the
formulation of the spoof detection problem as a binary classification. The
difficulty with using a binary classifier can be interpreted in terms of the
distribution of training and test data for spoof and bonafide samples as not
being the same. More specifically, samples in the test set generated by new
systems or conditions not found in training data cause differences in
distribution; which, however, is not the case for bonafide samples. The
problem was, therefore, redefined as a one-class classification problem, where
the distribution of a target class for a specific problem should be the same
in both training and test datasets, irrespective of whether other classes have
similar distributions or not. In such cases, the primary objective is to
obtain the bonafide distribution and define a rigid decision boundary around
it so that unseen samples from other classes cannot cross that decision
boundary. To this aim, a one-class softmax loss function was incorporated for
learning a feature space that can map bonafide samples in a dense space, while
maintaining a good margin with spoofing samples. Finally, by means of the
ResNet-18 network and the LFCC features, the authors succeeded in attaining
top-performing results for logical access attacks.

Figure 2: Proposed Efficient Attention Branch Network architecture.

##  3 Proposed Model

###  3.1 Network Architecture

The overall architecture of the proposed network was designed with three main
objectives in mind; that the architecture be small enough to explicate an
appropriate number of parameters, while maintaining an acceptable runtime in
order to achieve satisfactory performance in most ASV applications; the
architecture was designed to be interpretable by humans. To put differently,
the architecture was required to somehow express what discriminates bonafide
speech from speech made in a spoof attack as a means to improve systems in the
future; lastly, the model was configured to emulate comparable performance to
relevant classifiers used for this purpose.

To achieve all these goals, the Efficient Attention Branch Network (EABN) was
proposed in this study. The intended framework adopts a well-performed
Attention Branch Network [27] in computer vision as the main idea for the EABN
architecture. As shown in Figure 2, this network consists of two branches of
attention and perception. The attention branch seeks to improve the
performance of the perception branch by means of producing an attention mask,
which is then applied to make the discriminative parts of the input feature
map more. In addition to improving the performance of the perception branch,
masks produced by the attention branch are also interpretable from a human
point of view; as these masks are used for dummy classification tasks. The
primary work load is performed in the perceptual branch, where the probability
output of each class is produced.

###  3.2 Attention Branch

The attention branch itself comprises of two main parts, as shown in Figure 3.
As can be observed, the input feature map is initially fed into the attention
branch, which uses four consecutive Basic Blocks to extract the appropriate
features and to convert the input features to 16-feature maps. The blocks
consist of two convolution layers with 3×3 kernels, which are then linked to
the batch normalization layer. In addition to feature extraction, the first
convolution layer also increases the feature map size, while the second
convolution layer exclusively handles the feature extraction process. The
obtained feature maps are eventually transformed from a 16-size map to a
single feature using a convolution layer with a 1×1 kernel, which then goes
through a softmax layer to yield the final output attention mask.

The other branch produces a human-interpretable attention mask. This is
carried out by using a convolution operation to transform the 16 feature maps
into maps the same size as the number of classes for the problem—which in this
study includes the bonafide and spoof classes. Then, using a Global average
Pooling layer, these two feature maps are converted to a 2×1 tensor. Finally,
by applying softmax, the probability of a feature map belonging to each class
is obtained. These probabilities are later used in the optimization process
for the proposed combined loss function. Through the process of optimization,
feature maps are generated so that in addition to helping the perception
branch, they can also be used for classification and be made interpretable
from a human perspective.

Figure 3: Proposed architecture for Attention branch.

###  3.3 Perception Branch

The perception branch can constitute virtually any classifier. However, as the
primary objectives of this study call for low number of parameters, low
runtime, and good performance in network design, the EfficientNet
architecture—commonly introduced as a high performing model in image
classification tasks and speech processing tasks such as speech recognition
and keyword spotting—was employed. The fundamental architecture of the
EfficientNet family is called EfficientNet-B0, which has about 4 million
parameters. This number of parameters is not suitable for the target
applications of this study. Alternatively, the approach introduced in the
EfficientNet-Absolute Zero work [23], which applies the reverse of the
compound scaling method, was used. The scaling method ($S$) is designed to
shrink a base model ($M$) by decreasing the depth ($\alpha$), width ($\beta$),
and resolution of the input image ($\gamma$), simultaneously. A formulation of
this method is given below as an optimization problem, in which the goal is to
satisfy the intended conditions so that the final model has the best
performance.

$\displaystyle\begin{aligned} max_{d,r,w}\ Accuracy(S(M,\ d,\ r,\
w))\end{aligned}$ s.t.
$\displaystyle\qquad\qquad\frac{1}{20}\leq\alpha\cdot\beta^{2}\cdot\gamma^{2}\leq\frac{1}{16}$
$\displaystyle\begin{aligned} \qquad\qquad&0.2\leq\alpha,\ \beta\leq 0.6,\
\gamma=2\\\ \end{aligned}$

The two parameters α and β are set by applying a grid search on intervals
[$0.2-0.6]$ with steps of 0.005. Eventually, 19 models were evaluated with a
small subset of samples, with parameters $\alpha$ and $\beta$ set at values
0.2 and 0.25, respectively. $\gamma$ was also set at $\approx$ 2, given the
input image size (513×400) and EfficieNet-B0 input-size of 256×256. Figure 4
illustrates the final model obtained for the perception branch with 95,000
parameters. The input to this branch is $m(x_{i})$, where $x_{i}$ is the input
image for the $i^{th}$ sample and is calculated from the following equation:

$m\left(x_{i}\right)=\left(1+g\left(x_{i}\right)\right)\times x_{i}$

where $g(x_{i})$ is the attention mask produced for the $i^{th}$ sample by
attention branch. The output of this network is a vector of length 256, which
represents the embedded vector of the input image and is applied for two
scenarios: the first uses the vector along with a fully connected layer and
the softmax layer to yield probabilities for each individual sample; the
second scenario uses the vector as input to a loss function. Thus, samples are
embedded in a 256-dimensional space in the most distinctive way.

Figure 4: Proposed architecture for perception branch operating via the
reverse compound scaling method.

###  3.4 Loss function

To train model parameters, a combined loss function was used to account for
all study objectives. To train an attention branch capable of producing
interpretable masks, the $AB_{output}$ was used as input to a weighted Cross-
Entropy (CE) loss function. It should be noted that by introducing the
proposed loss function with coefficient $\lambda_{AB}$, values in equation 3
are altered. Proceeding forward, the Triplet Center Loss (TCL) function was
used to train the embedding vectors. TCL works in the same way as the triplet
loss function, except that it no longer needs to mine triplets for training,
and this difference makes the training process faster and more stable. This
loss function considers center points for each class in the problem, which are
initially assigned random values. The loss function then approaches to make it
so that samples of one class are close to the center of their class and away
from the nearest center of other classes. In other words, each sample tries to
be closer to the center of its class and away from the remaining centers. The
two centers used in this study to represent spoof and bonafide samples were
$C_{spoof}$ and $C_{bonafide}$, respectively. The goal here was to ensure that
bonafide samples be close to the center of their respective target class,
$C_{bonafide}$, and away from $C_{spoof}$. As a result, samples of a specific
class in a dense space are closer to each other; representing feature vectors
embedded for each sample in the desired space. TCL can be obtained for the
$x_{i}$ sample as follows:

$L_{\text{focal
}}\left(p_{t}\right)=-\alpha_{t}\left(1-p_{t}\right)^{0.005}\log\left(p_{t}\right)$
(1) $L_{PB}\left(x_{i}\right)=L_{tc}\left(x_{i}\right)+\lambda_{\text{focal
}}L_{\text{focal }}\left(x_{i}\right)$ (2)
$L_{\text{total}}=L_{PB}+\lambda_{AB}L_{\mathrm{AB}}$ (3)

Table 1: Summary of the ASVspoof2019 dataset. Partition | PA | LA
---|---|---
# Spoof | # Bonafide | # Spoof | # Bonafide
Train | 48600 | 5400 | 22800 | 2580
Dev | 24300 | 5400 | 22296 | 2548
Eval | 116640 | 18090 | 63882 | 7355
Table 2: t-DCF hyperparameters value. Attack Type | Probabilities | ASV costs | Countermeasure costs
---|---|---|---
$\pi_{\mathrm{tar}}$ | $\pi_{\mathrm{non}}$ | $\pi_{\mathrm{spoof}}$ | $C_{\mathrm{fa}}^{\mathrm{asv}}$ | $C_{\mathrm{miss}}^{\mathrm{asv}}$ | $C_{\mathrm{fa}}^{\mathrm{cm}}$ | $C_{\mathrm{miss}}^{\mathrm{cm}}$
PA | 0.9405 | 0.0095 | 0.05 | 10 | 1 | 10 | 1
LA | 0.9405 | 0.0095 | 0.05 | 10 | 1 | 10 | 1

##  4 Experimental configuration

###  4.1 Dataset and evaluation metrics

The proposed method was evaluated using the ASVspoof 2019 dataset, which
includes two scenarios: physical access (PA) and logical access (LA). Details
of this dataset are shown in Table 1. Furthermore, considering that one of the
objectives of this research is the simultaneous use of countermeasure and ASV
system, the tandem-detection cost function (t-DCF) and the equal error rate
(EER) metrics were used. This metric was introduced as the primary evaluation
metric of the 2019 challenge, which is calculated as:

$\mathrm{t}-\mathrm{DCF}(s)=C_{1}P_{\mathrm{miss}}^{\mathrm{cm}}(s)+C_{2}P_{\mathrm{fa}}^{\mathrm{cm}}(s)$

where $P_{\mathrm{fa}}^{\mathrm{cm}}(s)$ and $P_{\mathrm{miss}}^{\mathrm{cm}}$
are the false acceptance error rate and the false rejection error rate of the
countermeasure, respectively. Considering the threshold, s, values for the two
error rates can be obtained as follows:

$\displaystyle P_{\text{miss }}^{\mathrm{cm}}(s)$
$\displaystyle=\frac{\\#\\{\text{ bona fide trials with CM score }\leq
s\\}}{\\#\\{\text{ Total bona fide trials }\\}}$ $\displaystyle
P_{\mathrm{fa}}^{\mathrm{cm}}(s)$ $\displaystyle=\frac{\\#\\{\text{ spoof
trials with CM score }>s\\}}{\\#\\{\text{ Total spoof trials }\\}}$

The two constants $C_{1}$ and $C_{2}$ represent the predefined cost for the
errors, which are determined based on prior probabilities as shown below:

$\left\\{\begin{array}[]{l}C_{1}=\pi_{\operatorname{tar}}\left(C_{\mathrm{miss}}^{\mathrm{cm}}-C_{\mathrm{miss}}^{\mathrm{asv}}P_{\mathrm{miss}}^{\mathrm{asv}}\right)-\pi_{\mathrm{non}}C_{\mathrm{fa}}^{\mathrm{asv}}P_{\mathrm{fa}}^{\mathrm{asv}}\\\
C_{2}=C_{\mathrm{fa}}^{\mathrm{cm}}\pi_{\mathrm{spoof}}\left(1-P_{\mathrm{miss},\mathrm{spoof}}^{\mathrm{asv}}\right)\end{array}\right.$

Here, $C_{\mathrm{miss}}^{\mathrm{asv}}$ represents the cost incurred by the
error of the ASV system for the false rejection error rate of the genuine
person, and $C_{\mathrm{fa}}^{\mathrm{asv}}$ represents the false acceptance
error rate when ASV authorizes the wrong person. Each countermeasure error
also corresponds to two costs; $C_{\mathrm{miss}}^{\mathrm{cm}}$, which
indicates the cost in recognizing a bonafide sample as a spoof, and
$C_{\mathrm{fa}}^{\mathrm{cm}}$, which indicates a mistake in accepting a
sample produced by a spoof system as bonafide. In addition, the probability of
occurrence of any class of genuine ($\pi_{\mathrm{tar}}$), non-target or
imposter ($\pi_{\mathrm{non}}$) and spoof attack ($\pi_{\mathrm{spoof}}$) are
also considered with the condition
$\pi_{\mathrm{tar}}+\pi_{\mathrm{non}}+\pi_{\mathrm{spoof}}=1$. Cost and
probability values are calculated as in Table 2.

###  4.2 Feature extraction and engineering

Based on past researches and works, a single acoustic feature was considered
for each of the attacks. For the PA scenario, we used the logarithm power of
the spectrogram (logPowSpec) with 25 ms frames, 10ms step size with 1024
samples (with zero padding applied if needed), using Hamming window. All the
samples are first transformed into 4s voice segments. To do this, samples that
are less than 4 seconds are repeated to achieve a 4s segment. Longer samples
are also divided into 4s segments with no overlap, and each segment is
considered an individual utterance. The final input form consists of a
spectrogram with 513×400 dimensions. For the LA scenario, the LFCC feature was
extracted according to the procedure used in the base model presented in the
ASVspoof 2019 challenge. Here, 20ms frames with 512 Fourier transform points
and 20 filters were used along with their first and second derivatives.
Finally, a two-dimensional tensor with dimensions of 60×400 was obtained.

As a further step, specAug [28] techniques were applied for better training
and generalization. This method has worked well for other speech tasks, such
as speaker verification, speech recognition, and keyword potting [23]. The
method was implemented by applying zero masks on the time and frequency axis
for each training sample with a probability of 0.25. The size of this band is
randomly between 20 and 80 frames on the time axis. In the frequency axis; the
size of this band is selected randomly between 5 and 20 for the LFCC and
between 25 and 100 for the logPowSpec. As a result, the model is capable of
accounting for all time frames and frequency bins and preventing overfitting
at specific time or frequency points.

###  4.3 Perception branch models

In addition to the proposed EfficientNet-A0 architecture, a SE-ResNet50
architecture was also used, which achieved significant results. The models
were then compared in terms of both efficiency and performance, and the EABN
modularity idea was evaluated accordingly.

###  4.4 Training procedure

The final results obtained from experiments on small subsets of the ASVspoof
2019 dataset yielded values of 0.1, 0.005, and 32 for $\lambda_{AB}$,
$\lambda_{focal}$ and $m$, respectively. To optimize the loss function with
assigned values, configurations for the SE-ResNet50 architecture were adopted.
In the case of Adam optimization, $\beta_{1}$, $\beta_{2}$, and learning rate
were obtained at $0.9$, $0.98$, and $10^{-9}$, respectively. The learning rate
initially drops linearly for the first 1000 steps and then decreases in
proportion to the inverse of the square root of the number of steps. All
models were trained with 40 epochs and the model with the lowest EER on the
development set of the dataset was selected as the optimal choice. Batch-sizes
were set at 64 and 128 when using EfficienNet-A0 as the perception branch
module with LFCC and logPowSpec, respectively. Due to the relatively greater
number of parameters for the SE-Res2Net50 model compared to EfficientNet-A0, a
batch-size of 8 was used for LFCC and logPowSpec features. The models were
implemented on a GTX-1080ti GPU on Linux OS. The source code of our
implementations based on Python and Pytorch is publicly
available111https://github.com/AmirmohammadRostami/ASV-anti-spoofing-with-
EABN.

##  5 Results

###  5.1 Perception branch’s models evaluation

This section evaluates the overall architectural EABN and the EfficientNet-A0
network as a classifier for spoof detection. To investigate EABN performance,
the EfficientNet-A0 and SE-ResNet50 architectures were used for the perception
branch, which have the lowest EER as a single model to the best of our
knowled.ge. The results for both attacks are shown in Table 3. In the PA
scenario, EfficientNet-A0 showed better performance than SE-ResNet50 and
nearly ten times fewer parameters and seven times fewer FLOPS. On the other
hand, the SE-ResNet50 model performs better when using LFCC feature for the LA
scenario. This can be explained in terms of the enhanced performance of the
EfficientNet-A0 model in extracting features from the spectrogram. On the
other hand, the SE-ResNet50 model works best when the feature input is
processed, and the EfficientNet-A0 model converges very quickly or suffers
from overfitting.

Table 3: Result of models used in perception branch and input features on ASVspoof 2019 evaluation dataset for PA and LA scenarios. K, M, and G represent Kilo, Mega, and Giga, respectively.  # | Perception branch model | Input feature | #Parameters | #Flops | PA | LA
---|---|---|---|---|---|---
EER(%) | t-DCF | EER(%) | t-DCF
1 | EfficientNet-A0 | LFCC | 95k | 198M | - | - | 3.68 | 0.0931
2 | EfficientNet-A0 | LogPowSepc | 1.696G | 0.86 | 0.0239 | - | -
3 | SE-Res2Net50 | LFCC | 964k | 1.519G | - | - | 1.89 | 0.0597
4 | SE-Res2Net50 | LogPowSepc | 12.929G | 0.98 | 0.2769 | - | -

(a)

(b)

Figure 5: Feature embedding visualization of our proposed loss function for
evaluation (a) and training (b) sets of the ASVspoof 2019 LA attack. Features
were reduced to 2-D space using PCA.

###  5.2 Loss function

The proposed combined loss function was used for the first time in this work
to achieve a discriminative vector space to distinguish spoof samples from
bonafide samples. More precisely, the triplet center loss was used to map
input samples to a discriminative space. As shown in Figure 5, the training
samples mapping space is suitable for the classification problem. Examining
test samples that include unseen attacks also demonstrate that the resulting
space is reasonably discriminative. It can therefore be said that the model
shows good generalization for unseen attacks. The best value for margin 32 was
obtained in this study by testing three values of 16, 32, and 64.

###  5.3 Attention masks

One of the main concerns about the proposed architecture was to obtain
attention masks that could be interpreted from a human point of view. This was
investigated for the LFCC feature, with the averaged masks generated for all
samples in the evaluation set shown in Figure 6. As for the logPowSpec mask,
few samples from the evaluation set of the PA attack are shown in Figure 7.
The raw input features and results of the applied mask on the input feature,
which is input for the perception branch, are also shown in this figure.
Examining the LFCC feature masks obtained for different attack systems reveals
that higher-frequency domain information is more effective in detecting spoof
patterns. This is consistent with the fact that models using LFCC features
tend to outperform those that work with MFCCs in most studies. As presented by
the findings, resolution values are lower at high frequencies for MFCC than
LFCC. Examining the masks created for physical access attacks also show that
the model emphasizes silent parts of speech. This is in all likelihood to the
fact that the effects of record and play devices on recorded speech are more
considerable during silences, and the countermeasure can more easily detect
attacks accordingly. Convolution operations in the perception branch also tend
to converge faster by blurring and dominating some values at different
frequencies. This can in effect be achieved by decreasing the impacts of
frequencies that show lower capacity to discriminate spoof attacks from
bonafide samples. In this regard, it can be said that paying attention to
silence intervals and reducing the impact of human speech frequencies lead to
better detection of physical access attacks.

Figure 6: Average of produced LFCC attention masks for some spoof attacks in ASVspoof 2019 evaluation. As can be seen, the attention branch tries to dominant high frequencies to detect spoof attacks. Class |  Inp. feature |  Att. mask |  Perc. input
---|---|---|---
A03 |  |  |
A09 |  |  |
A05 |  |  |
A06 |  |  |
B |  |  |
Figure 7: Input (Inp.) feature, produced attention (Att.) mask, and final input feature for perception (Perc.) branch of some samples in the evaluation set for logPowSpec feature (B is bonafide class). Red boxes are parts of input features that the attention branch wants to dominate and are interpretable from a human’s perspective. Table 4: Performance comparison of the proposed systems with known single systems tested on the ASVspoof 2019 PA and LA evaluation set. Models are named base on their input feature, the classification model, and the loss function. Input feature + Classifier + Loss function | PA | LA
---|---|---
EER(%) | t-DCF | EER(%) | t-DCF
(Baseline) CQCC+GMM+EM [9] | 11.04 | 0.2454 | 9.57 | 0.2366
(Baseline) LFCC+GMM+EM [9] | 13.54 | 0.3017 | 8.09 | 0.2116
Spect+ResNet+CE [29] | 3.81 | 0.9940 | 9.68 | 0.2741
MFCC+ResNet+CE [29] | - | - | 9.33 | 0.2042
Spect+ResNet+CE [20] | 1.29 | 0.0360 | 11.75 | 0.2160
Joint-gram+ResNet+CE [10] | 1.23 | 0.0305 | - | -
LFCC+LCNN+A-softmax [30] | 4.60 | 0.1053 | 5.06 | 0.1000
Spect+LCNN+A-softmax [30] | - | - | 4.53 | 0.1028
FG-CQT+LCNN+CE [31] | - | - | 4.07 | 0.1020
Spect+LCGRNN+GKDE-softmax [32] | 1.06 | 0.0222 | 3.77 | 0.0842
Spect+LCGRNN+triplet | 0.92 | 0.0198 | - | -
Fbank&CQT+ResNeWt+CE [33] | 0.52 | 0.0134 | - | -
CQTMGD+ResNeWt+CE [33] | 0.94 | 0.0250 | - | -
Spect+SE-Res2Net50+CE [25] | 0.74 | 0.0207 | 8.73 | 0.2237
LFCC+SE-Res2Net50+CE [25] | 1.46 | 0.434 | 2.87 | 0.0786
CQT+SE-Res2Net50+CE [25] | 0.46 | 0.0116 | 2.50 | 0.0743
Raw signal+SincNet+CE [18] | - | - | 20.11 | 0.3563
logCQT&powSpect+VGG+CE [18] | 2.11 | 0.527 | - | -
LFCC+ResNet18+OCS [26] | - | - | 2.19 | 0.0590
Proposed: LFCC+SE-ResABNet+CombLoss | - | - | 1.89 | 0.0507
Proposed: logPowSpec+EABNet+CombLoss | 0.86 | 0.0239 | - | -

###  5.4 Comparison with other single models

The proposed models have been compared with some of the single models and the
baseline models according to the presented objectives. Some of the top-
performing models used for relevant purposes are shown and compared with the
proposed model in Table 4. For the LA attack, the LFCC+SEResABNet+CombLoss
model achieved an EER=1.89% and t-DCF=0.507, which outperforms the baseline
model LFCC-GMM. The proposed model also improved by approximately 0.98% from
its corresponding base model (LFCC+SEResNet50+CE). Also, by comparing the
results obtained with other works, it can be seen that this model outperforms
LFCC+ResNet18+OCS, which to the best of our knowledge, shows state-of-the-art
performance. For physical access attacks, the LogPowSpec+EFFA0+CombLoss model
achieved EER=0.86% and t-DCF=0.0239. This result is significantly better than
the base models. Compared to results reported in the 2019 challenge, the
proposed model also appears to outperform 90% of methods which use fusion
models. These results, and other favorable features such as fewer parameters
and shorter runtime compared to other models, prove the efficiency of the
proposed EABN model.

##  6 Conclusion

Spoof detection is considered a major security concern in authentication
systems, particularly the ASV system, demonstrating a clear need for solutions
to combat this issue. There are generally two approaches to detecting spoofing
attacks on ASV systems: the first is to develop an appropriate classifier
targeted specifically at detecting the mentioned attacks, while the second
approach is conducted as a preliminary step for extracting discriminative
features. In the case of the former, most classifiers fail to consider the
issue of optimality in terms of number of parameters and runtime. On the other
hand, most proposed models are not interpretable from a human perspective, and
features are chosen according to expert’s knowledge, and therefore lack
generalization to unseen attacks. However, a modular architecture based on
branches of attention and perception gives the system the ability to easily
utilize any classifier or method to produce an interpretable attention mask
and improve classification. To this end, the proposed combined loss function,
particularly the triplet center loss, succeeded in yielding a discriminative
feature space that can help achieve a more generalized model for unseen
attacks.

The proposed model and loss function were evaluated on ASVspoof 2019 data.
Using LogPowSpec and LFCC features, along with the first-time use of the
EfficientNet-A0 architecture and the well-performed SE-Res2Net50, this study
provides a novel method for detecting spoofs. The findings show that, the
LFCC+SEResNet50+CE model runs with an EER of 1.89% and t-DCF of 0.507 in the
logical access scenario, which to the best of our knowledge, outperforms all
state-of-the-art methods. The EFFA0+CombLoss also obtained an EER of 0.86% and
t-DCF of 0.0239 for the physical access scenario, which is better than 90% of
the models presented for the ASVspoof 2019 challenge. It worth noting that the
EfficientNet-A0 consists of only 95,000 parameters. The findings also shed
light on certain special cases observed for the produced attention masks. For
example, LFCC features outperformed MFCCs in detecting logical access attacks.
Alternatively, to detect replay attacks, focusing more on silent segments and
frequencies in the human speech frequency range can improve the performance.

## References

  * [1] D. A. Reynolds and R. C. Rose, “Robust text-independent speaker identification using gaussian mixture speaker models,” _IEEE transactions on speech and audio processing_ , vol. 3, no. 1, pp. 72–83, 1995.
  * [2] J. H. Hansen and T. Hasan, “Speaker recognition by machines and humans: A tutorial review,” _IEEE Signal processing magazine_ , vol. 32, no. 6, pp. 74–99, 2015.
  * [3] A. Hadid, N. Evans, S. Marcel, and J. Fierrez, “Biometrics systems under spoofing attack: an evaluation methodology and lessons learned,” _IEEE Signal Processing Magazine_ , vol. 32, no. 5, pp. 20–30, 2015.
  * [4] S. K. Ergünay, E. Khoury, A. Lazaridis, and S. Marcel, “On the vulnerability of speaker verification to realistic voice spoofing,” in _2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS)_. IEEE, 2015, pp. 1–6.
  * [5] M. Sahidullah, H. Delgado, M. Todisco, T. Kinnunen, N. Evans, J. Yamagishi, and K.-A. Lee, “Introduction to voice presentation attack detection and recent advances,” in _Handbook of Biometric Anti-Spoofing_. Springer, 2019, pp. 321–361.
  * [6] Z. Wu, T. Kinnunen, N. Evans, J. Yamagishi, C. Hanilçi, M. Sahidullah, and A. Sizov, “Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [7] H. Delgado, M. Todisco, M. Sahidullah, N. Evans, T. Kinnunen, K. A. Lee, and J. Yamagishi, “Asvspoof 2017 version 2.0: meta-data analysis and baseline enhancements,” in _Odyssey 2018-The Speaker and Language Recognition Workshop_ , 2018.
  * [8] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K. A. Lee, “The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection,” 2017.
  * [9] M. Todisco, X. Wang, V. Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. H. Kinnunen, and K. A. Lee, “ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection,” in _Proc. Interspeech 2019_ , 2019, pp. 1008–1012. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2249
  * [10] W. Cai, H. Wu, D. Cai, and M. Li, “The DKU replay detection system for the asvspoof 2019 challenge: On data augmentation, feature representation, classification, and fusion,” pp. 1023–1027, 2019. [Online]. Available: https://doi.org/10.21437/Interspeech.2019-1230
  * [11] M. Todisco, H. Delgado, and N. W. Evans, “A new feature for automatic speaker verification anti-spoofing: Constant q cepstral coefficients.” in _Odyssey_ , vol. 2016, 2016, pp. 283–290.
  * [12] M. J. Alam, G. Bhattacharya, and P. Kenny, “Boosting the performance of spoofing detection systems on replay attacks using q-logarithm domain feature normalization.” in _Odyssey_ , vol. 2018, 2018, pp. 393–398.
  * [13] G. Suthokumar, V. Sethu, C. Wijenayake, and E. Ambikairajah, “Modulation dynamic features for the detection of replay attacks.” in _Interspeech_ , 2018, pp. 691–695.
  * [14] D. Li, L. Wang, J. Dang, M. Liu, Z. Oo, S. Nakagawa, H. Guan, and X. Li, “Multiple phase information combination for replay attacks detection.” in _INTERSPEECH_ , 2018, pp. 656–660.
  * [15] X. Xiao, X. Tian, S. Du, H. Xu, E. S. Chng, and H. Li, “Spoofing speech detection using high dimensional magnitude and phase features: The ntu approach for asvspoof 2015 challenge,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [16] M. R. Kamble, H. Tak, and H. A. Patil, “Amplitude and frequency modulation-based features for detection of replay spoof speech,” _Speech Communication_ , vol. 125, pp. 114–127, 2020.
  * [17] Y. Qian, N. Chen, and K. Yu, “Deep features for automatic spoofing detection,” _Speech Communication_ , vol. 85, pp. 43–52, 2016.
  * [18] H. Zeinali, T. Stafylakis, G. Athanasopoulou, J. Rohdin, I. Gkinis, L. Burget, and J. Černocký, “Detecting spoofing attacks using vgg and sincnet: but-omilia submission to asvspoof 2019 challenge,” pp. 1073–1077, 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2892
  * [19] B. Chettri, T. Kinnunen, and E. Benetos, “Deep generative variational autoencoding for replay spoof detection in automatic speaker verification,” _Computer Speech & Language_, vol. 63, p. 101092, 2020.
  * [20] C.-I. Lai, N. Chen, J. Villalba, and N. Dehak, “Assert: Anti-spoofing with squeeze-excitation and residual networks,” _arXiv preprint arXiv:1904.01120_ , 2019.
  * [21] L. Huang and C.-M. Pun, “Audio replay spoof attack detection by joint segment-based linear filter bank feature extraction and attention-enhanced densenet-bilstm network,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 1813–1825, 2020.
  * [22] P. Korshunov and S. Marcel, “Impact of score fusion on voice biometrics and presentation attack detection in cross-database evaluations,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 4, pp. 695–705, 2017.
  * [23] A. M. Rostami, A. Karimi, and M. A. Akhaee, “Efficientnet-absolute zero for continuous speech keyword spotting,” _arXiv preprint arXiv:2012.15695_ , 2020\.
  * [24] C.-I. Lai, A. Abad, K. Richmond, J. Yamagishi, N. Dehak, and S. King, “Attentive filtering networks for audio replay attack detection,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6316–6320.
  * [25] X. Li, N. Li, C. Weng, X. Liu, D. Su, D. Yu, and H. Meng, “Replay and synthetic speech detection with res2net architecture,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6354–6358.
  * [26] Y. Zhang, F. Jiang, and Z. Duan, “One-class learning towards synthetic voice spoofing detection,” _IEEE Signal Processing Letters_ , vol. 28, pp. 937–941, 2021.
  * [27] H. Fukui, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Attention branch network: Learning of attention mechanism for visual explanation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 705–10 714.
  * [28] D. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. Cubuk, and Q. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” pp. 2613–2617, 09 2019.
  * [29] M. Alzantot, Z. Wang, and M. B. Srivastava, “Deep Residual Neural Networks for Audio Spoofing Detection,” pp. 1078–1082, 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-3174
  * [30] G. Lavrentyeva, S. Novoselov, A. Tseren, M. Volkova, A. Gorlanov, and A. Kozlov, “STC Antispoofing Systems for the ASVspoof2019 Challenge,” pp. 1033–1037, 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-1768
  * [31] Z. Wu, R. K. Das, J. Yang, and H. Li, “Light Convolutional Neural Network with Feature Genuinization for Detection of Synthetic Speech Attacks,” pp. 1101–1105, 2020. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-1810
  * [32] A. Gomez-Alanis, J. A. Gonzalez-Lopez, and A. M. Peinado, “A kernel density estimation based loss function and its application to asv-spoofing detection,” _IEEE Access_ , vol. 8, pp. 108 530–108 543, 2020.
  * [33] X. Cheng, M. Xu, and T. F. Zheng, “Replay detection using cqt-based modified group delay feature and resnewt network in asvspoof 2019,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2019, pp. 540–545.


# Zero-Shot Hierarchical Classification on the Common Procurement Vocabulary
Taxonomy

Federico Moiraghi  University of Milano Bicocca
SpazioDati s.r.l.
Italy
<EMAIL_ADDRESS>Matteo Palmonari  University of Milano
Bicocca
Italy
<EMAIL_ADDRESS>Davide Allavena  SpazioDati s.r.l.
Italy
Federico Morando  SpazioDati s.r.l.
Italy

###### Abstract

111Full-length version of the short paper accepted at COMPSAC 2024.

Classifying public tenders is a useful task for both companies that are
invited to participate and for inspecting fraudulent activities. To facilitate
the task for both participants and public administrations, the European Union
presented a common taxonomy (Common Procurement Vocabulary, CPV) which is
mandatory for tenders of certain importance; however, the contracts in which a
CPV label is mandatory are the minority compared to all the Public
Administrations activities. Classifying over a real-world taxonomy introduces
some difficulties that can not be ignored. First of all, some fine-grained
classes have an insufficient (if any) number of observations in the training
set, while other classes are far more frequent (even thousands of times) than
the average. To overcome those difficulties, we present a zero-shot approach,
based on a pre-trained language model that relies only on label description
and respects the label taxonomy. To train our proposed model, we used
industrial data, which comes from contrattipubblici.org, a service by
SpazioDati s.r.l. that collects public contracts stipulated in Italy in the
last 25 years. Results show that the proposed model achieves better
performance in classifying low-frequent classes compared to three different
baselines, and is also able to predict never-seen classes.

##  I Introduction

“Accounting for a trading volume of EUR 2.448 billion (approx. 16% of the 2017
EU GDP), European public procurement is a major driver for economic growth,
job creation, and innovation”222Quoted from a study titled “Contribution to
Growth European Public Procurement” published on the official European
Parliament website and available online at
https://www.europarl.europa.eu/RegData/etudes/STUD/2018/631048/IPOL_STU(2018)631048_EN.pdf..
Since public tenders are advertised on different European or national
institutional websites, many companies have developed solutions to monetize
simplified and improved access to knowledge about public tenders at scale.
These solutions are usually based on processing crawled data to improve their
quality and make them accessible in different modalities (query,
recommendation, subscriptions, dashboards, etc.); typical processing
operations include cleaning, interlinking, enriching with additional metadata,
storing, and so on. A crucial task to improve access is to classify the data
properly.

The Common Procurement Vocabulary (CPV) has been developed by the European
Union to harmonize the classification of public tenders, which covers multiple
domains and human activities, and “facilitate the processing of invitation to
tender published in the Official Journal of the European Union
(OJEU)”333Definition from wikipedia.. The last version (2008) of the taxonomy
has $9454$ labels, in a tree that has a maximum depth of seven444The taxonomy
and more information are available on the official website:
https://simap.ted.europa.eu/web/simap/cpv.. Each CPV label is represented by a
code of 8 digits and a control code (for manual verification); a natural
language description is also attached to every label (in 24 different
languages) to simplify interpretability by humans. For many tender data
published online, the CPV labels are not available, not accurate, or too
generic. Improving the capability of classifying public tenders based on CPV
taxonomy is relevant to improving access to these data in both the public and
industry sectors. In this paper, we discuss an approach aimed at classifying
unlabeled public tenders to improve their recommendation to customers with
more precise information.

The classification of tender with the CPV taxonomy is a challenging problem
for several reasons. First, data available about tenders are limited: a sample
of this data is provided in the box in Figure 1. Second, classifying over a
taxonomy is more challenging than classifying with “flat” labels: the target
label may be a not-leaf node, enormously increasing the output size. There are
also some natural correlations between labels that can not be ignored (it may
be suitable to classify an observation as a parent or descendant of the true
label). In addition, in the CPV taxonomy, many fine-grained labels are seldom
or never used, thus making it difficult to find training examples for a large
number of labels (e.g. “sesame oil” is hardly ever used, since foodstuff is
usually acquired in a more general way and not by single products).
Classifying over a taxonomy, considering also non-leaf nodes, is a sub-task
referred to as not leaf-mandatory hierarchical classification [1]. This task
has two main difficulties: iterative mechanisms may be needed to classify
observations into classes of increased complexity, as sketched in Figure 1,
and to understand when it does not make sense to keep on exploring finer-
grained labels; it should have a system to avoid error propagation (an error
in a certain point of the tree implies an error in all the following
predictions).

Figure 1: An example of an input document (above), composed by the main field
"object" and other meta-data, and a portion of the taxonomy used for
classification (below). Each label (numerical code) has a canonical
description in 24 different languages. Notice “reservoy” instead of
“reservoir”: such errors are quite common.

This work investigates the exploitation of a bi-directional Pre-trained
Language Model (PLM) [2] to improve the classification of tender data into CPV
taxonomies taking inspiration from the process that a human makes: the task is
easier with a direct comparison of the tender object with the label
description. Our main contribution consists of proposing an architecture for
zero-shot entity linking [3] to support zero-shot classification iteratively
over the taxonomy.

A practical use case for a model that can predict the CPV of a public tender
is filling missing-value of open knowledge bases about invitations to tender
such as TheyBuyForYou [4] or ContrattiPubblici555Made by SpazioDati:
https://spaziodati.eu [5]. Those data are useful in fraud detection (since in
some circumstances some information is not mandatory for the records) as well
as improving integration between European Countries [6].

##  II Related Works

Since CPV prediction is a task not well explored in literature, we divided
this section in two: the first part shows some references that are useful in
understanding both the task and the model development, presenting the
difficulties and strategies of hierarchical (zero-shot) classification. The
second part instead shows previous works on the CPV prediction task, showing
why a zero-shot system may be of interest.

###  II-A Hierarchical Classification

Classifying through a taxonomy is a sub-task of HC, a task well analyzed by
[1] [1]. They define the task as a supervised classification through a
taxonomy that is defined a priori (and not generated on-the-fly). The
difference with Structured Classification is the fact that HC defines the
structure between labels specifically as a tree or a DAG (Direct Acyclic
Graph) and not in free form. Notice that HC output can be either multi-label
or single-label classification. HC is not only aimed at making predictions
when a formal taxonomy is explicitly defined in the dataset (e.g. [7] which
uses an attention-based model for music genre prediction according to a
commercial taxonomy, [8] that classify insects according to flying noise,
dividing between pollinators and not pollinators) but also when the taxonomy
exists and is (in general) accepted by domain experts: for example, [9]
propose a dataset for bird recognition without providing the taxonomy, however
biological taxonomy (although discussed and modified by biologists) can be
accepted.

Depending on the dataset structure and aim, HC models may or may not predict
leaf labels; in addition, various strategies to exploit the tree structure
have been proposed in literature. The most common is the top-down approach,
which consists of a series of classifiers each one predicting the next step
(with growing granularity); on the opposite direction, bottom-up (not well
explored in literature) and big-bang have a classifier which predicts
respectively the leaf labels (whose probabilities are then combined) or the
taxonomy as a whole. Both have their pros and cons: bing-bang models reduce
error propagation given by error multiplication, while on the other hand top-
down models are more computationally expensive yet have fewer labels to manage
(per sub-model). There are also other new or hybrid approaches: for example,
[10] proposes a tree architecture that hard-codes the taxonomy; this approach
however considers only leaf labels. [11] uses a custom loss to penalize
taxonomy violations in a flat classifier for protein-function prediction (a
multi-label task). [12] does the same thing on a Computer Vision multi-label
classification task in order to obtain a better list of tags. As far as we
know, those model-based approaches have not been tested yet on a document
classification dataset.

This task has some peculiar difficulties compared to flat classification.

#### Error Propagation.

In a top-down model, if a sub-classifier makes a wrong classification at some
point, all the subsequent classifications are necessarily wrong, and therefore
the error is propagated through the taxonomy. A simple solution may be using a
single big-bang model which considers all the output labels (either leaves
only or the whole taxonomy); however, this approach is possible only with
small taxonomies. To mitigate the error, [13] proposes a model that considers
multiple root labels first and then expands them all for a single-label
classification task; this strategy however does not improve the results. [14]
instead uses a system for providing each solution sub-tree a probability
score. [15] expands the idea, considering at each step the first $k$ results
for the expansion. We opted for a similar approach, considering all the
classes with a score above a given threshold. With a completely different
approach, [16] adds to each sub-model a fake class that represents the label
parent: if this class is predicted, the second (or third, fourth…) best result
of the previous prediction is explored instead; this is possible since their
model stops on a leaf label.

#### Stopping Strategy.

This feature of an HC system is crucial when non-leaf labels are allowed: it
checks if the classification should go deeper or not, returning the last
results as output. A good stopping may, in addition, mitigate error
propagation since it may claim that the classification is wrong.
While a common strategy is a threshold method (e.g. [17]), there are also
other more sophisticated approaches. [14] uses a threshold over a combined
probability in order to decide when to stop the classification. [18] uses a
simpler approach: it stops when the probability of a child’s label is lower
than his parent’s; the idea is that as granularity increases, the model should
be more certain about the result. With a more sophisticated approach, [19]
presents a binary classifier that considers label probability distribution to
decide if going deeper or not. We took inspiration from this last work, using
a small neural network instead of a logistic regression.

#### Sampling Strategy.

Finally, a particularity of hierarchical classification is that, even in
single-label classification, an observation may belong to multiple classes
with parent-child relationships (e.g. all cats are mammals). This task
characteristic imposes the choice of a sampling strategy, which shows positive
and negative examples for each classifier. [20, 21, 22] show six different
strategies. However, [23], after extensive experimentation, demonstrates that
there is no best strategy for sampling, but it must be chosen according to
task semantics. After some preliminary tests, we opted to train our model
using the exclusive sibling strategy, since it is conceptually similar to hard
negative training used to train BLINK [3], which uses a model similar to ours.

For zero-shot hierarchical document classification, [24] uses a pre-trained
BERT encoder in order to obtain a latent representation of the label; then
another BERT instance actually makes the classification aligning the latent
representation of the input with the associated labels, in a bi-encoder
fashion. Compared to us, it needs long textual descriptions of the labels to
obtain a good latent representation, while in our taxonomy most labels have
very short descriptions, even one word long. [25] instead suggests that
simpler pre-trained embeddings may not be sufficient for fine-grained
classification (on an image-classification task): they propose a method for
obtaining better embeddings fine-tuning a transformer-based model. This
method, however, uses a big-bang strategy for a multi-label task, with poor
performances on the first candidate (with a strong improvement considering
also the followings) and therefore we think it is not well suited for single-
label classification. Compared to our approach, those methods define a fixed
representation of the label in a common space, being more interested in multi-
label than single-label classification.

###  II-B CPV Prediction

Classification through the Common Procurement Vocabulary is a task little
explored in literature. The most recent and more complete work about this
niche domain is [26], which focuses on the Spanish language and approaches the
task as both structured and Extrime Multi-Label Text Classification (XMTC)
task; however, their model is limited to root predictions (45 classes). Their
approach is based on a RoBERTa [27] classifier; according to the authors, they
suffered from data unbalance in the training data. Also [28, 29] predict only
the root labels using SVM models, while [30] uses a BERT model. As far as we
know, there is no work that considers the whole CPV taxonomy. The only work
that considers also (a portion of) non-roots classes is [31], a technical
business report that claims the existence of a model that can predict the
first four digits (roots and two levels above) “with an accuracy of 70%” for a
new product; however, according to [26], they manually engineer the taxonomy.

##  III CPV Taxonomy & the Dataset

The CPV taxonomy is composed of 9454 classes (6531 of which are leaves),
belonging to 45 main labels. The average number of children per class is
$1.00\pm 1.99$, until a maximum depth of $7$. In the whole database, which has
$6.7e7$ contracts recorded, 492 CPV labels have never been observed, and more
than half (5366 labels, affecting about $4.5e6$ observations) have less than
100 records.

Figure 2: Some descriptive metrics about both the taxonomy and the dataset:
frequency of the labels in the training data (top left, logarithmic scale);
IRlBP distribution (top right, logaritmic scale); number of children per node
(middle left, logarithmic scale); cumulative number of classes per depth
(middle right, cumulative); number of words per label description (bottom).

Our task is to predict the CPV code for a contract, given its object and some
meta-data (such as month, value, and typology of contract parties). All the
meta-data are categorical except for the value field, which is numeric and
positive by definition: for this reason, this value was pre-processed to make
it readable by a language model, replacing it with a special token
representing the magnitude order (`€` repeated as many time as the digits of
the integer part of the value, with a maximum of 9). For example, a contract
with a value of €$1000.00$ is represented by the special token `[€€€€]` (which
is the same representation for €$9000.00$).

The main difficulties of the task are the fact that classes are very
imbalanced and that not all the labels are included in the training data. The
fact that an observation may also be labeled with all the ancestor classes
implies that it is not trivial to calculate how unbalanced classes may be. For
this reason, [32] proposes a metric to calculate the imbalance for each class
and for the whole dataset666We use the same mathematical notation of[1].:

$\displaystyle\text{IRLbP}(Y)$
$\displaystyle=\frac{\underset{Y^{\prime}\in\Upsilon}{\max}\sum^{|D|}_{i=1}h(Y^{\prime},Y_{i})}{\sum^{|D|}_{i=1}h(Y,Y_{i})}\hskip
10.0pt\in[1,+\infty)$ HMeanIR
$\displaystyle=\frac{1}{|Y|}\sum_{Y_{i}}^{\Upsilon}\text{IRLbP}(Y_{i})\hskip
10.0pt\in[+1,+\infty)$

Where $D$ is the dataset, $Y$ is the set of labels (and $\Upsilon$ the
taxonomy), $h(Y,Y_{i})=1\Leftrightarrow Y\preceq Y_{i}$. IRLbP score is
essentially the ratio between the frequency of the most-represented class
$Y^{\prime}$ and of the target class $Y$ in the dataset $D$:
$\text{IRLbP}(Y)=1$ means that the class is the most frequent one, higher the
score higher the imbalance. In our dataset, HMeanIR is equal to $119.16$
(meaning that the most frequent class is $119.16$ times more populated than
the average frequency of the classes); IRLbP distribution can be seen in
Figure 2.

In addition, annotation granularity is not always as specific as the declared
labels in the dataset: a contract that only says “foodstuff” may have a label
that indicates a precise kind of food (like, for example, “03210000-6
potatoes”); on the other hand a contract that indicates very specific medical
equipment (such as a specific medicine or an ECG machine) may is labeled just
as “`33000000-0` Medical Equipment”. Other contract objects are not
understandable even by a human but should be inferred by a context that is not
included in the dataset (e.g. previous contracts of the same PA or additional
information provided by the notice board in the physical office); an example
may be “Results of the public notice May 2018.” or “Payment invoice 23/0645”.

This kind of noise can not be labeled as label noise [33] since, for this
task, hyper-specific labels, as well as generic labels, are considered true;
while insufficient information is not a kind of noise at all. However, those
dataset characteristics reduce drastically the model performances either
because incomplete information is included in the dataset or because the
evaluation is not well suited. However, for the current work, we are more
interested in comparing the proposed zero-shot approach with a traditional
hierarchical classifier, and therefore raw performances are not a concern.

##  IV Hierarchical Classification with Cross-encoder

To achieve zero-shot by design we propose a Hierarchical Cross-Encoder (HCE),
which is inspired by cross-encoders used in entity linking and matching tasks
[3]. A cross-encoder evaluates the similarity of an input pair $\langle
x,y\rangle$ using a pre-trained language model to validate the couple as a
sentence classification task. Cross-encoders use bidirectional language models
such, as BERT, which we use in our model. Using a cross-encoder to support
classification is inspired by the observation that both the main input
features (the object of the contract) and the label description are written in
a natural language. Other meta-data are either left as category descriptions
(if categorical) or quantized and introduced as special tokens (if numerical).
In this way, the task is presented to the model as a binary task, which [35]
presents as an effective strategy for hierarchical classification; this asset
reduces also the impact of data imbalance: positive and negative labels are
arbitrarily provided by us. The idea of comparing the input with the output
for hierarchical text classification is not new: [36] trains a model that
indicates if a vector representation of the output fits the input. This
approach, however, is presented as a few-shot architecture (it requires
examples for the vector encoder) and trains more than one model for the
classification; we instead train only one model on the whole architecture,
providing each time a different label description to it.

However, we intend to use a cross-encoder in a way that is different from the
one proposed by [3] for Named Entity Linking: we are interested in evaluating
the similarity between an instance of tender data and a label in the CPV
taxonomy $\Upsilon$, and not between two entity descriptions. In fact, our
hierarchical approach differs in the candidates processed by the model:
usually, a cross-encoder architecture is used to provide an accurate score for
a set of candidates retrieved by a similarity function in a dense space. In
our case, this is not possible for three main reasons: the first is that our
taxonomy has short label descriptions (even one word long) that are difficult
to represent in a dense space; the second one is that is not trivial to
include information about taxonomy structure in a latent representation.
Finally, some descriptions, especially generic ones, have negations (e.g.
“Installation services (except software)”) which complicates the building of a
dense representation. For the second problem, dividing the space into multiple
local sub-problems (e.g. a dense space for each label, to represent its
children) may be a solution; however, the benefits are considerable only if a
label has a higher number of children (notice that the average number of
children per node is low). In our case, it is more efficient in both resources
and inference time to analyze all the descending labels instead of retrieving
a smaller sample.

###  IV-A Encoding and Interpretation

The input of the LM is therefore so composed: at the first position, after the
`[CLS]` special token, there is the contract object (or a reduced version to
not overflow the 512 sub-word limit imposed by BERT architecture777This
happens very hardly, about 1:10’000 times; and a human reader can easily
understand the meaning from the first words. Therefore we think that an
architecture specifically designed for long documents is not needed.). Then,
each feature is preceded by a special token that represents its meaning (e.g.
`[MONTH] December`). Notice that this architecture permits good missing-value
management as well: if a feature is missing, it is just skipped. Finally, a
`[SEP]` token precedes the label description. Here is an example:

`[CLS]` Food stuff `[MONTH]` April `[VALUE]` `[€€€]` […] `[SEP]` Agricultural
and horticultural products

We preferred this setting over using only the `[SEP]` special token since the
validation of the couple improves a little (about $0.02-0.04$) in both
precision and recall. Notice that deep fine-tuning of the network has to be
made in any case because we introduced special tokens for the representation
of the `value` field (which domain experts suggest is useful for the task).

The output is a real number in the interval $[0;+1]$ which represents the
likelihood that the label description fits the input features; notice that,
despite we worked on an Italian dataset, the approach works in the same way in
any languages as well, as there are label descriptions. As with any other
cross-encoder, the output provides information only about one label and
therefore the model requires more than one iteration per observation to
predict between multiple labels, even if siblings: this is the main drawback
of the proposed model.

###  IV-B Training

This input is then used to train the LM for a document classification task, to
validate the couple $\langle x,y\rangle$. In the training step, each
observation has an equal probability of being presented as a positive or
negative example, as well as an equal probability of being an instance of
$Y_{i}\preceq Y_{\text{observed}}$. For the example in Figure 1, the
generation probabilities are summarised in Table I.

Probability | Generation Result
---|---
$\frac{1}{2}$ | ${}^{1}/_{3}$ | a false root label (44 labels)
${}^{1}/_{3}$ | a false children of `15000000-9` (9 labels)
${}^{1}/_{3}$ |  $\langle$ `51810000-3`, $\bot\rangle$
$\frac{1}{2}$ | ${}^{1}/_{3}$ |  $\langle$ `51000000-9`, $\top\rangle$
${}^{1}/_{3}$ |  $\langle$ `51800000-8`, $\top\rangle$
${}^{1}/_{3}$ |  $\langle$ `51820000-6`, $\top\rangle$
Table I: Probabilities of the training generation process.
$P(y=\top)=P(y=\bot)$, so that the output label is now balanced during the
training process. The probability of generating a specific false label depends
on the number of siblings: it becomes, for example:
$P(\text{{15000000-8}})=P(y=\bot)\cdot
P(\text{depth}=0)\cdot\frac{1}{44}=0.38\%$

Negative examples are generated with the exclusive sibling strategy: randomly
choosing one label between the node siblings. This generation procedure is not
fixed and is made at the beginning of each training epoch. We trained the LM
for five epochs using the default parameters provided by `transformers` Python
library by Huggingface. At the end of the training, the BERT model achieves an
accuracy of $0.9441$ on the test set in recognizing if the couple is correct
or not.

###  IV-C Hierarchical Inference

Our model is aimed at computing an advanced similarity function between the
input and output by exploring the taxonomy: the inference procedure resembles
the one used for a top-down local-per-node approach. However, we do not
exclude any path, as [15, 13] do, but we consider all labels whose probability
reach a predefined threshold (set to $0.5$ in this work).

We explore one taxonomy node at a time and compute confidence scores over its
descendants. The exploration is made recursively until either a leaf node or
some exit conditions are reached: either the label score is above a given
threshold or a stopper algorithm, which considers score distribution, triggers
the stop. Notice that, if no root label reaches the given threshold, the HCE
abstains from classification, since no label guarantees the desired
confidence. The inference process is described in detail in Algorithm 1: at
the end of the process, it provides a list of labels descendingly sorted by
score: we intend to retrieve multiple labels ordered for probability and not
only the most probable one.

function Predict($x$, $\Upsilon$, $label=\varnothing$, $score=\varnothing$)

if $score<threshold$ then

return $\varnothing$

else if $\textsc{leaf?}(label,\Upsilon)$ then

return $\\{\langle label,score\rangle\\}$

end if

if $label=\varnothing$ then

$candidates\leftarrow\textsc{roots}(\Upsilon)$

else

$candidates\leftarrow\textsc{children}(label,\Upsilon)$

end if

$\hat{y}\leftarrow\textsc{cross-encoder}(x,candidates)$

if $label\neq\varnothing\wedge\textsc{stopper}(\hat{y})$ then

return $\\{\langle label,score\rangle\\}$

end if

$output\leftarrow\bigcup_{c\in
candidates}\textsc{Predict}(x,\Upsilon,c,\hat{y}_{c})$

return $\textsc{sort by score}(output)$

end function

Algorithm 1: Pseudo-code of the zero-shot classification pipeline.

The reason to use a top-down pipeline is that the number of iterations is
lower compared to a bottom-up or big-bang approach: the main drawback of an
LM-based model, in this context, is its long inference time. An advantage of
our approach, on the other hand, is that we train and use a single model,
which can be used with all the taxonomy labels.

##  V Experiments

We are interested in analyzing if the proposed model reaches good performances
in zero-shot classification, and on uncommon labels. In fact, since labels are
highly imbalanced, we are not interested in good micro-performances but in
good performances on low-frequent or new classes. As discussed in Section II,
previous methods for CPV prediction did not tackle the classification of
tender data at any depth of the CPV taxonomy, but focused on the most generic
classes (root nodes) [26, 28, 29, 30]. As a consequence, they do not propose
any solution to address CPV prediction as a not leaf-mandatory hierarchical
classification problem, which is the main focus of our paper. In addition, as
[26] stress, data are highly imbalanced and some labels are missing: we are
interested in zero-shot classification.

###  V-A Data pre-processing

In our experiments, we use some data transformations on the input that were
found useful during exploratory analyses (omitted in this paper) to aggregate
data and reduce input dimension. In particular, we grouped the date in months
and the municipalities in four macro-areas (North, Middle, South Italy &
Islands). Despite being dataset-specific, municipalities aggregation can be
easily generalized and adapted to other Countries; the HCE however is not
affected by those transformations, receiving the data as is. The value field
instead is processed with a logarithmic transformation (since it is positive
by definition, values equal to zero are considered missing).

###  V-B Baselines

To make a valid comparison, we test our HCE with three baselines, also a
contribution of ours, which makes the classification in three different ways
using the same data as input. In addition, we try to integrate the HCE with
the most promising baseline for predicting unseen classes: this test aims to
investigate if this model may be used with an existing one.

We introduce three baselines which obtain a dense representation of the tender
data and meta-data without relying on an LM (as made in contrattipubblici.org
before transformer-based models become popular). Then, we tested three main
strategies to handle not leaf-mandatory hierarchical classification problems,
which we discussed in Section II and apply different approaches to explore the
taxonomy.

To extract initial features of the input data, each baseline model considers a
vector that is composed of (1)

  * •

the tf-idf for each term in the object field (e.g., “Reservoy supply and
installation” in Figure 1) that appears at least five times in the vocabulary
(to avoid uncommon misspellings)

  * •

a one-hot encoding of categorical labels (i.e., values for the attributes
“contractual choice”, “legal form”, “municipality”, “date”)

  * •

the numeric values (“value”)

Then the input matrix, i.e., the matrix that consists of the initial features
for each input tender data, is compressed with Singular Value Decomposition
(SVD) to obtain a dense representation. SVD was preferred over Principal
Component Analysis (PCA) due to better performances in processing sparse
arrays (such as tf-idf matrices) based on our preliminary results not included
in this work. The result is a dense vector, referred to SVD in the rest of
this work. Observe that, differently from our method, the baseline approach to
feature extraction requires that the full input is pre-processed to compute
the dense representations. Then, the taxonomy is explored in three strategies:

#### Big-bang

One classifier is trained to predict a confidence score for all labels,
independently from their position in the taxonomy. As a classifier, we use a
random forest888Using scikit-learn implementation; the parameter class_weight
is set to balanced to mitigate the imbalance problem presented above.. We
refer to this experiment as RF with SVD (big-bang). This is the only model
tested here to avoid taxonomy exploration since it is predicted all at once;
therefore we can consider this baseline also as a “flat” classifier.

#### Top-down

A classifier is trained for each non-leaf node to predict a confidence score
for its descendants. The exploration of the taxonomy is from the root to
leaves and stops when either a leaf is reached or stop conditions are met.
This approach also uses random forest models as classifiers and therefore we
refer to it as RF with SVD (top-down).

#### Per node

This approach uses a different classifier for every node in the taxonomy to
predict whether the input belongs to that very node or not. Exploration of the
taxonomy proceeds as in the per-node approach, but when a node is explored,
its node-specific Boolean classifier is invoked. Observe that this approach is
the most similar to ours, but our approach uses a single model capable of
computing a score also for unseen classes. Due to computational reasons, we
preferred to use a logistic regression for this approach. So, we refer to this
experiment as LR with SVD (per-node).

The differences between the baselines are summarized in Table II.
For all the baselines we adopt the same stopping conditions used in our model:
we used a small feed-forward neural network with one hidden layer that takes
as input distribution descriptive indexes of the probability scores (maximum
and mean values, variance, kurtosis, skewness, difference between the first
score and second one, and norm of the output vector) and yields an output in
the interval $[0,+1]$. This approach is an evolution of the one presented by
[19] that uses logistic regression and is demonstrated to be effective for the
task. Since we are not interested in analyzing this model, but only the
classification, we do not report any statistics about this component. Another
sub-task that has been willingly ignored is the extraction of the best
candidate from the results: it was just selected in a greedy way choosing the
one with the highest score.

Approach | Exploration | Label range
---|---|---
big bang | no exploration | all-at-once
top-down | root $\to$ leaves | one per parent (over its descents)
per-node | root $\to$ leaves | one per label (boolean)
Table II: Differences between baseline strategies.

Those approaches were preferred over any LM-based classifiers due to their
simplicity: at the end of the process, we have more than $400$ models for the
last two experiments, reaching only 15% of the taxonomy possible sub-trees.

###  V-C HCE Configuration

Our proposed zero-shot model was tested in two modalities. In stand-alone
mode, i.e., our Hierarchical Cross-encoder (HCE) as described in Section IV
and as an additional prediction layer on top of the RF with SVD (top-down)
approach (the best baseline according to our results). This experiments aims
to evaulate if a combination of the two may speed-up the zero-shot
classification prediction, since HCE is slower compared to the baselines.
Since ranges are different, however, the missing label probabilities are then
compared with the most probable label according to the baseline model, and the
higher is taken.

###  V-D Measures

To evaluate the performance of all models we used a precision measure proposed
to evaluate hierarchical classification in [37].

$\displaystyle hP(y,\hat{y})$
$\displaystyle=\frac{\sum_{i}{|y_{i}\cap\hat{y}_{i}|}}{\sum_{i}{|\hat{y}_{i}|}}$
(1) $\displaystyle hR(y,\hat{y})$
$\displaystyle=\frac{\sum_{i}{|y_{i}\cap\hat{y}_{i}|}}{\sum_{i}{|y_{i}|}}$ (2)

Those formulas calculate the number of common ancestors between ground truth
and the predicted class, dividing it either for the predicted label depth
($hP$) or for the ground truth label depth ($hR$). Although those formulas
were originally proposed for micro-precision and micro-recall, we propose to
use them as truth scores for the single predictions and aggregate them by
ground truth to compute macro scores. In this way, it becomes possible to test
macro-aggregated results by calculating the average truth level per class.
Observe that even a model incapable of zero-shot classification may reach a
positive score since this score is positive if an observation with some
ancestors in common with the ground truth is predicted (e.g. a model has never
seen a cat, but recognizes it is a mammal).

###  V-E Results

Table III reports hierarchical precision as defined in Equation 1 for the four
experiments.

Approach | Micro | Macro | $\varrho$
---|---|---|---
RF with SVD (big-bang) | .2174 | .1190 | .2074
RF with SVD (top-down) | .5399 | .3246 | .3109
LR with SVD (per-node) | .1913 | .0647 | .3234
RF with SVD + HCE (top-down) | .5334 | .3226 | .3102
HCE (top-down) | .5334 | .3274 | .2367
Table III: Hierarchical Precision@1 of the experiments. $\varrho$ is the
Pearson correlation between label precision and frequency in the training set.

We notice that top-down baseline model and our HCE achieve comparable
performance: the RF with SVD (top-down) baseline surpasses HCE in micro-
precision by less than $1\%$. However, the correlation between class macro-
precision and class support (column $\varrho$) suggests that performances are
dependent on the class frequency in the training data, despite the model
mitigation. Performances of the other two baselines are poor in comparison to
our approach: the RF with SVD (big-bang) model has over-fit training data (low
correlation is due to a random choice of the target variable, mitigated by the
model, and not due to a good generalization). The LM with SVD (per-node)
approach instead has the worst performance at all. Our model has a low
correlation (the lowest, with the exclusion of the big-bang baseline),
suggesting better performances with unseen or uncommon labels. Our model in
conjunction with the baseline, instead, reaches lower performances compared to
both its sub-components, suggesting poor performances.

To investigate the zero-shot capability of our model, we consider 22 classes
that are included in the test set but not in the train set, and therefore were
never seen by any model: results are reported in Table IV. On the unseen
classes, the zero-shot model (HCE) performs quite similarly compared to the
rest of the dataset. A t-test over the two distributions does not exclude the
hypothesis that the two results are identical in mean and therefore we can
state that our model has similar performances over previously seen and unseen
labels. As a consequence, it is a valid zero-shot approach that relies only on
(short) label descriptions.

Approach | Seen | Unseen | p-value
---|---|---|---
RF with SVD (big-bang) | .1203 | .0474 | .0356
RF with SVD (top-down) | .3304 | .0192 |  $<$ 1e-4
LR with SVD (per-node) | .0647 | .0000 | .0227
RF with SVD + HCE (top-down) | .3284 | .0192 |  $<$ 1e-4
HCE (top-down) | .3273 | .3333 | .9593
Table IV: Hierarchical macro Precision@1 divided in previously seen and unseen
labels. The only two zero-shot models are the HCE ones: our approach can
achieve good performances in the zero-shot task and we can not exclude the
hypothesis that it has similar performances on both seen and unseen labels
(the third column shows p-values for $H_{0}:seen=unseen$).

The conjunction of the two approaches (RF with SVD + HCE) instead obtains the
same low performances as the baseline it is composed of. This result and those
reported in Table III show that the zero-shot model can not be easily used as
is in conjunction with an existing model. Integrating the two models may
require additional effort which is beyond the scope of the present work.

Since our model yields a list of candidates sorted by score, we are interested
also in analyzing how results improve considering more labels than the first
one and how results are distributed through the taxonomy.

| Micro | seen | unseen | $\varrho$
---|---|---|---|---
1 | .5334 | .3273 | .3333 | .2367
2 | .5893 | .3944 | .4333 | .2122
3 | .5978 | .4074 | .4333 | .2053
4 | .6005 | .4107 | .4333 | .2031
5 | .6017 | .4116 | .4333 | .2028
Table V: Considering other retrieved candidates, the model precision has a
significant improvement (initially also for unseen labels). A good choice may
be considering the first two or three results; considering the following
candidates retrieved by the algorithm does not improve the final result for
new labels.

Table V reports results calculated considering the best-suited output for the
classification between the best $k$ retrieved candidates: there is
considerable growth considering the first three candidates (especially for
unseen classes). Also, the coefficient $\varrho$ decreases, showing that if no
unseen classes are found, at least under-represented ones are better
classified.

| micro | macro
---|---|---
depth | seen | unseen | seen | unseen
root | .5203 | .4688 | .5172 | .4598
1 | .5964 | .5197 | .3784 | .3817
2 | .5992 | .1828 | .3315 | .0920
3 | .5506 | .3333 | .3175 | .5000
4 | .5342 | .3333 | .3282 | .4000
5 | .5335 | .2143 | .3302 | .2500
6 | .5334 | .3000 | .3273 | .3333
Table VI: Results per label depth: results deteriorate with the growing of
granularity; unseen classes deteriorate faster than seen classes.

Finally, we analyze how the performances for seen and unseen classes are
distributed through the taxonomy: results are reported in Table VI. We can see
that, for previously seen classes results are quite stable, while, on the
other hand, new classes are more variable. However, the lowest p-value for the
hypothesis that the two distributions are equal in means (for $k=2$,
$pval=0.1822$) can never be rejected for any reasonable value. From those
results, we can notice a significant drop in macro-precision at depth $1$,
meaning that root labels are easier to recognize compared to more specific
labels. Despite the low (yet positive) correlation between label support and
performance, future works may consider different sampling strategies for the
training procedure, prioritizing deeper ones.

##  VI Conclusions

In this work we proposed a BERT-based zero-shot method to make document
classification over a taxonomy whose labels are described in natural language;
we tested it on the Common Procurement Vocabulary taxonomy, using industry
data in the Italian language. Our method, consisting in a Hierarchical Cross-
Encoder, works for single-label classification without the need for output
latent representations: it is not always possible nor trivial to obtain a
valid representation of unseen labels. We tested our proposal against three
baselines composed of the three main approaches in literature; we were not
able to test it against other zero-shot models due to some peculiarity of the
taxonomy, which has short label descriptions (even one word long) and
negations.

Results show that our model has good generalization capabilities also for
previously unseen classes, with very similar performances. Considering the
three more likely candidates, results drastically improve especially for new
labels. However, despite the promising results, our model is quite slow
compared to the baseline: we tested if a simple integration of our model with
an existing pipeline (here represented by the best-performing baseline) may be
successful but with poor results. Future works may explore strategies in how
to speed up the zero-shot model to achieve comparable inference time.

The work presented in this paper has been partially funded by enRichMyData (HE
101070284) and by the PRIN 2022 Project “Discount quality for responsible data
science: Human-in-the-Loop for quality data”.

## References

  * [1] Carlos N Silla and Alex A Freitas  “A survey of hierarchical classification across different application domains”  In _Data Mining and Knowledge Discovery_ 22 Springer, 2011, pp. 31–72
  * [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019  arXiv:1810.04805 [cs.CL]
  * [3] Ledell Wu et al.  “Scalable zero-shot entity linking with dense entity retrieval”  In _arXiv preprint arXiv:1911.03814_ , 2019
  * [4] Ahmet Soylu et al.  “TheyBuyForYou platform and knowledge graph: Expanding horizons in public procurement with open linked data”  In _Semantic Web_ 13.2 IOS Press, 2022, pp. 265–291
  * [5] Giuseppe Futia et al.  “ContrattiPubblici. org, a semantic knowledge graph on public procurement information”  In _AI Approaches to the Complexity of Legal Systems: AICOL International Workshops 2015-2017: AICOL-VI@ JURIX 2015, AICOL-VII@ EKAW 2016, AICOL-VIII@ JURIX 2016, AICOL-IX@ ICAIL 2017, and AICOL-X@ JURIX 2017, Revised Selected Papers 6_ , 2018, pp. 380–393  Springer
  * [6] Ahmet Soylu et al.  “Data Quality Barriers for Transparency in Public Procurement”  In _Information-an International Interdisciplinary Journal_ 13.99, 2022  DOI: 10.3390/info13020099
  * [7] Zhi Zhong et al.  “An Attention-Based Approach to Hierarchical Multi-Label Music Instrument Classification”  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5  IEEE
  * [8] Antonio R.S. Parmezan et al.  “Hierarchical classification of pollinating flying insects under changing environments”  In _Ecological Informatics_ 70, 2022, pp. 101751  DOI: https://doi.org/10.1016/j.ecoinf.2022.101751
  * [9] Stefan Kahl et al.  “Overview of BirdCLEF 2019: large-scale bird recognition in soundscapes”  In _CLEF 2019-Conference and Labs of the Evaluation Forum_ 2380.256, 2019  CEUR
  * [10] Jie Zhou et al.  “Hierarchy-Aware Global Model for Hierarchical Text Classification”  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ Online: Association for Computational Linguistics, 2020, pp. 1106–1117  DOI: 10.18653/v1/2020.acl-main.104
  * [11] Jonatas Wehrmann, Ricardo Cerri and Rodrigo Barros  “Hierarchical Multi-Label Classification Networks”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, Proceedings of Machine Learning Research  PMLR, 2018, pp. 5075–5084  URL: https://proceedings.mlr.press/v80/wehrmann18a.html
  * [12] Luca Bertinetto et al.  “Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [13] Julio Hernandez, L Enrique Sucar and Eduardo F Morales  “Multidimensional hierarchical classification”  In _Expert Systems with Applications_ 41.17 Elsevier, 2014, pp. 7671–7677
  * [14] Mallinali Ramírez-Corona, L Enrique Sucar and Eduardo F Morales  “Hierarchical multilabel classification based on path evaluation”  In _International Journal of Approximate Reasoning_ 68 Elsevier, 2016, pp. 179–193
  * [15] Shunxin Guo and Hong Zhao  “Hierarchical classification with multi-path selection based on granular computing”  In _Artificial Intelligence Review_ 54 Springer, 2021, pp. 2067–2089
  * [16] Yuntao Liu et al.  “Hierarchical learning with backtracking algorithm based on the visual confusion label tree for large-scale image classification”  In _The Visual Computer_ 38.3 Springer, 2022, pp. 897–917
  * [17] Antonio Rafael Sabino Parmezan, Diego Furtado Silva and Gustavo EAPA Batista  “A Combination of Local Approaches for Hierarchical Music Genre Classification.”  In _ISMIR_ , 2020, pp. 740–747
  * [18] Shou Feng, Ping Fu and Wenbin Zheng  “A Hierarchical Multi-Label Classification Algorithm for Gene Function Prediction”  In _Algorithms_ 10.4, 2017  DOI: 10.3390/a10040138
  * [19] Yu Wang, Qinghua Hu, Hao Chen and Yuhua Qian  “Uncertainty instructed multi-granularity decision for large-scale hierarchical classification”  In _Information Sciences_ 586, 2022, pp. 644–661  DOI: https://doi.org/10.1016/j.ins.2021.12.009
  * [20] Roman Eisner et al.  “Improving protein function prediction using the hierarchical structure of the gene ontology”  In _2005 IEEE symposium on computational intelligence in bioinformatics and computational biology_ , 2005, pp. 1–10  IEEE
  * [21] Tiziano Fagni and Fabrizio Sebastiani  “On the selection of negative examples for hierarchical text categorization”  In _Proceedings of the 3rd language technology conference_ , 2007, pp. 24–28
  * [22] Michelangelo Ceci and Donato Malerba  “Classifying web documents in a hierarchy of categories: a comprehensive study”  In _Journal of Intelligent Information Systems_ 28 Springer, 2007, pp. 37–78
  * [23] Bruna Zamith Santos, Gean Trindade Pereira, Felipe Kenji Nakano and Ricardo Cerri  “Strategies for Selection of Positive and Negative Instances in the Hierarchical Classification of Transposable Elements”  In _2018 7th Brazilian Conference on Intelligent Systems (BRACIS)_ , 2018, pp. 420–425  DOI: 10.1109/BRACIS.2018.00079
  * [24] Ilias Chalkidis et al.  “An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels”  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 7503–7515
  * [25] Kai Yi, Xiaoqian Shen, Yunhao Gou and Mohamed Elhoseiny  “Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classification”  In _Computer Vision – ECCV 2022_ , Lecture Notes in Computer Science  Cham: Springer Nature Switzerland, 2022, pp. 116–132  DOI: 10.1007/978-3-031-20044-1˙7
  * [26] Maria Navas-Loro, Daniel Garijo and Oscar Corcho  “Multi-label Text Classification for Public Procurement in Spanish”  In _Procesamiento del Lenguaje Natural_ 69.0, 2022, pp. 73–82  URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6429
  * [27] Yinhan Liu et al.  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019  DOI: 10.48550/arXiv.1907.11692
  * [28] Sangramsing Kayte and Peter Schneider-Kamp  “A Mixed Neural Network and Support Vector Machine Model for Tender Creation in the European Union TED Database.”  In _KMIS_ , 2019, pp. 139–145
  * [29] Oussama Ahmia  “Assisted strategic monitoring on call for tender databases using natural language processing, text mining and deep learning”, 2020
  * [30] “MKaan/multilingual-cpv-sector-classifier”, 2021  URL: https://huggingface.co/MKaan/multilingual-cpv-sector-classifier
  * [31] Deloitte  “Study on up-take of emerging technologies in public procurement.”, 2020  URL: https://ec.europa.eu/docsroom/documents/40102/attachments/1/translations/en/renditions/native
  * [32] Rodolfo M. Pereira, Yandre M.G. Costa and Carlos N. Silla Jr.  “Toward Hierarchical Classification of Imbalanced Data Using Random Resampling Algorithms”  In _Information Sciences_ 578, 2021, pp. 344–363  DOI: 10.1016/j.ins.2021.07.033
  * [33] Benoı̂t Frénay and Michel Verleysen  “Classification in the presence of label noise: a survey”  In _IEEE transactions on neural networks and learning systems_ 25.5 IEEE, 2013, pp. 845–869
  * [34] Thomas Wolf et al.  “Transformers: State-of-the-Art Natural Language Processing”  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ Online: Association for Computational Linguistics, 2020, pp. 38–45  URL: https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * [35] Adriano Rivolli et al.  “An empirical analysis of binary transformation strategies and base algorithms for multi-label learning”  In _Machine Learning_ 109 Springer, 2020, pp. 1509–1563
  * [36] Anqi Wang, Qingcai Chen and Dongfang Li  “FHTC: Few-Shot Hierarchical Text Classification in Financial Domain”  In _Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part II 28_ , 2021, pp. 657–668  Springer
  * [37] Svetlana Kiritchenko, Stan Matwin and A Fazel Famili  “Functional annotation of genes using hierarchical text categorization”  In _Proc. of the ACL Workshop on Linking Biological Literature, Ontologies and Databases: Mining Biological Semantics_ , 2005