#  ProPML: Probability Partial Multi-label Learning

Łukasz Struski1, Adam Pardyl123, Jacek Tabor1 and Bartosz Zieliński13
1Jagiellonian University, Faculty of Mathematics and Computer Science
Emails: {lukasz.struski, jacek.tabor<EMAIL_ADDRESS><EMAIL_ADDRESS>2Jagiellonian University, Doctoral School of
Exact and Natural Sciences  3IDEAS NCBR

###### Abstract

Partial Multi-label Learning (PML) is a type of weakly supervised learning
where each training instance corresponds to a set of candidate labels, among
which only some are true. In this paper, we introduce ProPML, a novel
probabilistic approach to this problem that extends the binary cross entropy
to the PML setup. In contrast to existing methods, it does not require
suboptimal disambiguation and, as such, can be applied to any deep
architecture. Furthermore, experiments conducted on artificial and real-world
datasets indicate that ProPML outperforms existing approaches, especially for
high noise in a candidate set.

##  I Introduction

Deep neural networks are highly effective in many practical applications.
However, their success is heavily dependent on the availability of a large
dataset with accurate labeling. Obtaining such datasets is challenging due to
the cost and discrepancies between labeling experts, like in [1], where four
radiologists review each lung CT independently. Consequently, many
contemporary datasets are weakly labeled, forcing researchers to propose
adequate learning strategies within the Weakly Supervised Learning (WSL)
paradigm [2].

Partial Multiple-label Learning (PML) [3, 4, 5] is a type of WSL problem where
each training instance corresponds to a set of candidate labels, among which
only some are true (see Figure 1). It occurs in many real-world tasks, such as
sentiment analysis and document categorization [6] or classification of
images, audio and video [5].

Simple PML approaches treat all candidate labels as relevant and employ the
off-the-shelf multi-label image classification methods [7, 4]. However, they
perform poorly because of the misleading false labels from the candidate set.
Therefore, many methods adopt a disambiguation strategy, which aims to
identify the relevant labels in the candidate set, e.g. using prior knowledge
[8, 9] or auxiliary information [3]. They show improvement compared to simple
solutions but have two issues. First, prior knowledge or auxiliary information
can be inaccessible in many real-world scenarios. Second, disambiguation is
usually defined on the whole dataset, which is not suitable for deep models.
As a remedy, the most recent methods [5] introduce curriculum-based
disambiguation with consistency regularization. However, they require complex
training that alternates between learning the model and updating the
disambiguation weights, which results in a suboptimal solution.

|  Candidate labels: ✓ Apple ✓ Banana Lemon ✓ Lime ✓ Orange Peach ✓ Pineapple
---|---
Figure 1: In partial multiple-label learning, each training instance
corresponds to a set of candidate labels. Only some of them are true (here,
checkmarked), but we do not know which. This situation can appear, e.g., if
many experts label the same image. Some of them give correct answers, while
others can make mistakes.

In this paper, we introduce ProPML111The code of ProPML is provided together
with the Supplementary Materials at https://github.com/gmum/propml (abbr. from
Probabilistic Partial Multiple-label Learning), a novel probabilistic approach
to partial multiple-label learning problems that address the above-mentioned
issues. For this purpose, we establish a probability function composed of two
components. The first encourages the model to find true labels in the
candidate set, while the second penalizes it for predicting the labels outside
of the candidate set. We show that ProPML can potentially be applied to any
deep architecture and target tasks (such as classification, detection, or
segmentation). Moreover, the experiments conducted on seven artificial and
five real-world datasets indicate that it outperforms existing methods,
especially if the number of fake labels in the candidate set increases (see
Figure 2).

$0.1$$0.2$$0.4$$70$$75$$80$$85$$90$Noise ratiomAP
(%)VOC2007BCEASLQuery2LabelCDCRProPML (our) Figure 2: The mean Average
Precision (mAP) obtained for the top-5 methods on the VOC2007 dataset
artificially corrupted to PML by flipping negative labels into positive with
probability $q$ (noise ratio) equals $\\{0.1,0.2,0.4\\}$. ProPML obtains
outperforms existing methods for higher noise ratio. Moreover, compared to the
second-best complex CDCR based on curriculum learning, ProPML requires only
loss function modification.

Therefore, our contributions can be summarized as follows:

  * •

We introduce ProPML, a novel probabilistic approach to the PML problem, which
does not require a suboptimal disambiguation strategy or other auxiliary
algorithms.

  * •

Our approach requires only loss function modification and, as such, can be
applied to any deep architecture and any target tasks.

  * •

We empirically prove that ProPML outperforms the existing, more composite
methods, both for artificial and real-world datasets.

##  II Related works

Partial Multi-label Learning (PML) deals with noisy labels, combining two
popular branches of machine learning: multi-label learning and partial-label
learning (PLL) [10, 11, 12, 13, 14]. In PLL, each training instance
corresponds to a set of labels among which only one is true. PML, on the other
hand, extends this problem to a multi-label setup, where each training
instance corresponds to a set of labels, among which some (one or more) are
true.

Multiple methods have been proposed to solve the PML problem. Most of them are
based on disambiguation. They try to recover the ground-truth labels from the
candidate set and then use them in a standard multi-label setup. For instance,
methods proposed in [15], PML-FP and PML-LC, that estimate the confidence that
labels are true by exploring the structural information of feature or label
space. Another approach proposed in [8] denoises the candidate labels using
low-rank and sparse matrix decomposition. Method proposed in [9] estimates
label confidence using either pairwise label ranking (PAR-VLS) or maximum a
posteriori reasoning (PAR-MAP). More complicated approaches use two-phase
disambiguation. PML-LRS [8] splits the observed label set into two label
matrices, one with true and one with false labels. Then, the former is
constrained to be low rank using the augmented Lagrange multiplier algorithm
[16] to prevent an overfitting method, and the second is constrained to be
sparse. PML-NI [3] uses a similar technique. However, it additionally jointly
learns the multi-label classifier and noisy label identifier under the
supervision of the observed noise-corrupted label matrix. However, the above
methods may suffer from the problem of cumulative error in the propagation
process.

More recent methods move toward deep learning, like ASL [4] loss function-
based approach. It dynamically down weights easy negative samples, hard
thresholds (discards) the easiest ones, and removes possibly mislabeled
samples. This idea was further extended in CDCR [5] with a curriculum-based
disambiguation that progressively identifies ground-truth labels while
incorporating the varied difficulties of different classes, such as class
imbalance or easiness of distinguishing. Finally, many works transfer the PML
problem to various learning scenarios, including multi-view learning [17],
semi-supervised learning [18], adversarial training [19].

##  III Probabilistic Partial Multi-label Learning

The main idea behind our Probability Partial Label Learning (ProPML) is to
extend Binary Cross Entropy (BCE) loss to the PML setup as closely as
possible. For this purpose, we establish a probability function constructed
out of two components. The first one encourages the model to find true labels
in the candidate set, while the second one penalizes it for predicting the
labels outside of it.

To provide a detailed description of our method, let us recall that in partial
multi-label learning, each training sample $x$ corresponds to a candidate set
$S\subset\\{1,\ldots,C\\}$, which contains at least one true label and some
amount of false labels ($C$ corresponds to the number of classes). Moreover,
let $p_{c}$, where $c\in C$, be the probability of predicting class c, i.e.
obtaining $1$ at $c$th output of any deep architecture. Then, our ProPML loss
is defined as

$L_{ProPML}=-\log\sum\limits_{i\in S}p_{i}-\lambda\cdot\sum\limits_{j\not\in
S}\log(1-p_{j}),$ (1)

where $\lambda$ is the hyperparameter responsible for the balance between the
accurate prediction of labels from $S$ and the remaining labels, and
$\sum_{i\in S}p_{i}\in[0,|S|]$.

To understand the idea behind ProPML, let us analyze its two components
separately. The first component $-\log\sum_{i\in S}p_{i}$ corresponds to the
labels from a candidate set, among which only some labels are true. Its value
is very high if none of the candidate set labels is predicted. However, as
soon as at least one label from $S$ is predicted, the value of this component
goes beyond $0$ and decreases more smoothly, together with increasing sum
$\sum_{i\in S}p_{i}$ (see Figure 3). This way, ProPML puts high pressure on
predicting at least one of the candidate labels, and after that, only
delicately navigates toward predicting other $S$ labels. From this
perspective, ProPML fundamentally differs from the existing PML loss
functions, like ASL [4], which aims to predict all labels from a candidate set
during the entire training.

The second component $-\sum_{j\not\in S}\log(1-p_{j})$ corresponds to labels
outside the candidate set, which we know are all false. Therefore, this
component is taken directly from BCE loss and, similarly like in BCE, it aims
to prevent predicting labels from outside $S$.

Finally, we would like to note that when $S$ is one element set (a single
label case) and $\lambda=1$, the ProPML loss reduces to standard BCE loss.

Figure 3: The first component of ProPML loss corresponds to the expected
number of positively predicted S labels. If this expected value is small, then
none of the labels from $S$ is predicted, and we put strong pressure on
predicting at least one of them. However, when the expected is higher than
$1$, at least one label from $S$ is probably predicted. Hence, we only
delicately navigate toward predicting other S labels, which can be false. We
should keep in mind that each $p_{i}\in[0,1]$, therefore $\sum_{i\in
S}p_{i}\in[0,|S|]$.

##  IV Experiments

In this section, we evaluate the effectiveness of ProPML by comparing its
classification performance with a few existing state-of-the-art PML
approaches. We used a variety of metrics to analyze the performance, including
average precision, coverage loss, Hamming loss, label ranking, and one error
[20]. For the first four metrics, a lower value indicates better performance.
For the average precision, a higher value is better. We run the experiments on
several real-world and synthetic datasets.

### Datasets.

We started with evaluating our model on five artificial
datasets222http://mulan.sourceforge.net/datasets-mlc.html corrupted to the MLP
setup: Bibtex, Birds, Enron, Medical, and Scene. To obtain corrupted datasets,
we first train the SVM classifier on the original labels, and then we analyze
probabilities predicted by this classifier. As false labels, we take those
with the highest probability (excluding originally true labels). We consider
three scenarios, 50%, 100%, or 150%, depending on the ratio of false labels to
true labels. Furthermore, we use five real-world PML datasets: MIRFlickr [9],
Music-emotion [21], Music-style [21], YeastBP, and YeastCC [22]. Finally, we
use two vision datasets, the PASCAL Visual Object Classes Challenge 2017
(VOC2007) [23] and Microsoft Common object in Context 2014 (COCO2014) [24].
Both of them are corrupted by randomly and independently flipping negative
labels into positive ones with probability $q$ equals $\\{0.1,0.2,0.4\\}$ and
$\\{0.05,0.1,0.2\\}$, respectively. All datasets are described in Table I,
including the number of samples, features, classes, and the average size of a
candidate set.

TABLE I: Statistics on artificial, real-world, and vision datasets show significant differences in all properties. Dataset | Size | Dim | $\bm{C}$ | $\bm{\mathbb{E}(|S|)}$
---|---|---|---|---
Artificial datasets
Bibtex | 7395 | 1836 | 159 | 1.23
Birds | 351 | 260 | 19 | 1.86
Enron | 1702 | 1001 | 53 | 3.38
Medical | 978 | 1449 | 45 | 1.25
Scene | 2407 | 294 | 6 | 1.07
Real-world datasets
MIRFlickr | 10433 | 100 | 7 | 1.77
Music-emotion | 6833 | 98 | 11 | 2.42
Music-style | 6839 | 98 | 10 | 1.44
YeastBP | 6139 | 6139 | 217 | 2.22
YeastCC | 6139 | 6139 | 50 | 3.40
Vision datasets333The images from the VOC2017 and COCO2014 datasets were
rescaled to $224\\!\times\\!224$ in the RGB color scale to improve the
processing performance of the neural network.
VOC2007 | 9963 | $224\\!\\!\times\\!\\!224\\!\\!\times\\!\\!3$ | 20 | 1.56
COCO2014 | 122218 | $224\\!\\!\times\\!\\!224\\!\\!\times\\!\\!3$ | 80 | 2.92

### Baseline methods.

The proposed method ProPML is compared to seven state-of-the-art PML
approaches. These approaches include CPLST [25], ML-kNN [26], PAR-MAP and PAR-
VLS (family of PARTICLE [9]), and PML-NI [3]. CPLST is a label-embedding
approach that integrates the concepts of principal component analysis and
canonical correlation analysis. ML-kNN is a nearest neighbor-based multi-label
classification method. PARTICLE transforms the PML task into a multi-label
problem through a label propagation procedure and then induces a calibrated
label ranking model. It can generate a multi-label classifier with either
virtual label splitting (PAR-VLS) or maximum a posteriori inference (PAR-MAP).
PML-NI uses the generation process of noisy labels in the candidate set to
solve PML problems. Each of these methods has distinct advantages and
disadvantages. For example, CPLST allows for integrating multiple features,
while ML-kNN is simpler and easier to use.

Furthermore, for experiments on VOC2007 and COCO2014, we additionally compare
ProPML to other state-of-the-art PML approaches: fPML [22] and PML-LRS [8].
Moreover, we also consider three state-of-the-art multi-label classification
methods: Binary Cross Entropy (BCE), ASL [4] and Query2Label [27]. Finally, we
consider the most recent curriculum disambiguation-based method CDCR [5]. fPML
learns a matrix that associates instances with labels based on estimated
association confidence. PML-LRS decomposes the label matrix into a ground-
truth label matrix and an irrelevant label matrix, with the former constrained
to be low-rank and the latter constrained to be sparse. ASL is based on the
asymmetric properties of PML classification tasks, operating differently on
positive and negative samples. Finally, CDCR uses highly confident prediction
to create a curriculum for label disambiguation.

### Setups.

We conducted experiments in two different setups. The first setup is for
relatively small artificial and real-world datasets commonly used with shallow
machine-learning baseline methods. Another setup is for larger datasets,
typically used with deep networks.

Figure 4: Results for four real-world datasets depending on the value of the $\lambda$ coefficient from the ProPML loss from Eq. 1. For average precision, the higher value, the better. For the remaining metrics, the lower value, the better. We observe a strong trend of decreasing Hamming loss for increasing $\lambda$. However, it does not generalize on other metrics, including the most important average precision, which behavior significantly differs between datasets. Nevertheless, most importantly, we observe the stability of the method for various hyperparameter values. TABLE II: Mean results with standard deviations from five ProPML cross-validation runs on the small real-world datasets. For average precision, the higher value, the better. For the remaining metrics, the lower value, the better. The best result is shown in bold, and the second best is in bold italic. Our method has been regarded as the best or second-best result in 22 out of 25 cases. Moreover, it is the best in 13 out of 25 cases, demonstrating its effectiveness and efficiency despite its simplicity. | Method | Average precision | Coverage | Hamming loss | Label ranking | One error
---|---|---|---|---|---|---
MIRFlickr | CPLST | 78.67 (0.53) | 22.88 (0.20) | 62.66 (0.48) | 12.62 (0.85) | 29.48 (2.39)
ML-kNN | 69.12 (0.40) | 27.07 (0.22) | 76.13 (0.13) | 18.43 (1.35) | 45.17 (12.32)
PAR-MAP | 86.34 (0.35) | 20.22 (0.09) | 15.51 (0.21) | 8.78 (0.10) | 14.30 (0.73)
PAR-VLS | 69.39 (0.27) | 26.55 (0.18) | 17.34 (0.16) | 20.46 (0.19) | 14.00 (0.45)
PML-NI | 78.98 (0.38) | 22.83 (0.15) | 22.14 (0.29) | 12.26 (0.22) | 29.71 (1.38)
ProPML (our) | 86.24 (0.52) | 20.18 (0.37) | 15.98 (0.13) | 8.70 (0.42) | 14.57 (0.80)
Music-emotion | CPLST | 61.45 (0.42) | 40.82 (0.44) | 77.86 (0.06) | 24.36 (0.36) | 46.14 (0.66)
ML-kNN | 55.27 (0.36) | 46.86 (0.33) | 85.05 (0.33) | 30.44 (0.37) | 54.65 (0.75)
PAR-MAP | 59.12 (0.50) | 42.02 (0.58) | 22.62 (0.24) | 25.84 (0.50) | 51.88 (0.86)
PAR-VLS | 61.16 (0.48) | 40.93 (0.44) | 21.25 (0.06) | 26.08 (0.36) | 42.66 (1.23)
PML-NI | 60.74 (0.24) | 40.87 (0.32) | 25.63 (0.78) | 24.57 (0.25) | 47.81 (0.70)
ProPML (our) | 64.57 (0.23) | 39.45 (0.25) | 20.92 (0.13) | 22.61 (0.22) | 39.89 (1.29)
Music-style | CPLST | 73.23 (0.55) | 20.56 (0.57) | 85.57 (0.04) | 14.53 (0.57) | 34.79 (0.72)
ML-kNN | 68.33 (0.25) | 26.32 (0.54) | 85.62 (0.05) | 19.98 (0.46) | 38.18 (0.29)
PAR-MAP | 72.27 (0.53) | 20.44 (0.63) | 11.87 (0.21) | 14.60 (0.48) | 36.73 (0.68)
PAR-VLS | 72.07 (0.37) | 20.35 (0.51) | 11.90 (0.11) | 16.44 (0.36) | 35.82 (0.85)
PML-NI | 73.66 (0.45) | 19.80 (0.42) | 16.01 (1.92) | 13.83 (0.46) | 34.65 (0.71)
ProPML (our) | 75.79 (0.47) | 18.25 (0.48) | 12.40 (0.38) | 12.38 (0.43) | 31.96 (0.80)
YeastBP | CPLST | 23.63 (1.31) | 53.30 (0.99) | 52.54 (0.42) | 30.80 (0.84) | 74.96 (1.65)
ML-kNN | 17.27 (1.63) | 64.38 (2.30) | 99.90 (0.02) | 35.76 (1.25) | 83.20 (2.42)
PAR-MAP | 13.19 (0.70) | 65.44 (1.14) | 4.83 (0.09) | 38.77 (0.94) | 90.63 (1.35)
PAR-VLS | 12.67 (0.42) | 72.13 (0.49) | 3.70 (0.04) | 43.57 (0.61) | 88.56 (0.59)
PML-NI | 40.06 (1.21) | 40.91 (1.05) | 4.24 (0.09) | 21.30 (0.46) | 55.77 (1.80)
ProPML (our) | 32.14 (1.27) | 42.44 (1.88) | 5.54 (0.78) | 20.55 (0.77) | 67.44 (1.40)
YeastCC | CPLST | 41.02 (1.29) | 37.59 (1.14) | 53.48 (0.52) | 26.02 (0.84) | 64.46 (2.66)
ML-kNN | 30.42 (1.76) | 52.30 (1.17) | 99.79 (0.04) | 36.37 (1.40) | 75.06 (2.94)
PAR-MAP | 21.61 (0.80) | 50.82 (0.83) | 9.41 (0.35) | 38.12 (0.90) | 88.13 (0.45)
PAR-VLS | 17.78 (0.47) | 63.10 (0.41) | 8.28 (0.10) | 48.03 (0.74) | 88.74 (0.51)
PML-NI | 56.25 (1.30) | 28.88 (1.42) | 6.64 (0.19) | 18.24 (1.16) | 46.84 (1.79)
ProPML (our) | 55.37 (1.73) | 28.04 (1.74) | 7.20 (0.80) | 18.58 (1.52) | 52.42 (2.38)

For smaller datasets, we use cross-validation with 80% training and 20%
testing sets to demonstrate our results with standard deviations. We followed
the guidelines from the original publications and applied the specified
hyperparameter where possible. Moreover, we applied an MLP model with two
hidden layers to our approach. The training was carried out for 500 epochs
with Adam’s optimizer and stochastic gradient descent as the optimization
method. We chose the parameter $\lambda$ in our cost function Eq. 1 separately
for each dataset using the grid search technique and narrowing it to an
interval between $0.02$ and $1$ (see Figure 4). For large vision experiments,
we use the official train-test splits. Following [4] as a backbone we employ
the TResNet-L architecture [28], pretrained on the ImageNet1K dataset [29].
All images are resized to $224\times 224$. Moreover, as augmentation, we use
standard RandAugment [30] combined with random cutout. Adam optimizer is
applied with a one-cycle learning rate policy [31]. A bayesian optimization
and hyperband [32] is used to tune hyper-parameters, finding the optimal
learning rate between $2\cdot 10^{-5}$ and $2\cdot 10^{-4}$, weight decay
between $10^{-7}$ and $5\cdot 10^{-4}$, and the $\lambda$ for ProPML function
between $0.02$ and $1$. The exponential moving average is applied to model
parameters with a decay rate of 0.9997.

All experiments were implemented using PyTorch444https://pytorch.org and ran
on NVIDIA GeForce RTX $3080$ and Tesla V100.

##  V Results and discussion.

Figure 5: Critical difference diagrams comparing results on small real-world datasets shown in Table II (smaller is better). ProPML performs significantly better than all baseline methods except PML-NI. TABLE III: Mean results with standard deviations from five ProPML cross-validation runs on the small artificial datasets for three false to true label ratios $q=50\%$, $100\%$, or $150\%$. For average precision, the higher value, the better. For the coverage, the lower value, the better. The results of the remaining metrics are provided in the Supplementary Materials. The best result is shown in bold, and the second best is in bold italic. Our method has been regarded as the best or second-best result in 21 out of 30 cases. Moreover, it is the best in 13 out of 30 cases, demonstrating its effectiveness and efficiency despite its simplicity. | Method | Average precision | Coverage
---|---|---|---
|  | $q=50\%$ | $q=100\%$ | $q=150\%$ | $q=50\%$ | $q=100\%$ | $q=150\%$
Bibtex | CPLST | 54.06 (0.49) | 49.19 (0.69) | 48.42 (0.69) | 18.70 (0.83) | 17.42 (0.61) | 16.74 (0.26)
ML-kNN | 32.04 (0.68) | 29.25 (0.76) | 28.48 (0.83) | 34.35 (0.40) | 34.28 (0.51) | 34.61 (0.45)
PAR-MAP | 19.86 (0.91) | 18.13 (0.67) | 16.97 (0.42) | 46.30 (0.33) | 46.93 (0.09) | 47.31 (0.21)
PAR-VLS | 25.55 (0.25) | 21.28 (0.43) | 18.23 (0.54) | 49.92 (0.35) | 52.12 (0.42) | 52.43 (0.59)
PML-NI | 54.45 (0.52) | 49.54 (0.65) | 48.76 (0.71) | 18.00 (0.83) | 16.84 (0.56) | 16.19 (0.30)
ProPML (our) | 53.22 (0.64) | 48.59 (0.95) | 47.46 (0.64) | 23.53 (0.88) | 21.65 (0.54) | 26.88 (0.34)
Birds | CPLST | 56.25 (3.67) | 53.45 (1.93) | 52.25 (1.27) | 29.93 (2.25) | 30.06 (1.37) | 30.25 (0.94)
ML-kNN | 55.66 (4.49) | 51.00 (1.91) | 51.16 (2.46) | 27.22 (2.11) | 27.66 (1.84) | 27.64 (2.52)
PAR-MAP | 47.17 (2.55) | 43.90 (2.06) | 42.96 (1.69) | 34.22 (2.85) | 36.39 (3.44) | 36.55 (2.33)
PAR-VLS | 57.56 (1.92) | 53.00 (3.62) | 49.44 (3.66) | 32.78 (1.16) | 36.82 (3.54) | 37.30 (3.59)
PML-NI | 63.74 (1.44) | 58.15 (1.97) | 57.31 (2.86) | 24.15 (1.27) | 24.19 (1.26) | 24.57 (1.23)
ProPML (our) | 66.88 (2.00) | 66.77 (2.64) | 62.02 (3.18) | 24.35 (1.20) | 22.50 (1.85) | 28.13 (2.46)
Enron | CPLST | 52.30 (0.83) | 50.20 (0.75) | 47.84 (0.52) | 45.46 (1.19) | 39.89 (1.43) | 39.49 (0.62)
ML-kNN | 62.53 (1.48) | 56.82 (0.69) | 56.07 (0.70) | 24.93 (0.58) | 25.58 (0.75) | 26.00 (0.65)
PAR-MAP | 59.29 (1.93) | 50.58 (0.98) | 49.68 (0.71) | 28.19 (1.13) | 28.03 (1.07) | 27.73 (1.11)
PAR-VLS | 53.32 (2.11) | 42.54 (1.80) | 40.49 (0.87) | 45.29 (0.93) | 41.76 (1.28) | 40.67 (1.88)
PML-NI | 64.35 (1.26) | 61.42 (0.89) | 59.72 (0.40) | 32.09 (1.62) | 28.41 (1.30) | 27.37 (0.66)
ProPML (our) | 70.10 (1.01) | 66.97 (0.50) | 66.92 (0.41) | 26.63 (1.11) | 29.59 (1.66) | 27.69 (1.02)
Medical | CPLST | 79.57 (2.55) | 68.06 (2.84) | 67.17 (2.72) | 8.12 (1.90) | 7.86 (2.47) | 8.70 (2.04)
ML-kNN | 80.78 (0.65) | 71.05 (1.05) | 70.20 (1.40) | 5.78 (0.63) | 6.53 (0.64) | 6.57 (0.68)
PAR-MAP | 63.86 (1.46) | 59.57 (1.94) | 59.34 (1.44) | 9.96 (0.48) | 13.00 (0.62) | 12.98 (0.85)
PAR-VLS | 78.07 (1.05) | 63.06 (2.05) | 61.00 (2.37) | 11.22 (0.43) | 16.97 (0.36) | 17.14 (0.36)
PML-NI | 90.84 (0.52) | 82.77 (1.07) | 81.28 (0.79) | 2.58 (0.52) | 3.26 (0.33) | 3.57 (0.36)
ProPML (our) | 89.19 (0.91) | 77.81 (0.51) | 77.47 (3.90) | 3.75 (0.79) | 4.71 (0.72) | 4.67 (0.80)
Scene | CPLST | 82.74 (2.20) | 79.88 (1.66) | 80.24 (1.69) | 10.44 (1.73) | 10.41 (1.01) | 10.26 (1.03)
ML-kNN | 86.27 (1.18) | 84.12 (1.09) | 83.89 (1.24) | 8.10 (0.52) | 8.95 (0.62) | 9.04 (0.74)
PAR-MAP | 85.06 (1.02) | 83.87 (0.81) | 84.09 (0.78) | 8.61 (0.83) | 9.08 (0.53) | 8.89 (0.55)
PAR-VLS | 84.48 (0.80) | 83.85 (0.80) | 84.07 (0.82) | 8.79 (0.58) | 8.95 (0.55) | 8.81 (0.60)
PMLNI | 84.00 (2.15) | 80.68 (2.07) | 80.84 (1.77) | 9.56 (1.56) | 9.88 (1.08) | 9.83 (1.01)
ProPML (our) | 88.70 (1.08) | 86.78 (1.17) | 86.77 (1.01) | 7.10 (0.82) | 7.28 (0.74) | 7.21 (0.64)
TABLE IV: Results on VOC2007 dataset for three noise ratios $q\in\\{0.1,0.2,0.4\\}$. For all metrics (presented in %), the higher value, the better. The best result is shown in bold, and the second best is in bold italic. Baseline results are presented as reported in [5]. Our method achieves the top results for the noise ratio $q\in\\{0.2,0.4\\}$ and is on par with more complex CDCR for $q=0.1$. Moreover, it is the least affected by the increase in noise ratio. Method | $q=0.1$ | $q=0.2$ | $q=0.4$
---|---|---|---
mAP | CF1 | OF1 | mAP | CF1 | OF1 | mAP | CF1 | OF1
PML-NI | 80.32 | 65.64 | 68.56 | 77.63 | 63.44 | 67.26 | 69.95 | 58.21 | 63.39
fPML | 70.36 | 61.49 | 66.01 | 75.29 | 64.41 | 66.83 | 63.22 | 55.20 | 60.70
PMLRS | 80.68 | 66.10 | 68.72 | 77.08 | 62.99 | 66.96 | 63.86 | 55.45 | 61.01
PAR-MAP | 69.61 | 66.69 | 67.12 | 67.66 | 64.43 | 66.38 | 65.69 | 61.18 | 64.45
PAR-VLS | 72.50 | 64.64 | 65.40 | 70.54 | 63.56 | 65.12 | 69.25 | 62.05 | 64.28
BCE | 84.04 | 68.46 | 70.09 | 80.43 | 63.86 | 68.63 | 71.35 | 59.95 | 65.05
ASL | 86.62 | 68.50 | 72.29 | 83.89 | 67.34 | 71.07 | 74.99 | 62.54 | 67.52
Query2Label | 87.14 | 68.10 | 72.82 | 84.81 | 66.75 | 71.72 | 79.90 | 63.88 | 68.84
CDCR | 90.12 | 71.07 | 72.96 | 88.94 | 70.99 | 73.00 | 86.15 | 70.07 | 71.94
ProPML (Our) | 89.43 | 71.39 |  73.10 | 88.96 | 72.89 |  73.40 | 86.82 | 72.36 | 72.72
Supervised ($q=0$) | 90.93 | 78.77 | 77.06 | – | – | – | – | – | –
TABLE V: Results on COCO2014 dataset for three noise ratios $q\in\\{0.1,0.2,0.4\\}$. For all metrics (presented in %), the higher value, the better. The best result is shown in bold, and the second best is in bold italic. Baseline results are presented as reported in [5]. Our method is on par with more complex CDCR for $q=0.05,0.1$ and achieves slightly worse scores for $q=0.2$. However, it is still better than other loss-based methods, such a ASL. Method | $q=0.05$ | $q=0.1$ | $q=0.2$
---|---|---|---
mAP | CF1 | OF1 | mAP | CF1 | OF1 | mAP | CF1 | OF1
PML-NI | 60.36 | 52.97 | 58.70 | 59.87 | 52.61 | 58.56 | 58.92 | 51.81 | 58.20
fPML | 57.93 | 51.29 | 57.55 | 57.75 | 51.08 | 57.54 | 57.14 | 50.86 | 57.47
BCE | 72.39 | 68.17 | 72.47 | 70.43 | 64.41 | 69.30 | 67.30 | 48.80 | 56.02
ASL | 76.92 | 61.11 | 66.98 | 75.14 | 60.00 | 66.26 | 72.28 | 57.65 | 64.88
Query2Label | 76.56 | 60.83 | 66.69 | 75.39 | 60.50 | 66.30 | 73.59 | 59.36 | 65.54
CDCR | 77.95 | 73.48 |  77.42 | 77.14 | 72.99 |  77.00 | 76.13 | 72.58 | 76.58
ProPML (Our) | 77.15 | 71.96 | 76.64 | 76.03 | 71.53 | 75.57 | 74.92 | 62.86 | 67.01
Supervised ($q=0$) | 80.23 | 75.03 | 77.75 | – | – | – | – | – | –

In this section, we compare the results of our method with those of baseline
approaches in two different setups of experiments for three types of datasets.
We investigate how our approach fares against existing techniques and how the
results vary in different scenarios.

### The real-world datasets.

Table II shows the results of the experiment conducted on real-world data.
Five-fold cross-validation was used to measure the mean and variance for each
method and dataset. The two highest values for each dataset and measure are
highlighted, with the best result shown in bold and the second best result
presented in bold italic. Our method has been regarded as the best or second-
best result in 22 out of 25 cases. Moreover, it is the best in 13 out of 25
cases, with up to $3.12$ gap to the second-best approach in the case of
average precision and up to $1.55$ gap in the case of coverage. This trend is
also confirmed by the Critical Difference (CD) diagrams [33] in Figure 5,
generated using all the results shown in Table II. They show that for small
artificial datasets, ProPML performs significantly better than all baseline
methods except PML-NI. While in the case of large datasets PML-NI method
produces much worse results than our (see Tables IV and V).

In Figure 4, we observe how the results change depending on the $\lambda$
parameter. There is a clear impact of this parameter on the Hamming loss. It
is expected because Hammin loss is the fraction of incorrectly predicted
labels. Therefore, it is calculated by summing up the number of false labels
predicted as true and dividing it by the total number of labels. At the same
time, the second part of the ProPML loss is also responsible for penalizing
incorrectly classified false labels. Moreover, $\lambda$ is a coefficient of
the second component of the ProPML loss. Therefore, its increase is expected
to decrease the second component and, in consequence, the Hamming loss.
Unfortunately, similar trends are not observed for other metrics. Therefore,
the best solution requires searching for the optimal $\lambda$. Nevertheless,
most importantly, we observe the stability of the method for various
hyperparameter values.

### Artificial datasets.

In Table III and the Supplementary Materials, we show results for synthetic
datasets created by adding noisy labels with false to true labels ratio
$q=50\%$, 100%, and 150%. These results confirm the effectiveness of our
method against the considered approaches, with up to $7.2$ gap to the second-
best approach in the case of average precision and up to $1.6$ gap in the case
of coverage. Similar to the real-world datasets, this trend is also confirmed
by the Critical Difference (CD) diagrams in Figure 6, generated using all the
results shown in Table III and the Supplementary Materials. Moreover, again,
ProPML performs significantly better than all baseline methods except PML-NI.
However, interestingly, the gap between those two models and the remaining
approaches grows together with increased $q$. Please, consult the
Supplementary Materials to analyze the influence of parameter $\lambda$ from
Eq. 1 on the considered metrics.

### Vision datasets.

Table IV shows the results of our experiments on the VOC2007 dataset. As
presented, our method outperforms all state-of-the-art PML approaches, as well
as methods dedicated to multi-label image classification. The performance is
on par with more complex, curriculum disambiguation-based CDCR, providing
slightly better results in higher noise conditions, with the improvement of
$0.67$ percentage points in mean Average Precision (mAP) and $2.29$ percentage
points Class average F1 score (CF1) for $q=0.4$. Results for COCO2014 are
presented in Table V. Again our method outperforms standard PML and multi-
label image classification methods. However, contrary to VOC2007, in the case
of COCO2014, ProPML falls slightly behind the CDCR approach. We hypothesize
that this is caused by a much larger number of possible labels (80 in COCO2014
vs. 20 in VOC2007) and a larger variance in semantic characteristics between
classes making this scenario much more imbalanced. However, note that ProPML
is much more straightforward than CDCR as it does not require an external
curriculum-building process nor consistency regularization and can be used as
a drop-in replacement for regular binary cross-entropy loss.

(a) False to true label ratio $q=50\%$.

(b) False to true label ratio $q=100\%$.

(c) False to true label ratio $q=150\%$.

Figure 6: Critical difference diagrams comparing results on small artificial
datasets with false to true label ratio $q=50\%$, $100\%$, or $150\%$ shown in
Table III and in the Supplementary Materials (smaller is better). ProPML
performs significantly better than all baseline methods except PML-NI.
Moreover, the gap between those two models and the remaining approaches grows
together with increased $q$.

##  VI Conclusions

In this paper, we introduced ProPML, a direct and easy-to-implement
probabilistic approach to partial multi-label learning that adapts binary
cross entropy to PML setup. It considers the expected value of positively
predicted labels from a candidate set and uses it to encourage the model to
find the true labels. As a result, it commonly overpasses the existing
solutions, especially for the high number of false labels in a candidate set,
as we present in multiple experiments.

In the future, we want to focus on adopting our function to other target
tasks, such as detection or segmentation, and other modalities, such as
tabular data, text, sound, or graphs.

## Acknowledgements

This research was partially funded by the National Science Centre, Poland,
grants no. 2020/39/D/ST6/01332 (work by Łukasz Struski), 2022/47/B/ST6/03397
(work by Bartosz Zieliński), and 2021/41/B/ST6/01370 (work by Adam Pardyl and
Jacek Tabor). Some experiments were performed on servers purchased with funds
from a grant from the Priority Research Area (Artificial Intelligence
Computing Center Core Facility) under the Strategic Programme Excellence
Initiative at Jagiellonian University. We gratefully acknowledge Polish high-
performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH)
for providing computer facilities and support within computational grant no.
PLG/2022/015753.

## References

  * [1] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman _et al._ , “The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans,” _Medical physics_ , vol. 38, no. 2, pp. 915–931, 2011.
  * [2] Z.-H. Zhou, “A brief introduction to weakly supervised learning,” _National Science Review_ , vol. 5, no. 1, pp. 44–53, 2018.
  * [3] M.-K. Xie and S.-J. Huang, “Partial multi-label learning with noisy label identification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 7, pp. 3676–3687, 2021.
  * [4] T. Ridnik, E. Ben-Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, and L. Zelnik-Manor, “Asymmetric loss for multi-label classification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 82–91.
  * [5] F. Sun, M.-K. Xie, and S.-J. Huang, “A deep model for partial multi-label image classification with curriculum based disambiguation,” _arXiv preprint arXiv:2207.02410_ , 2022.
  * [6] W. Liu, H. Wang, X. Shen, and I. W. Tsang, “The emerging trends of multi-label learning,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 11, pp. 7955–7974, 2021.
  * [7] Z.-M. Chen, X.-S. Wei, P. Wang, and Y. Guo, “Multi-label image recognition with graph convolutional networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5177–5186.
  * [8] L. Sun, S. Feng, T. Wang, C. Lang, and Y. Jin, “Partial multi-label learning by low-rank and sparse decomposition,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 5016–5023.
  * [9] M.-L. Zhang and J.-P. Fang, “Partial multi-label learning via credible label elicitation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 10, pp. 3587–3599, 2020.
  * [10] M.-L. Zhang, F. Yu, and C.-Z. Tang, “Disambiguation-free partial label learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 29, no. 10, pp. 2155–2167, 2017.
  * [11] X. Gong, D. Yuan, and W. Bao, “Top-partial label machine,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 11, pp. 6775–6788, 2021.
  * [12] Y. Yan and Y. Guo, “Partial label learning with batch label correction,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 04, 2020, pp. 6575–6582.
  * [13] J. Lv, M. Xu, L. Feng, G. Niu, X. Geng, and M. Sugiyama, “Progressive identification of true labels for partial-label learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 6500–6510.
  * [14] Ł. Struski, J. Tabor, and B. Zieliński, “Propall: Probabilistic partial label learning,” _arXiv preprint arXiv:2208.09931_ , 2022.
  * [15] M.-K. Xie and S.-J. Huang, “Partial multi-label learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [16] Z. Lin, M. Chen, and Y. Ma, “The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices,” _arXiv preprint arXiv:1009.5055_ , 2010.
  * [17] Z.-S. Chen, X. Wu, Q.-G. Chen, Y. Hu, and M.-L. Zhang, “Multi-view partial multi-label learning with graph-based disambiguation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 3553–3560.
  * [18] M.-K. Xie and S.-J. Huang, “Semi-supervised partial multi-label learning,” in _2020 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2020, pp. 691–700.
  * [19] Y. Yan and Y. Guo, “Adversarial partial multi-label learning with label disambiguation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 12, 2021, pp. 10 568–10 576.
  * [20] M.-L. Zhang and Z.-H. Zhou, “A review on multi-label learning algorithms,” _IEEE transactions on knowledge and data engineering_ , vol. 26, no. 8, pp. 1819–1837, 2013.
  * [21] M. J. Huiskes and M. S. Lew, “The mir flickr retrieval evaluation,” in _Proceedings of the 1st ACM international conference on Multimedia information retrieval_ , 2008, pp. 39–43.
  * [22] G. Yu, X. Chen, C. Domeniconi, J. Wang, Z. Li, Z. Zhang, and X. Wu, “Feature-induced partial multi-label learning,” in _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2018, pp. 1398–1403.
  * [23] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, pp. 303–308, 2009.
  * [24] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [25] Y.-N. Chen and H.-T. Lin, “Feature-aware label space dimension reduction for multi-label classification,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [26] M.-L. Zhang and Z.-H. Zhou, “Ml-knn: A lazy learning approach to multi-label learning,” _Pattern recognition_ , vol. 40, no. 7, pp. 2038–2048, 2007.
  * [27] S. Liu, L. Zhang, X. Yang, H. Su, and J. Zhu, “Query2label: A simple transformer way to multi-label classification,” _arXiv preprint arXiv:2107.10834_ , 2021.
  * [28] T. Ridnik, H. Lawen, A. Noy, E. Ben Baruch, G. Sharir, and I. Friedman, “Tresnet: High performance gpu-dedicated architecture,” in _proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2021, pp. 1400–1409.
  * [29] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [30] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 702–703.
  * [31] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in _Artificial intelligence and machine learning for multi-domain operations applications_ , vol. 11006. SPIE, 2019, pp. 369–386.
  * [32] S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1437–1446.
  * [33] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” _The Journal of Machine learning research_ , vol. 7, pp. 1–30, 2006\.