# Debiased Pairwise Learning from Positive-Unlabeled Implicit Feedback

Bin Liu2 , Qin Luo2 , Bang Wang4 2Bin Liu and Qin Luo contributed equally.
4Correspondence to<EMAIL_ADDRESS>School of Electronic Information and
Communications,
Huazhong University of Science and Technology (HUST), Wuhan, China,
Email: {liubin0606, luo_qin<EMAIL_ADDRESS>
###### Abstract

Learning contrastive representations from pairwise comparisons has achieved
remarkable success in various fields, such as natural language processing,
computer vision, and information retrieval. Collaborative filtering algorithms
based on pairwise learning also rooted in this paradigm. A significant concern
is the absence of labels for negative instances in implicit feedback data,
which often results in the random selected negative instances contains false
negatives and inevitably, biased embeddings. To address this issue, we
introduce a novel correction method for sampling bias that yields a modified
loss for pairwise learning called debiased pairwise loss (DPL). The key idea
underlying DPL is to correct the biased probability estimates that result from
false negatives, thereby correcting the gradients to approximate those of
fully supervised data. The implementation of DPL only requires a small
modification of the codes. Experimental studies on five public datasets
validate the effectiveness of proposed learning method.

###### Index Terms:

pairwise learning, contrastive learning, PU learning, Bayesian personalized
ranking

##  I Introduction

Pairwise learning is a kind of learning paradigm that leverages pairwise
comparisons to capture relative relationships between data pairs [1, 2]. In
contrast to pointwise learning, pairwise learning encourages an encoder to
encode differential features between samples rather than pixel-level features
of individual samples, usually leading to better generalization performance,
particularly in scenarios where the absolute values of the samples are less
meaningful [3, 4, 1, 5, 3]. Pairwise learning has become a fundamental
component of many modern machine learning algorithms and has facilitated
significant advancements in various domains, including natural language
processing, image and speech recognition, and recommendation systems [7, 3, 8,
9].

In the context of collaborative filtering, pairwise learning has been widely
employed to predict rankings by contrasting positive and negative examples,
with the most well-known approach being Bayesian Personalized Ranking (BPR)
[9], which has dominated the task of learning to rank from implicit feedback
and achieved state-of-the-art performance. From a statistical perspective, BPR
maximizes the posterior probability of observed ordered pairs between positive
and negative samples. From a numerical computation perspective, the BPR loss
function encourages the model to assign higher scores to positive examples
than negative examples. In the embedding space, the BPR loss aims to pull the
embedding of positive examples closer to the anchor embedding (i.e., user)
while pushing negative examples apart from the anchor embedding.

However, a prominent issue encountered in implicit collaborative filtering is
obtaining negative feedback data can be challenging. This is mainly due to the
fact that users usually only provide positive feedback by indicating their
preferences or interests through interactions such as clicks, purchases, or
ratings. As a result, the training set is often in the form of positive
unlabeled (PU) data (Fig 1), where only positive samples are available. PU
data is commonly existed in various machine learning domains [10, 11] such as
unsupervised image classification [12, 13, 8, 12], positive examples are
obtained data augmentation. In practice, a common approach to handling
positive-unlabeled implicit feedback is to treat un-interacted items as
negative samples for model training [9, 14, 15]. The resulting false
negatives, i.e., items that user are preferred in future (positively labeled)
but unseen during the training phase, can significantly harm the performance
of the model by introducing biases and incorrect representations of the users
and items [16, 17].

Figure 1: An illustrative example of the positive-unlabeled (PU) problem in
machine learning. In natural language processing (NLP), positive samples can
be obtained from the context, while negative examples are randomly sampled
from vocabulary. In computer vision (CV), positive samples are obtained via
data augmentation, while negative samples are sampled from unlabeled images.
In recommendation systems (RS), positive samples are obtained from user
interactions, while negative examples are randomly sampled from uninteracted
items. Specifically, in recommendation systems, uninteracted items may be
unseen item but be potentially liked in the future by the user, referred to as
false negatives.

To address the this issue, negative sampling has been widely investigated, and
have shown promising results. Negative sampling can be classified into two
types: the first kind is static negative sampling, which employs a fixed
sampling distribution using some kind of side information that is independent
of the model’s training status, can result in easy samples, leading to
suboptimal performance compared to dynamic negative sampling. Moreover, static
negative sampling is severely limited by the availability of side information
that serving as effective supervision signal. In contrast, dynamic negative
sampling adjusts the sampling distribution using the model-dependent
information such as predicted scores, aiming at sampling hard negative samples
with high scores or top rankings to boost performance, which is prone to
encounter false negative examples [16, 17]. In addition, mini-batch training
based on GPU batch computation require fixed positive and negative samples to
be loaded into the dataloader before starting training, which may not be
compatible with dynamic negative sampling. Dynamic negative sampling is
typically implemented with additional computational and storage overhead, such
as memorizing the predictive scores of previous training epochs or calculating
the samples’ predictive scores outside the mini-batch.

In this paper, we focus on the most general form of implicit feedback data,
where there is no side information available for supervision. Specifically, we
propose a correction for sampling bias from unlabeled data that yields a
modified loss for pairwise learning called debiased pairwise loss (DPL). The
key idea underlying DPL is to correct the biased probability estimates that
result from false negatives, thereby correcting the gradients to approximate
those of fully supervised data. The proposed objective is easy to implement
and does not require additional side information for supervision or excessive
storage and computational overhead.

##  II Preliminaries

In this section, we introduce the notation, review Bayesian personalized
ranking for implicit collaborative filtering.

###  II-A Notation

Denote an user item pair $(u,i)$ as a sample $\mathbf{x}$, where
$u\in\mathcal{U},i\in\mathcal{I}$. Let
$\mathcal{X}=\\{\mathbf{x}|u\in\mathcal{U},i\in\mathcal{I}\\}$ be the sample
space indicating all the user item pairs and $\mathcal{Y}=\\{-1,+1\\}$ be the
class label indicating whether user prefer the item or not. A decision
function $g:\mathcal{X}\rightarrow\mathbb{R}$ assigns value indicating the
predicted preference level $g(\mathbf{x})\in\mathbb{R}$. Denote the class
conditional density of positives as $p^{+}(\mathbf{x})=p(\mathbf{x}|+1)$,
while class conditional density of positives as
$p^{-}(\mathbf{x})=p(\mathbf{x}|-1)$. So the marginal distribution
$p(\mathbf{x})=p^{+}(\mathbf{x})\tau^{+}+p^{-}(\mathbf{x})\tau^{-}$, where
$\tau^{+}=1-\tau^{-}$ is the prior probability $p(c(\mathbf{x})=+1)$.

###  II-B Bayesian Personalized Ranking

In implicit collaborative filtering, personalized ranking of a set of items
are learned from pairwise comparisons of two randomly samples
$(\mathbf{x}^{+},\mathbf{x}^{-})$. BPR[9] adopts the well-known Bradley-Terry
model to describe the likelihood of observing positive instance
$\mathbf{x}^{+}$ been preferred over negative instances $\mathbf{x}^{-}$

$\displaystyle\mathbb{P}(g(\mathbf{x}^{+})>g(\mathbf{x}^{-})|\Theta)=\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}^{-}))$
(1)

where $\sigma(x)=\frac{1}{1+\exp(-x)}$ is the sigmoid function. BPR loss
maximizes the probability of ordered pairs
$g(\mathbf{x}^{+})>g(\mathbf{x}^{-})$ consisting of a positive instance and a
negative instance:

$\displaystyle\mathcal{L}_{BPR}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}^{+}\sim
p^{+}(\mathbf{x})\\\ \mathbf{x}^{-}\sim
p^{-}(\mathbf{x}^{-})\end{subarray}}\log\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}^{-}))$
$\displaystyle=$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}^{+}\sim
p^{+}(\mathbf{x})\\\ \mathbf{x}^{-}\sim
p^{-}(\mathbf{x})\end{subarray}}\log\frac{1}{1+\exp(-g(\mathbf{x}^{+})+g(\mathbf{x}^{-}))}$
$\displaystyle=$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}^{+}\sim
p^{+}(\mathbf{x})\\\ \mathbf{x}^{-}\sim
p^{-}(\mathbf{x}^{-})\end{subarray}}\log\frac{\exp(g(\mathbf{x}^{+}))}{\exp(g(\mathbf{x}^{+}))+\exp(g(\mathbf{x}^{-}))}$
(3)

It is worth noting that Eq. (3), an equivalent form of the Bayesian
Personalized Ranking (BPR) loss, is identical to the noise contrastive
estimation (NCE) loss [18], which is a special instance of the InfoNCE loss
[7] with a single negative sample (i.e., N=1). Moreover, in the collaborative
filtering scenario, where users and items form a bipartite graph, typically
only user embeddings are selected as anchor point.

In practice, positive instances are sampled from items that have been
interacted with, denoted as $\mathbf{x}^{+}\in\mathcal{D}^{+}$, while negative
instances are sampled from items that have not been interacted with, denoted
as $\mathbf{x}^{-}\in\mathcal{D}^{-}$. Therefore, the empirical counterpart of
Eq (II-B) is given by:

$\displaystyle\mathcal{L}_{\text{BPR}}=-\frac{1}{|\mathcal{D}^{+}|\times|\mathcal{D}^{-}|}\sum_{\mathbf{x}^{+}\in\mathcal{D}^{+}}\sum_{\mathbf{x}^{-}\in\mathcal{D}^{-}}$
$\displaystyle\ln\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}^{-}))$ (4)
$\displaystyle-\lambda||\Theta||^{2},$

where $\lambda||\Theta||^{2}$ is the regularization term to balance the
variance and bias and avoid over-fitting. Notably, the regularization term
$\lambda||\Theta||^{2}$ is equivalent to the logarithm of the prior density of
a Gaussian distribution, thereby offering a posterior probability-based
interpretation to Eq (4). In the original Bayesian Personalized Ranking (BPR)
paper [9], Eq (4) is interpreted as the maximum posterior estimator of
observed ordered pairs.

##  III Proposed Method

###  III-A Sampling bias

Due to the absence of labeled negative samples, one can only sample negative
examples from unlabeled data during training for optimizing Eq (4), resulting
in the following biased optimization objective:

$\displaystyle\mathcal{L}=-\frac{1}{|\mathcal{D}^{+}|\times|\mathcal{D}^{u}|}\sum_{\mathbf{x}^{+}\in\mathcal{D}^{+}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}$
$\displaystyle\ln\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))$ (5)
$\displaystyle-\lambda||\Theta||^{2},$

Next, we investigate the impact of biased optimization objectives on the
learned embeddings of user-item pairs $\Theta$:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\Theta}$ $\displaystyle=$
$\displaystyle\frac{\partial\mathcal{L}_{\text{BPR}}}{g(\mathbf{x})}\cdot\frac{g(\mathbf{x})}{\Theta}$
(6) $\displaystyle=$
$\displaystyle[1-\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))]\cdot\frac{g(\mathbf{x})}{\Theta}$
(7)

Eq (7) is the result of the differential chain rule, where the first term
$[1-\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))]$ is determined by the form of the
loss function and the second term $\frac{g(\mathbf{x})}{\Theta}$ is determined
by the decision function. For a fixed model, the second term remains the same.

In the first term of Eq (7), the real-valued sigmoid function
$\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))\in[0,1]$ is been interpreted as the
likelihood of positive item $\mathbf{x}^{+}$ being preferred over negative
item [9]. However, since $\mathbf{x}$ is an unlabeled sample with a positive
class prior $\tau^{+}$, this leads to a biased estimate of the probability
(see Fig 2). Intuitively, biased
$\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))$-values will lead to incorrect
gradient magnitude ${\partial\mathcal{L}_{\text{BPR}}}/{\partial\Theta}$,
resulting in inaccurate user-item representations when performing stochastic
gradient descent learning algorithm.

To maintain notational brevity in mathematical expressions, we define a
mapping
$h:\mathcal{X}\times\mathcal{X}\rightarrow\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))$
that maps two samples into a sigmoid function. The problem then becomes how to
approximate the value of $h(\mathbf{x}^{+},\mathbf{x}^{-})$ using samples from
the positive and unlabeled populations, respectively. Specifically, given a
set of positive samples $\\{\mathbf{x}^{+}_{i}\\}_{i=1}^{M}$ and a set of
unlabeled samples $\\{\mathbf{x}_{j}\\}_{j=1}^{N}$, we aim to estimate the
value of $h(\mathbf{x}^{+},\mathbf{x}^{-})$. By doing so, we can correct the
gradients to approximate those of fully supervised data, thereby leading to
better generalization performance of learned user-item representations.

###  III-B Bias Correction

Figure 2: An illustrative example that explains the value
$\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))$ calculate using positive-unlabeled
data pairs is a biased probability estimate. Based on the principle of
inclusion-exclusion, the event
$\mathcal{A}(+,u)=\mathcal{A}_{1}(+,-)+\mathcal{A}_{2}(+,+)$. Note the
objective is to maximize the likelihood of positive item being preferred over
negative item, so the probability, which models the event
$\mathcal{A}_{1}(+,-)$ consisting of positive-negative pairs, is the
optimization objective. However, in practice we optimize the biased
probability that models the event $\mathcal{A}(+,u)$.

In order to approximate the value of $h(\mathbf{x}^{+},\mathbf{x}^{-})$ using
a set of positive samples and a set of unlabeled samples, we start by
establishing a relationship between the joint distribution of positive-
unlabeled sample pairs, denoted by $p_{\textsc{pu}}$, and the joint
distribution of positive-negative sample pairs, denoted by $p_{\textsc{pn}}$.

The first sample of a sample pair, denoted by $\mathbf{x}_{1}$, is
deterministically drawn from the positive class conditional probability
$p^{+}(\mathbf{x})$, while the second sample, denoted by $\mathbf{x}_{2}$, is
drawn from the marginal distribution $p(\mathbf{x})$. Consequently, the joint
distribution of positive-unlabeled sample pairs $p_{\textsc{pu}}$ can be
expressed as follows:

$\displaystyle p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=$ $\displaystyle p^{+}(\mathbf{x}_{1})p(\mathbf{x}_{2})$ (8)
$\displaystyle=$ $\displaystyle
p^{+}(\mathbf{x}_{1})[p^{+}(\mathbf{x}_{2})\tau^{+}+p^{-}(\mathbf{x}_{2})\tau^{-}]$
(10) $\displaystyle=$
$\displaystyle\tau^{+}p^{+}(\mathbf{x}_{1})p^{+}(\mathbf{x}_{2})$
$\displaystyle+\tau^{-}p^{+}(\mathbf{x}_{1})p^{-}(\mathbf{x}_{2})$

Eq. (8) is obtained since $(\mathbf{x}_{1},\mathbf{x}_{2})$ are independently
drawn.Meanwhile, Eq. (10) is the full probability decomposition of the
probability of the marginal distribution $p(\mathbf{x})$. By rearranging Eq.
(10), we can establish a relationship between the desired joint distribution
$p_{\textsc{pn}}(\mathbf{x}_{1},\mathbf{x}2)$ of positive-negative sample
pairs and the joint distribution
$p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})$ of positive-unlabeled sample
pairs, expressed as:

$\displaystyle p_{\textsc{pn}}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=$ $\displaystyle p^{+}(\mathbf{x}_{1})p^{-}(\mathbf{x}_{2})$
(11) $\displaystyle=$
$\displaystyle\frac{1}{\tau^{-}}[p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})-\tau^{+}p^{+}(\mathbf{x}_{1})p^{+}(\mathbf{x}_{2})]$

Therefore, the expected value of $h(\mathbf{x}_{1},\mathbf{x}_{2})$ over the
desired joint distribution $p_{\textsc{pn}}(\mathbf{x}_{1},\mathbf{x}_{2})$ of
positive-negative sample pairs, which represents the expected likelihood of a
positive sample $\mathbf{x}_{1}$ being preferred over $\mathbf{x}_{2}$ under
fully labeled data, can be computed as follows:

$\displaystyle\mathbb{E}_{p_{\textsc{pn}}(\mathbf{x}_{1},\mathbf{x}_{2})}h(\mathbf{x}_{1},\mathbf{x}_{2})$
(12) $\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{1}}\int_{\mathbf{x}_{2}}h(\mathbf{x}_{1},\mathbf{x}_{2})p_{\textsc{pn}}(\mathbf{x}_{1},\mathbf{x}_{2})d{\mathbf{x}_{1}}d{\mathbf{x}_{2}}$
(15) $\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{1}}\int_{\mathbf{x}_{2}}h(\mathbf{x}_{1},\mathbf{x}_{2})[\frac{1}{\tau^{-}}p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle-\frac{\tau^{+}}{\tau^{-}}p^{+}(\mathbf{x}_{1})p^{+}(\mathbf{x}_{2})]d{\mathbf{x}_{1}}d{\mathbf{x}_{2}}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{1}}\int_{\mathbf{x}_{2}}h(\mathbf{x}_{1},\mathbf{x}_{2})[\frac{1}{\tau^{-}}p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})d{\mathbf{x}_{1}}d{\mathbf{x}_{2}}$
$\displaystyle-\int_{\mathbf{x}_{1}}\int_{\mathbf{x}_{2}}\frac{\tau^{+}}{\tau^{-}}p^{+}(\mathbf{x}_{1})p^{+}(\mathbf{x}_{2})]d{\mathbf{x}_{1}}d{\mathbf{x}_{2}}$
$\displaystyle=$
$\displaystyle\frac{1}{\tau^{-}}\mathbb{E}_{p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})}h(\mathbf{x}_{1},\mathbf{x}_{2})-\frac{\tau^{+}}{\tau^{-}}\mathbb{E}_{p_{\textsc{pp}}(\mathbf{x}_{1},\mathbf{x}_{2})}h(\mathbf{x}_{1},\mathbf{x}_{2})$
(16)

Input: Mini-batch data $\mathcal{R}$, number of extra positive items $M$,
number of negative items $N$, positive class prior tau, batch size bs, score
function $g(\cdot)$ ($M,N\geq 1$).

Output: Embeddings $\Theta\in\mathbb{R}^{d}$

1 `scores = ` $g(\mathcal{R})$` #[bs*(1+M+N)] the scores of each items in the
mini-batch data `

2 `pos_scores = scores[:,:M+2] ` `#[bs*(1+M)]`

3 `neg_scores = scores[:,M+2:] ` `#[bs*N]`

4 `pu_prob = sigmoid(pos_scores[:, 0:1] - neg_scores) ` `#[bs*N]`

5 `pp_prob = sigmoid(pos_scores[:, 0:1] - pos_scores[:,1:]) ` `#[bs*M]`

6 `pn_prob = pu_prob.mean(dim=-1)/(1-tau) - tau*pp_prob.mean(dim=-1)/(1-tau)`
`#[bs, ]`

7 `dpl_loss = - log (pn_prob).mean() `

8 `Update embeddings based on gradient w.r.t. dpl_loss.`

Result: Final embeddings.

Algorithm 1 Pseudocode of proposed Debiased Pairwise Learning (DPL) method in
a PyTorch-like style.

The first term of Eq.(16) represents the expectation over positive-unlabeled
sample pairs, and this term can be estimated empirically using $N$ unlabeled
samples $\\{\mathbf{x}_{n}\\}_{n=1}^{N}$ given the first positive sample
$\mathbf{x}^{+}$:

$\displaystyle\mathbb{E}_{p_{\textsc{pu}}(\mathbf{x}_{1},\mathbf{x}_{2})}h(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{n=1}^{N}h(\mathbf{x}^{+},\mathbf{x}_{n}),$ (17)

and the second term of Eq.(16) represents the expectation over positive-
positive sample pairs. This term can be estimated empirically using $M$
additional positive samples $\\{\mathbf{x}^{\prime}_{n}\\}_{m=1}^{M}$ given
the first positive sample $\mathbf{x}^{+}$:

$\displaystyle\mathbb{E}_{p_{\textsc{pp}}(\mathbf{x}_{1},\mathbf{x}_{2})}h(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=$
$\displaystyle\frac{1}{M}\sum_{m=1}^{M}h(\mathbf{x}^{+},\mathbf{x}_{m}^{\prime}).$
(18)

Inserting Eq. (17) and Eq. (18) back to Eq. (16), we obtain the final
empirical estimate:

$\displaystyle\hat{P}_{\textsc{pn}}=\frac{1}{N\tau^{-}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}\in\mathcal{D}^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$
(19)

As shown, the first term in the equation is computed using positive-unlabeled
sample pairs, but with an additional correction term included to offset the
biased likelihood estimation resulting from false negative samples being
included in the unlabeled set. The correction term computed from positive-
positive pairs in Eq. (19) may appear peculiar, and we refer the reader to
Fig. 3 for an intuitive explanation of this correction term. The final
empirical form of the debiased pairwise loss (DPL) is presented below:

$\displaystyle\mathcal{L}_{\textsc{dpl}}=-\frac{1}{|\mathcal{D}^{+}|\times|\mathcal{D}^{u}|}\sum_{\mathbf{x}^{+}\in\mathcal{D}^{+}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}$
$\displaystyle\ln\hat{P}_{\textsc{pn}}.$

###  III-C Implementation

When training with the BPR loss, a negative sample $j$ is required for each
$(u,i)$ pair, resulting in a data entry represented as a $(u,i,j)$ triplet. To
address sampling bias, the DPL loss requires $M\geq 1$ additional positive
examples and $N\geq 1$ negative examples for each $(u,i)$ pair. Following the
same data entry format as BPR, each DPL data entry is organized as
$(u,i,i_{1},i_{2},\cdots,i_{M},j_{1},j_{2},\cdots,j_{N})$. This can be easily
implemented by rewriting the `collate_fn` function of the `Dataloader`. As a
result, each mini-batch data is structured as follows:

$\displaystyle\text{batch
size}\left\\{\left[\begin{array}[]{cc:cccc:cccc}u^{1}&{i}&i_{1}&i_{2}&\ldots&i_{M}&j_{1}&j_{2}&\ldots&j_{N}\\\
u^{2}&i&i_{1}&i_{2}&\ldots&i_{M}&j_{1}&j_{2}&\ldots&j_{N}\\\
\vdots&\vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\vdots&\ddots&\vdots\\\
u^{bs}&i&i_{1}&i_{2}&\ldots&i_{M}&j_{1}&j_{2}&\ldots&j_{N}\end{array}\right]\right.$
(24)

Each data entry includes N unlabeled items $j_{1},j_{2},\cdots,j_{N}$, and N
predicted scores $\hat{x}_{j}^{1},\hat{x}_{j}^{2},\cdots,\hat{x}_{j}^{N}$ for
these items can be calculated. For the given positive sample $(u,i)$ with its
score $\hat{x}_{i}$, N pu probability values can be obtained based on the
predicted scores of the N negative samples. Therefore, the estimated PU
probability value based on the mini-batch data is given by:

$\hat{P}_{\textsc{pu}}=\frac{1}{N}\sum_{n=1}^{N}\sigma(\hat{x}_{i}-\hat{x}_{j}^{n})$

Similarly, M predicted scores
$\hat{x}_{i}^{1},\hat{x}_{i}^{2},\cdots,\hat{x}_{i}^{M}$ for M positive items
$i_{1},i_{2},\cdots,i_{M}$ can be computed, and the estimated PP probability
value based on the mini-batch data is given as follows:

$\hat{P}_{\textsc{pp}}=\frac{1}{M}\sum_{m=1}^{M}\sigma(\hat{x}_{i}-\hat{x}_{i}^{m})$

Therefore, the corrected probability value that approximates the probability
of user preferred positive item over negative item is given by:

$\hat{P}_{\textsc{pn}}=\frac{1}{\tau^{-}}\hat{P}_{\textsc{pu}}-\frac{\tau^{+}}{\tau^{-}}\hat{P}_{\textsc{pp}}$

Algorithm 1 presents the pseudocode for the DPL algorithm in a PyTorch-like
style.

Complexity: We first analyze the complexity of the baseline method BPR, which
is related to the scoring function $g$. Here, we take matrix factorization
with latent dimension $d$ as an example, and this analysis can be easily
extended to other models. Given a mini-batch data with a batch size of $bs$
consisting of $(u,i,j)$ training triples, the forward scoring calculation
involves a total of $2\times bs$ item score predictions, resulting in a time
complexity of $\mathcal{O}(2bs\times d)$. In backward propagation, at most
$3\times bs$ embeddings are updated, and a total of $5bs\times d$ operations
are involved, resulting in a time complexity of $\mathcal{O}(bs\times d)$.
Similarly, for DPL, a mini-batch data involves a total of $(M+N+1)\times bs$
scoring calculations and $(M+N+2)\times bs$ embedding updates, involving a
total of $(2M+2N+3)\times bs\times d$ operations and a time complexity of
$\mathcal{O}(bs\times d)$, since $M$ and $N$ are usually set to small
constants such as $M=3$ and $N=3$ in practice. Specifically, when $M=0$ and
$N=1$, the number of operations involved in DPL is the same as that in BPR.
Therefore, DPL has strictly linear complexity relative to BPR, without any
calculation or storage overhead outside of mini-batch data.

##  IV Theoretical Analysis

The main idea behind DPL is to improve the biased probability that calculated
using the positive-unlabeled data pairs. This is achieved by sampling
additional positive and negative examples to estimate the expected probability
value of users liking positive items more than negative items, which is given
by Eq. (19), and used to replace the original biased probability estimate. To
demonstrate Eq. (19) is a good estimator, we first prove that it is an
unbiased estimator of the AUC risk.

###### Lemma 0.1

Let positive data $\mathbf{x}^{+}$ i.i.d. drawn from positive class
conditional density $p^{+}(\mathbf{x})$, and unlabeled data $\mathbf{x}$
i.i.d. drawn from marginal density
$p(\mathbf{x})=\tau^{+}p^{+}(\mathbf{x})+\tau^{-}p^{-}(\mathbf{x})$. Then Eq.
(19) is the unbiased estimate of AUC risk:

$\mathbb{E}\hat{Pr}=R_{AUC}$

###### Proof 0.1

$\displaystyle\mathbb{E}\hat{Pr}$ $\displaystyle=$
$\displaystyle\int_{\mathbf{x}^{+}}[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}\mathbb{E}_{\mathbf{x}\sim
p}h(\mathbf{x}^{+},\mathbf{x})$ (26)
$\displaystyle-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}\in\mathcal{D}^{+}}\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]p^{+}(\mathbf{x}^{+})d\mathbf{x}^{+}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}^{+}}[\frac{1}{\tau^{-}}\mathbb{E}_{\mathbf{x}\sim
p}h(\mathbf{x}^{+},\mathbf{x})$
$\displaystyle-\frac{\tau^{+}}{\tau^{-}}\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]p^{+}(\mathbf{x}^{+})d\mathbf{x}^{+}.$

Since

$\displaystyle\frac{1}{\tau^{-}}\mathbb{E}_{\mathbf{x}\sim
p}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{\tau^{-}}\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$ $\displaystyle=$
$\displaystyle\frac{1}{\tau^{-}}\int_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})p(\mathbf{x})d\mathbf{x}-\frac{\tau^{+}}{\tau^{-}}\int_{\mathbf{x^{\prime}}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})p^{+}(\mathbf{x}^{\prime})d\mathbf{x^{\prime}}$
$\displaystyle=$
$\displaystyle\frac{1}{\tau^{-}}\int_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})[\tau^{+}p^{+}(\mathbf{x})+\tau^{-}p^{-}(\mathbf{x})]d\mathbf{x}$
$\displaystyle-\frac{\tau^{+}}{\tau^{-}}\int_{\mathbf{x^{\prime}}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})p^{+}(\mathbf{x}^{\prime})d\mathbf{x^{\prime}}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})p^{-}(\mathbf{x})d\mathbf{x}$
(28) $\displaystyle=$
$\displaystyle\int_{\mathbf{x^{-}}}h(\mathbf{x}^{+},\mathbf{x}^{-})p^{-}(\mathbf{x}^{-})d\mathbf{x}^{-}$
(29)

where Eq (0.1) is obtained by decomposing the marginal distribution
$p(\mathbf{x})=\tau^{+}p^{+}(\mathbf{x})+\tau^{-}p^{-}(\mathbf{x})$, and Eq
(29) replaces the integration variable $\mathbf{x}$ in Eq (28) with
$\mathbf{x}^{-}$ to enhance readability. Inserting Eq (29) back to Eq (26) we
obtain

$\displaystyle\mathbb{E}\hat{P}_{r}$ $\displaystyle=$
$\displaystyle\int_{\mathbf{x}^{+}}\int_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x}^{-})p^{-}(\mathbf{x}^{-})p^{+}(\mathbf{x}^{+})d\mathbf{x}^{+}d\mathbf{x}^{-}$
(30) $\displaystyle=$ $\displaystyle R_{AUC}.$ (31)

which completes the proof. If we substitute the function $h$ in Eq (30) with
the 0-1 loss $\mathbb{I}(\mathbf{x}^{+},\mathbf{x}^{-})$, then Eq (30) exactly
defines the AUC metric. However, due to the discrete nature of the 0-1 loss
function, a differentiable surrogate loss is often used in practice when
optimizing for AUC metric. We refer to Fig. 3 for an intuitive explanation of
Lemma 0.1.

Figure 3: An illustrative example that explains the origin of the correction
term involving two positive samples. Let us consider a positive-unlabeled
dataset, where ${x_{1},x_{2}}$ are positive samples and ${x_{3},x_{4},x_{5}}$
are unlabeled samples. The ground truth labels of the unlabeled samples are
not accessible during training. The area under the curve (AUC) that is
computed using positive examples and unlabeled data pairs is denoted as
$AUC_{\textsc{pu}}$. It can be seen that $AUC_{\textsc{pu}}$ is the sum of two
terms. The first term is the clean AUC estimate computed using positive-
negative (PN) data pairs and is denoted as $AUC_{\textsc{pn}}$. The second
term is the pseudo AUC estimate computed using positive-positive (PP) data
pairs and is denoted as $AUC_{\textsc{pp}}$. To optimize the clean AUC, we
should subtract the $AUC_{\textsc{pp}}$ from the $AUC_{\textsc{pu}}$.

Next, we seek a an idealized objective for DPL to approximate.

###### Definition 0.1

For fixed positive sample $\mathbf{x}^{+}$, let
$\mathbb{P}_{\textsc{pn}}=\mathbb{E}_{\mathbf{x}^{-}\sim
p^{-}}h(\mathbf{x}^{+},\mathbf{x}^{-})$ being the expected probability value
of positive sample $\mathbf{x}^{+}$ being preferred over true negative item
$\mathbf{x}^{-}$. Then, taking the log-likelihood over all positive samples,
we can define the supervised loss as:

$\displaystyle\mathcal{L}_{\textsc{sup}}=-\mathbb{E}_{\mathbf{x}^{+}\sim
p^{+}}\log\mathbb{P}_{\textsc{pn}}$ (32)

Minimizing the expected log-likelihood $-\log\mathbb{P}_{\text{PN}}$ over all
positive samples results in the maximization of the likelihood that any
positive sample is preferred over any true negative sample, which is exactly
our optimization objective under fully supervised data. Therefore,
$\mathcal{L}_{\textsc{sup}}$ is an idealized objective for DPL to approximate.
Lemma 0.2 proves that the DPL estimate is asymptotically consistent with this
idealized loss as M and N approach infinity.

###### Lemma 0.2

For $M,N\rightarrow+\infty$, we have

$\displaystyle\mathcal{L}_{\textsc{dpl}}\rightarrow\mathcal{L}_{\textsc{sup}}$
(33)

###### Proof 0.2

Lebesgue’s dominated converge theorem states that for a sequence of measurable
functions $f_{n}$ that are bounded, then

$\displaystyle\lim\limits_{n\rightarrow\infty}\int_{\Omega}f_{n}=\int_{\Omega}\lim\limits_{n\rightarrow\infty}f_{n}$

so

$\displaystyle\lim\limits_{\begin{subarray}{c}M,N\rightarrow+\infty\end{subarray}}\mathcal{L}_{\textsc{dpl}}$
$\displaystyle=$
$\displaystyle-\lim\limits_{\begin{subarray}{c}M,N\rightarrow+\infty\end{subarray}}\mathbb{E}_{\mathbf{x}^{+}\sim
p^{+}}\log[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}h(\mathbf{x}^{+},\mathbf{x})$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}\in\mathcal{D}^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle=$ $\displaystyle-\mathbb{E}_{\mathbf{x}^{+}\sim
p^{+}}\lim\limits_{\begin{subarray}{c}M,N\rightarrow+\infty\end{subarray}}\log[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}h(\mathbf{x}^{+},\mathbf{x})$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}\in\mathcal{D}^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle=$ $\displaystyle-\mathbb{E}_{\mathbf{x}^{+}\sim
p^{+}}\lim\limits_{\begin{subarray}{c}M,N\rightarrow+\infty\end{subarray}}\log[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}\in\mathcal{D}^{u}}h(\mathbf{x}^{+},\mathbf{x})$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}\in\mathcal{D}^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle=$ $\displaystyle-\mathbb{E}_{\mathbf{x}^{+}\sim
p^{+}}\log[\frac{1}{\tau^{-}}\mathbb{E}_{\mathbf{x}\sim
p}h(\mathbf{x}^{+},\mathbf{x})$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-\frac{\tau^{+}}{\tau^{-}}\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]$

Applying the result of Eq (29), we have

$\displaystyle\frac{1}{\tau^{-}}\mathbb{E}_{\mathbf{x}\sim
p}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{\tau^{-}}\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$ (38) $\displaystyle=$
$\displaystyle\int_{\mathbf{x^{-}}}h(\mathbf{x}^{+},\mathbf{x}^{-})p^{-}(\mathbf{x}^{-})d\mathbf{x}^{-}$
$\displaystyle=$ $\displaystyle\mathbb{E}_{\mathbf{x}^{-}\sim
p^{-}}h(\mathbf{x}^{+},\mathbf{x}^{-})$ $\displaystyle=$
$\displaystyle\mathbb{P}_{\text{PN}}$

Inserting Eq (38) back to Eq (0.2) we obtain

$\mathcal{L}_{\textsc{dpl}}\rightarrow\mathcal{L}_{\textsc{sup}},$

which completes the proof.

Lemma 0.2 demonstrates that the DPL loss $\mathcal{L}_{\textsc{dpl}}$ is
asymptotically consistent with the idealized loss $\mathcal{L}_{\textsc{sup}}$
as M and N approach infinity. However, in practical applications, only a
finite number of M and N are available, resulting in the empirical estimate
$\hat{\mathcal{L}}\textsc{dpl}$. Next, Lemma 0.3 bounds the estimation error
$|\hat{\mathcal{L}}_{\textsc{dpl}}-\mathcal{L}_{\textsc{sup}}|$.

###### Lemma 0.3

With probability at least $1-\delta$, we have

$\displaystyle|\hat{\mathcal{L}}_{\textsc{dpl}}-\mathcal{L}_{\textsc{sup}}|\leq
e^{2}\sqrt{\frac{2\pi}{N}}+e^{2}\tau^{+}\sqrt{\frac{2\pi}{M}}$

where $\mathfrak{R}_{n}$ is bipartite Rademacher complexity.

###### Proof 0.3

Without loss of generality, we use the cosine similarity as score function to
simplify the analysis, meaning that all embeddings are mapped on a hypersphere
with radius 1. Recall that $h$ is a function that maps two samples into a
sigmoid
function,$h(\mathbf{x}^{+},\mathbf{x})=\sigma(g(\mathbf{x}^{+})-g(\mathbf{x}))$.
As, $g(\cdot)\in[0,1]$, so $-2\leq g(\mathbf{x}^{+})-g(\mathbf{x})\leq 2$, and
$\frac{1}{1+e^{2}}\leq h(\mathbf{x}^{+},\mathbf{x})\leq\frac{1}{1+e^{-2}}$.

For fixed the positive sample $\mathbf{x}^{+}$, we denote the difference
between the integrands of the asymptotic and non-asymptotic objective as
$\triangle$:

$\displaystyle\triangle$ $\displaystyle=$
$\displaystyle|\log[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}}p(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle-\log\mathbb{E}_{\mathbf{x}^{-}\sim
p^{-}}h(\mathbf{x}^{+},\mathbf{x}^{-})|$ $\displaystyle=$
$\displaystyle|\log[\frac{1}{N\tau^{-}}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M\tau^{-}}\sum_{\mathbf{x}^{\prime}}p(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle-\log\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[\frac{1}{\tau^{-}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{\tau^{-}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]|$
$\displaystyle=$
$\displaystyle|\log\frac{\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})}{\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]}|$

We first seek to bound the probability that $\triangle$ exceeds $\epsilon$ for
fixed $\mathbf{x}^{+}$. Applying the fact
$\mathbb{P}(|X|\geq\epsilon)=\mathbb{P}(X\geq\epsilon)+\mathbb{P}(-X\geq\epsilon)$,
we have

$\displaystyle\mathbb{P}(\triangle\geq\epsilon)=\mathbf{I}(\epsilon)+\mathbf{II}(\epsilon)$
(39)

where

$\displaystyle\mathbf{I}(\epsilon)$ (40) $\displaystyle=$
$\displaystyle\mathbb{P}\left(\log\frac{\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})}{\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]}\geq\epsilon\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{P}\left(\frac{\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})}{\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]}-1\geq\epsilon\right)$
(42) $\displaystyle\leq$
$\displaystyle\mathbb{P}(\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim p(\mathbf{x})\\\
\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]\geq\frac{\epsilon}{1+e^{2}})$

where Eq (42) is obtained since $\log x\leq x-1$ for $x>0$. Eq (42) is
obtained since $\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim p(\mathbf{x})\\\
\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]=(1-\tau^{+})\mathbb{E}_{\mathbf{x}\sim
p^{-}(\mathbf{x})}h(\mathbf{x}^{+},\mathbf{x})\geq 1/(1+e^{2})$. The second
term is bounded similarly:

$\displaystyle\mathbf{II}(\epsilon)$ (43) $\displaystyle=$
$\displaystyle\mathbb{P}\left(\log\frac{\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]}{\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})}\geq\epsilon\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{P}\left(\frac{\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]}{\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})}-1\geq\epsilon\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{P}(\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim
p(\mathbf{x})\\\ \mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]$
$\displaystyle-[\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]\geq\frac{\epsilon}{1+e^{2}})$

Combining Eq (42) and Eq (43) we have

$\displaystyle\mathbb{P}(\triangle\geq\epsilon)$ $\displaystyle\leq$
$\displaystyle\mathbb{P}(|\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}\sim p(\mathbf{x})\\\
\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})\end{subarray}}[h(\mathbf{x}^{+},\mathbf{x})-\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]|\geq\frac{\epsilon}{1+e^{2}})$
$\displaystyle=$
$\displaystyle\mathbb{P}(|[\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\mathbb{E}_{\mathbf{x}\sim
p(\mathbf{x})}h(\mathbf{x}^{+},\mathbf{x})]$
$\displaystyle-[\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})-\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})}\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})]|\geq\frac{\epsilon}{1+e^{2}})$
$\displaystyle\leq$
$\displaystyle\mathbb{P}(|\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\mathbb{E}_{\mathbf{x}\sim
p(\mathbf{x})}h(\mathbf{x}^{+},\mathbf{x})|$
$\displaystyle+|\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})-\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})}\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})|\geq\frac{\epsilon}{1+e^{2}})$
$\displaystyle\leq$
$\displaystyle\mathbf{III}(\epsilon)+\mathbf{IV}(\epsilon).$ (47)

where

$\displaystyle\mathbf{III}(\epsilon)$ $\displaystyle=$
$\displaystyle\mathbb{P}(|\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})-\mathbb{E}_{\mathbf{x}\sim
p(\mathbf{x})}h(\mathbf{x}^{+},\mathbf{x})|$ (48)
$\displaystyle\geq\frac{\epsilon}{2(1+e^{2})})$
$\displaystyle\mathbf{IV}(\epsilon)$ $\displaystyle=$
$\displaystyle\mathbb{P}(|\frac{\tau^{+}}{M}\sum_{\mathbf{x}^{\prime}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})-\mathbb{E}_{\mathbf{x}^{\prime}\sim
p^{+}(\mathbf{x})}\tau^{+}h(\mathbf{x}^{+},\mathbf{x}^{\prime})|$ (49)
$\displaystyle\geq\frac{\epsilon}{2(1+e^{2})})$

Eq (0.3) is obtained due to $|X-Y|\leq|X|+|Y|$, Eq (47) is obtained due to
$\mathbb{P}(|X|+|Y|\leq\epsilon)\leq\mathbb{P}(|X|\leq\epsilon/2)+\mathbb{P}(|Y|\leq\epsilon/2)$.
McDiarmid’s inequality states that, for independent random variables
$X_{1},X_{2},\dots,X_{n}$, where ${\displaystyle X_{i}\in{\mathcal{X}}_{i}}$
for all $i$, if ${\displaystyle
f:{\mathcal{X}}_{1}\times{\mathcal{X}}_{2}\times\cdots\times{\mathcal{X}}_{n}\rightarrow\mathbb{R}}$
satisfy the bounded differences property with bounds ${\displaystyle
c_{1},c_{2},\dots,c_{n}}$, then, for any $\epsilon>0$

$\displaystyle\mathbb{P}(|f(X_{1},X_{2},\ldots,X_{n})-\mathbb{E}[f(X_{1},X_{2},\ldots,X_{n})]|\geq\epsilon)$
$\displaystyle\leq
2\exp\left(-{\frac{2\epsilon^{2}}{\sum_{i=1}^{n}c_{i}^{2}}}\right).$

In our particular case, let score of unlabeled sample $g(\mathbf{x})$ be
random variable, since $\frac{1}{1+e^{2}}\leq
h(\mathbf{x}^{+},\mathbf{x})\leq\frac{1}{1+e^{-2}}$, the function
$f:{\mathcal{X}}_{1}\times{\mathcal{X}}_{2}\times\cdots\times{\mathcal{X}}_{n}\rightarrow\frac{1}{N}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x})$
satisfies the bounded differences property with bounds ${\displaystyle
c_{1}=c_{2}=\dots=c_{n}}=\frac{1}{N}\frac{e^{2}-1}{e^{2}+1}$, yielding the
following bound:

$\displaystyle\mathbf{III}(\epsilon)$ $\displaystyle\leq$ $\displaystyle
2\exp\left(-\frac{N\epsilon^{2}}{2(e^{2}-1)^{2}}\right)$ (50)
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\frac{N\epsilon^{2}}{2e^{4}}\right).$ (51)

Similarly, let score of positive sample $g(\mathbf{x}^{\prime})$ be random
variable, the function
$f:{\mathcal{X}}_{1}\times{\mathcal{X}}_{2}\times\cdots\times{\mathcal{X}}_{m}\rightarrow\frac{\tau^{+}}{M}\sum_{\mathbf{x}}h(\mathbf{x}^{+},\mathbf{x}^{\prime})$
satisfies the bounded differences property with bounds ${\displaystyle
c_{1}=c_{2}=\dots=c_{m}}=\frac{\tau^{+}}{M}\frac{e^{2}-1}{e^{2}+1}$, yielding
the following bound:

$\displaystyle\mathbf{IV}(\epsilon)$ $\displaystyle\leq$ $\displaystyle
2\exp\left(-\frac{M\epsilon^{2}}{2(e^{2}-1)^{2}{\tau^{+}}^{2}}\right)$ (52)
$\displaystyle\leq$ $\displaystyle
2\exp\left(-\frac{M\epsilon^{2}}{2e^{4}{\tau^{+}}^{2}}\right).$ (53)

Inserting Eq (51) and Eq (53) back to Eq (47), we have

$\displaystyle\mathbb{P}(\triangle\geq\epsilon|\mathbf{x}^{+})\leq
2\exp\left(-\frac{N\epsilon^{2}}{2e^{4}}\right)+2\exp\left(-\frac{M\epsilon^{2}}{2e^{4}{\tau^{+}}^{2}}\right).$
(54)

Our objective is to bound the term
$|\mathcal{L}_{\text{DPL}}(g)-\hat{\mathcal{L}}_{\text{DPL}}(g)|$, to do this
we follow DCL [12] to push the absolute value inside the expectation by
Jensen’s inequality

$\displaystyle|\hat{\mathcal{L}}_{\textsc{dpl}}-\mathcal{L}_{\textsc{sup}}|$
(55) $\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{x}^{+}}\log\mathbf{I}-\mathbf{II}|$
$\displaystyle\leq$ $\displaystyle\mathbb{E}_{\mathbf{x}^{+}}\triangle$ (56)
$\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{x}^{+}}[\mathbb{E}_{\epsilon}[\triangle|\mathbf{x}^{+}]]$
(57) $\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{x}^{+}}\left[\int_{0}^{+\infty}\mathbb{P}(\triangle\geq\epsilon|\mathbf{x}^{+})d\epsilon\right]$
$\displaystyle\leq$
$\displaystyle\int_{0}^{+\infty}2\exp\left(-\frac{N\epsilon^{2}}{2e^{4}}\right)+2\exp\left(-\frac{M\epsilon^{2}}{2e^{4}{\tau^{+}}^{2}}\right)d\epsilon$
$\displaystyle=$ $\displaystyle
e^{2}\sqrt{\frac{2\pi}{N}}+e^{2}\tau^{+}\sqrt{\frac{2\pi}{M}}.$ (59)

The outer expectation in Eq. (0.3) disappears since the tail probably bound
holds uniformly for all fixed positive sample $\mathbf{x}^{+}$ [12].

##  V Experiment

###  V-A Experiment Settings

####  V-A1 Dataset

We conduct our experiments on five publicly available datasets:
MovieLens-100k, MovieLens-1M, Yahoo!-R3, Yelp2018 and Gowalla. These datasets
comprise users’ ratings on items using a discrete five-point grading system,
providing information on the items that the users have interacted with. The
first 3 datasets contain user ratings, we follow [9, 19, 20] to convert all
rated items to implicit feedback. For each dataset, we randomly allocate 20%
of the data as test data, with the remaining 80% used for training. Table I
presents a summary of the dataset statistics.

TABLE I: Dataset Statistics | users | items | interactions | training set | test set | density
---|---|---|---|---|---|---
MovieLens-100k | 943 | 1,682 | 100,000 | 80k | 20k | 0.06304
MovieLens-1M | 6,040 | 3,952 | 1,000,000 | 800k | 200k | 0.04189
Yahoo!-R3 | 5,400 | 1,000 | 182,000 | 146k | 36k | 0.03370
Yelp2018 | 31,668 | 38,048 | 1,561,406 | 1,249k | 312k | 0.00130
Gowalla | 29,858 | 40,981 | 1,027,370 | 821k | 205k | 0.00084

####  V-A2 Evaluation metric

In order to evaluate the performance of the recommendations, we adopt commonly
used metrics, namely precision (P), recall (R), and normalized discounted
cumulative gain (NDCG), to assess the Top-$K$ recommendations, where $K$ is
selected as 5, 10, and 20. For the sake of brevity, we assume that the
definitions of these metrics are widely known and do not provide them here.

####  V-A3 Experimental Setup

In our experimental setup, we consider two recommendation models: the classic
matrix factorization (MF)[21] and the more recent light graph convolution
network (LightGCN)[14]. The computations for the first three datasets were
conducted on a personal computer running the Windows 10 operating system with
a 2.1 GHz CPU, an RTX 1080Ti GPU, and 32 GB of RAM. The computations for the
last two datasets were performed on a cloud server running the Linux operating
system with a Xeon(R) Platinum 8358P CPU, an RTX A40 GPU, and 56GB of RAM. The
code and corresponding parameters have been released at:
https://github.com/liubin06/DPL for reproducibility.

####  V-A4 Baselines

  * •

BPR [9]: BPR introduces the pairwise learning approach based on maximum a
posteriori estimation for implicit collaborative filtering. BPR and NCE have
identical mathematical forms, but are described differently. Since implicit
feedback only involves positive data of interactions and unlabeled data, BPR
samples positives from the $(u,i)$ pairs that users interacted with, while the
negative examples are drawn from the unlabeled data $(u,j)$ that users have
not interacted with.

  * •

InfoNCE [7]: The InfoNCE loss is a popular loss function used in machine
learning, particularly in the context of representation learning.
Specifically, InfoNCE measures the similarity between a query sample
$\mathbf{x}^{+}$ and the set of negative samples
$\\{\mathbf{x}_{i}^{-}\\}_{i=1}^{N}$ and applying a softmax function:

$\displaystyle\mathcal{L}_{\text{InfoNCE}}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}^{+}\sim
p^{+}(\mathbf{x})\\\ \mathbf{x}_{i}^{-}\sim p^{-}(\mathbf{x})\end{subarray}}$
$\displaystyle\log\frac{\exp(g(\mathbf{x}^{+}))}{\exp(g(\mathbf{x}^{+}))+\sum_{i=1}^{N}\exp(g(\mathbf{x}_{i}^{-}))}$

InfoNCE can be seen as a generalization of Noise-Contrastive Estimation (NCE)
from one negative sample to N negative samples. In practice, since label of
negative samples are unavailable, $\mathbf{x}_{i}^{-}$ are typically sampled
from unlabeled samples.

  * •

DCL [12]: Due to the presence of false negatives in the unlabeled data, DCL
corrects the probability estimates to perform false negative debiasing.
Specifically, it proposes the estimator to replace the second term in the
denominator of the $\mathcal{L}_{\text{InfoNCE}}$:

$\displaystyle\mathcal{L}_{\text{DCL}}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\begin{subarray}{c}\mathbf{x}^{+}\sim
p^{+}(\mathbf{x})\\\ \mathbf{x}_{i}^{-}\sim
p^{-}(\mathbf{x})\end{subarray}}\log\frac{\exp(g(\mathbf{x}^{+}))}{\exp(g(\mathbf{x}^{+}))+Ng}$

where

$\displaystyle
g=\frac{1}{N\tau^{-}}(\sum_{i=1}^{N}\exp(g(\mathbf{x}_{i})-N\tau^{+}\cdot\frac{\sum_{j=1}^{K}\exp(g(\mathbf{x}^{+}_{j})}{K})$

The g estimator can be interpreted as the summation of scores of true negative
samples. Specifically, $N\tau^{+}$ estimates the number of false negative
samples, and $\frac{\sum_{j=1}^{K}\exp(g(\mathbf{x}^{+}_{j})}{K}$ estimates
the mean value of scores of $K$ false negative samples. Thus, the second term
inside the parentheses corresponds to the summation of scores of all false
negative samples among $N$ samples, while subtracting it from summation of N
unlabeled scores $\sum_{i=1}^{N}\exp(g(\mathbf{x}_{i})$ corresponds to the
summation of all true negative samples among $N$ randomly selected unlabeled
samples.

  * •

HCL [22]: Following the DCL debiasing framework, it also takes into
consideration of hard negative mining by up-weighting each randomly selected
unlabeled sample as follows.

$\displaystyle\omega_{i}^{\textsc{Hcl}}=\frac{g(\mathbf{x}^{+}_{j})^{\beta}}{\frac{1}{N}\sum_{j=1}^{N}g(\mathbf{x}^{+}_{j})^{\beta}}.$
(60)

where beta controls the hardness level for mining hard negatives. DCL is a
particular case of HCL with $\beta=0$.

###  V-B Experimental Results

####  V-B1 Recommendation Performance

TABLE II: Performance comparison of five learning algorithms on five datasets.

Dataset | CF Model | Learning Method | Top-5 | Top-10 | Top-20
---|---|---|---|---|---
Precision | Recall | NDCG | Precision | Recall | NDCG | Precision | Recall | NDCG
MovieLens-100k | MF | BPR | 0.3900 | 0.1301 | 0.4143 | 0.3363 | 0.2164 | 0.3967 | 0.2724 | 0.3298 | 0.3962
InfoNCE | 0.4168 | 0.1434 | 0.4458 | 0.3513 | 0.2291 | 0.4202 | 0.2835 | 0.3546 | 0.4207
DCL | 0.4081 | 0.1388 | 0.4324 | 0.3452 | 0.2266 | 0.4095 | 0.2793 | 0.3497 | 0.4118
HCL | 0.4263 | 0.1463 | 0.4539 | 0.3565 | 0.2323 | 0.426 | 0.2849 | 0.3564 | 0.4242
DPL(Proposed) | 0.4348 | 0.1523 | 0.4643 | 0.3635 | 0.2379 | 0.4356 | 0.2914 | 0.3588 | 0.4338
LightGCN | BPR | 0.3944 | 0.1231 | 0.4204 | 0.3346 | 0.2189 | 0.4017 | 0.2658 | 0.3281 | 0.3986
Info_NCE | 0.3924 | 0.1343 | 0.4209 | 0.3349 | 0.2183 | 0.4006 | 0.2679 | 0.3289 | 0.3976
DCL | 0.3962 | 0.1367 | 0.4243 | 0.3361 | 0.2194 | 0.4022 | 0.2695 | 0.3329 | 0.4006
HCL | 0.4197 | 0.1461 | 0.4501 | 0.3458 | 0.2256 | 0.4188 | 0.2802 | 0.3446 | 0.4182
DPL(proposed) | 0.4333 | 0.1486 | 0.4627 | 0.3596 | 0.2344 | 0.4324 | 0.2919 | 0.3585 | 0.4331
MovieLens-1M | MF | BPR | 0.3929 | 0.0922 | 0.4142 | 0.3411 | 0.152 | 0.3836 | 0.2839 | 0.237 | 0.368
InfoNCE | 0.4009 | 0.0934 | 0.4209 | 0.3472 | 0.1546 | 0.3894 | 0.289 | 0.2423 | 0.3731
DCL | 0.3820 | 0.0879 | 0.4003 | 0.3339 | 0.1478 | 0.3728 | 0.2821 | 0.2358 | 0.3605
HCL | 0.4112 | 0.0969 | 0.4317 | 0.3552 | 0.1585 | 0.3991 | 0.2959 | 0.2475 | 0.3825
DPL(proposed) | 0.4212 | 0.0998 | 0.4407 | 0.3624 | 0.1625 | 0.4071 | 0.2991 | 0.2518 | 0.3891
LightGCN | BPR | 0.3517 | 0.0739 | 0.3726 | 0.2997 | 0.1201 | 0.3385 | 0.2467 | 0.1884 | 0.3172
InfoNCE | 0.4121 | 0.0986 | 0.4386 | 0.359 | 0.1594 | 0.4041 | 0.2979 | 0.2482 | 0.3869
DCL | 0.4104 | 0.0982 | 0.4291 | 0.3544 | 0.1597 | 0.3977 | 0.2965 | 0.2511 | 0.3842
HCL | 0.4107 | 0.0948 | 0.4300 | 0.3514 | 0.1542 | 0.3950 | 0.2916 | 0.2413 | 0.3775
DPL(proposed) | 0.4217 | 0.1003 | 0.4429 | 0.3620 | 0.1625 | 0.1866 | 0.2989 | 0.2511 | 0.3896
Yahoo!-R3 | MF | BPR | 0.1417 | 0.1052 | 0.1587 | 0.1064 | 0.1573 | 0.1641 | 0.0768 | 0.2259 | 0.1913
Info_NCE | 0.1454 | 0.1083 | 0.1635 | 0.1091 | 0.1618 | 0.1692 | 0.079 | 0.2327 | 0.1974
DCL | 0.1429 | 0.1065 | 0.1615 | 0.1080 | 0.1601 | 0.1664 | 0.0786 | 0.2316 | 0.1952
HCL | 0.1460 | 0.1097 | 0.1638 | 0.1096 | 0.1628 | 0.1697 | 0.0792 | 0.2336 | 0.1976
DPL(proposed) | 0.1491 | 0.1091 | 0.1652 | 0.1108 | 0.1641 | 0.1712 | 0.0801 | 0.2351 | 0.2012
LightGCN | BPR | 0.1115 | 0.0838 | 0.01252 | 0.0881 | 0.1322 | 0.1346 | 0.0661 | 0.1976 | 0.1611
Info_NCE | 0.1456 | 0.1092 | 0.1642 | 0.1089 | 0.1622 | 0.1697 | 0.079 | 0.2333 | 0.1982
DCL | 0.1417 | 0.1074 | 0.1676 | 0.1099 | 0.1633 | 0.1719 | 0.0798 | 0.2354 | 0.2007
HCL | 0.1412 | 0.1139 | 0.1718 | 0.113 | 0.1683 | 0.1776 | 0.0812 | 0.2394 | 0.2059
DPL(proposed) | 0.1504 | 0.1111 | 0.1697 | 0.1131 | 0.1670 | 0.1757 | 0.0825 | 0.2412 | 0.2054
Yelp2018 | MF | BPR | 0.0398 | 0.0228 | 0.0435 | 0.0339 | 0.0389 | 0.0456 | 0.0284 | 0.065 | 0.0538
Info_NCE | 0.0429 | 0.0246 | 0.047 | 0.0365 | 0.0417 | 0.0491 | 0.0305 | 0.07 | 0.058
DCL | 0.0486 | 0.0278 | 0.0531 | 0.041 | 0.0466 | 0.0552 | 0.0342 | 0.0777 | 0.0648
HCL | 0.0535 | 0.0535 | 0.0586 | 0.0459 | 0.0541 | 0.0622 | 0.0383 | 0.0894 | 0.0736
DPL(proposed) | 0.0543 | 0.0325 | 0.0595 | 0.0463 | 0.0551 | 0.0630 | 0.0389 | 0.0914 | 0.0749
LightGCN | BPR | 0.0556 | 0.0330 | 0.0610 | 0.0473 | 0.0560 | 0.0644 | 0.0391 | 0.0914 | 0.0757
Info_NCE | 0.0553 | 0.0329 | 0.0607 | 0.0473 | 0.0558 | 0.0642 | 0.0390 | 0.0911 | 0.0754
DCL | 0.0559 | 0.0331 | 0.0612 | 0.0472 | 0.0557 | 0.0642 | 0.0391 | 0.0914 | 0.0756
HCL | 0.0563 | 0.0335 | 0.0617 | 0.0477 | 0.0564 | 0.0648 | 0.0393 | 0.0920 | 0.0760
DPL(proposed) | 0.0604 | 0.0364 | 0.0657 | 0.0513 | 0.0615 | 0.0696 | 0.0423 | 0.1003 | 0.0821
Gowalla | MF | BPR | 0.0728 | 0.0748 | 0.1000 | 0.0555 | 0.1116 | 0.1063 | 0.0414 | 0.1625 | 0.1209
Info_NCE | 0.0739 | 0.0757 | 0.1016 | 0.0560 | 0.1122 | 0.1076 | 0.0422 | 0.1650 | 0.1230
DCL | 0.0746 | 0.0769 | 0.1023 | 0.0568 | 0.1147 | 0.1088 | 0.0426 | 0.1664 | 0.1238
HCL | 0.0755 | 0.0774 | 0.1035 | 0.0574 | 0.1151 | 0.1098 | 0.0432 | 0.1693 | 0.1256
DPL(proposed) | 0.0815 | 0.0827 | 0.1100 | 0.0628 | 0.1243 | 0.1174 | 0.0473 | 0.1815 | 0.1340
LightGCN | BPR | 0.0735 | 0.0753 | 0.1007 | 0.0560 | 0.1119 | 0.1069 | 0.0419 | 0.1641 | 0.1218
Info_NCE | 0.0743 | 0.0760 | 0.1022 | 0.0566 | 0.1132 | 0.1084 | 0.0423 | 0.1649 | 0.1231
DCL | 0.0748 | 0.0763 | 0.1027 | 0.0569 | 0.1132 | 0.1088 | 0.0424 | 0.1656 | 0.1236
HCL | 0.0794 | 0.0804 | 0.1084 | 0.0608 | 0.1199 | 0.1147 | 0.0453 | 0.1740 | 0.1319
DPL(proposed) | 0.0867 | 0.0891 | 0.1164 | 0.0662 | 0.1329 | 0.1242 | 0.0494 | 0.1936 | 0.1417

The first observation from Table II is that DPL achieves the best performance.
Compared to BPR and InfoNCE without debiasing mechanisms, DCL shows
significant improvements, indicating the necessity of correcting biased
probability estimates in implicit feedback data. Compared to DCL and HCL with
debiasing mechanisms, DPL also achieves significant improvements, mainly due
to the advantage of DPL’s debiasing mechanism in the pairwise learning problem
setting. In the pairwise learning problem setting, where the number of
unlabeled samples is one and the true labels of the pairwise data
corresponding to user preference can be enumerated (see Fig 2), an unbiased
estimate of the probability that the user prefers positive examples over
negative examples can be obtained. However, in the case of N unlabeled
samples, according to the binomial theorem, there are $2^{N}$ possible
outcomes for the true label, making it difficult to enumerate every case.
Therefore, the debiasing mechanisms of DCL and HCL mainly rely on numerical
approximation.

The second observation is that as a special case of InfoNCE with a negative
sample size of 1, the BPR loss has a performance slightly inferior to that of
InfoNCE with N negative samples, especially on large and sparse datasets. This
is because a larger negative sample size N tighter lower bounds the mutual
information [7]. However, in PU datasets, a larger N is not always better
because a larger N usually result in larger gradient values to hard samples,
i.e., samples that are embedded closer to the anchor point (i.e., user
embedding) in the embedding space. If such a sample is a false negative, the
larger gradient value can seriously damage the model’s performance.

The third observation is that DCL, HCL, and DPL, which incorporate debiasing
mechanisms, generally outperform BPR and InfoNCE, which lack debiasing
mechanisms, highlighting the importance of debiasing on PU datasets,
especially on datasets with a high positive class prior. Moreover, HCL assigns
larger gradient values to negative samples by assigning higher weights to hard
samples with high scores, achieving good results through implicit hard
negative mining on top of debiasing. However, the corresponding hard negative
mining parameter should be carefully tuned to prevent false negatives from
harming model performance. This is because model performance will be harmed if
an unlabeled hard sample is a false negative, but model performance will
benefit from the hard sample if it is a true negative. This phenomenon is
referred to as the "exploration-and-exploitation trade-off"[17] in
collaborative filtering and the "uniformity-alignment dilemma" in computer
vision[23].

####  V-B2 Hyperparameter Analysis

Figure 4: The changes of the Precision@5 metric under different parameters on
the MovieLens-1M dataset. Generally, larger M leads to improved performance.
However, different situations occur for parameter N. A particular note is that
for matrix factorization models, a larger N should be set; for lightGCN
models, a smaller N value should be set, such as N=1.

Impact of M: The parameter M controls the number of additional positive
examples used to correct the sampling bias. When M = 0, there is no debiasing
mechanism. For the MF model, larger values of M and N usually lead to improved
performance, with a significant performance boost observed from M=0 to M=1,
highlighting the importance of using additional positive examples for
debiasing. However, when M and N exceed a small constant, performance no
longer improves. This is because the marginal gains in estimation accuracy
from further increasing M and N become limited, as the theoretical analysis
presented earlier.

Impact of N: The parameter N controls the number of negative examples used to
compute the PU probability Similar to the MF model, increasing M consistently
improves LightGCN performance. However, as the number of negative examples N
increases, the model performance unexpectedly decreases. We attribute this
non-intuitive result to the fact that a larger N reduces the gradient value of
hard negative samples. As Jensen’s inequality states,
$-\log(\frac{1}{N}\sum_{i=1}^{N}P_{n})\leq\frac{1}{N}\sum_{i=1}^{N}\log
P_{n}$; thus, the excessive value of N weakens the contribution of hard
negative samples to the learning algorithm. The theoretical analysis assumes
that the scores $g(\cdot)$ are independent and identically distributed
(i.i.d.) variables, and the estimation error decreases as N increases.
However, the aggregation mechanism of the graph neural network-based encoder
seriously affects the i.i.d. property of $g(\cdot)$ values. Therefore, we
recommend setting a small value of N for the LightGCN model.

Impact of positive class prior $\tau^{+}$: As $\tau^{+}$ increases, the model
performance of MF and LightGCN exhibits an inverted U-shaped curve with an
initial increase followed by a decrease. This is because setting a $\tau^{+}$
value that is too low or too high can lead to biased estimates of Formula 1. A
common method to set the $\tau^{+}$ value is to treat the number of observed
positive interactions $|\mathcal{D}^{+}|$ as a result of a Bernoulli trial,
which occurs a total of $|\mathcal{U}|\times|\mathcal{I}|$ times, and succeeds
in the number of interactions observed. The density
$|\mathcal{D}^{+}|/(|\mathcal{U}|\times|\mathcal{I}|)$ of the dataset can then
serve as a reference for setting the $\tau^{+}$ value. However, it should be
noted that the $\tau^{+}$ value set based on the dataset density is a biased
estimate that underestimates the true value, as all unobserved (u, i) pairs in
the data are treated as negative samples, leading to an underestimation of the
number of successes. We refer to [24, 25] for detailed discussion.

####  V-B3 DPL VS Negative Sampling

Negative sampling and loss correction represent two distinct technical
approaches in addressing the problem of sampling bias. The core idea of
negative sampling is to select hard negative samples and use them for model
training, which has demonstrated promising results. From a Bayesian
statistical perspective, negative sampling utilizes two types of information:
prior information, such as item category and popularity, which is static and
used for sampling negative samples that users do not prefer; and sample
information, such as scores and ranking positions, which is dynamic and
continuously adjusted during model training, and used for sampling hard
samples that are embedded close to the anchor embedding (higher scored). The
differences between various negative sampling algorithms lie in how they
utilize and process these two types of information. The latest Bayesian
negative sampling algorithm specifies the negative signal measure in terms of
posterior probability and proposes the theoretically optimal sampling rule,
which has achieved good results.We compare the performance of DPL and BNS in
Table III.

TABLE III: Performance comparison of five learning algorithms on five
datasets.

Dataset | CF Model | Method | Top-5 | Top-10 | Top-20
---|---|---|---|---|---
Precision | Recall | NDCG | Precision | Recall | NDCG | Precision | Recall | NDCG
MovieLens-100k | MF | BPR | 0.3900 | 0.1301 | 0.4143 | 0.3363 | 0.2164 | 0.3967 | 0.2724 | 0.3298 | 0.3962
BNS | 0.4205 | 0.1467 | 0.4558 | 0.3463 | 0.2290 | 0.4217 | 0.2762 | 0.3466 | 0.4176
DPL(Proposed) | 0.4348 | 0.1523 | 0.4643 | 0.3635 | 0.2379 | 0.4356 | 0.2914 | 0.3588 | 0.4338
DPL with Hard Samples | 0.4401 | 0.1579 | 0.4692 | 0.3713 | 0.2407 | 0.4395 | 0.2940 | 0.3592 | 0.4351
MovieLens-1M | MF | BPR | 0.3929 | 0.0922 | 0.4142 | 0.3411 | 0.152 | 0.3836 | 0.2839 | 0.237 | 0.368
BNS | 0.4207 | 0.1062 | 0.4324 | 0.3518 | 0.1703 | 0.4191 | 0.3045 | 0.2614 | 0.4002
DPL(proposed) | 0.4212 | 0.0998 | 0.4407 | 0.3624 | 0.1625 | 0.4071 | 0.2991 | 0.2518 | 0.3891
DPL with Hard Samples | 0.4251 | 0.1012 | 0.4412 | 0.3649 | 0.1701 | 0.4151 | 0.3012 | 0.2539 | 0.3922

Figure 5: Comparison of running time. The running time is tested on a personal
computer with a 2.1 GHz CPU, an RTX 1080Ti GPU, and 32 GB of RAM.

Performance: DPL achieved better performance on the MovieLens100k dataset and
performed similarly to BNS on the MovieLens1M dataset, demonstrating the
feasibility of using DPL as an alternative negative sampling algorithm based
on correction estimation. Furthermore, we found that training DPL with hard,
high-scored unlabeled samples can lead to some performance improvements. When
training DPL with difficult samples, a relatively large tao value needs to be
set. In Table III, $\tau^{+}$ were set to 0.3 and 0.25, respectively, which
are much higher than the density of the dataset itself. This is because
training DPL with hard samples increases the probability of the model
encountering false negative samples, which is equivalent to artificially
changing the positive class prior of training samples fed into the model.

Running Time: Figure 5 displays the running time of one epoch training for the
three algorithms on five datasets, with the CF model fixed as MF and the batch
size fixed as 1024. BNS involves computing the empirical distribution
function, and we implemented the in-batch approximation to save computational
costs when computing the empirical distribution function. As shown in Figure
5, the actual running time of DPL is only slightly longer than that of BPR,
which is consistent with the time complexity analysis presented earlier. Even
with the in-batch sample approximation to save computational costs, the
running time of BNS is still 3-5 times longer than that of BPR and DPL. This
is because dynamic negative sampling requires predicted scores to guide
negative sampling, but GPU-based batch computation requires fixed negative
samples before performing forward propagation to predict scores. As a result,
dynamic negative sampling is typically implemented by loading additional
negative samples as candidates into mini-batch data, leading to additional
computational costs. Additionally, some state-of-the-art dynamic negative
sampling algorithms require the ranking position of samples or the variance of
predicted scores [26] in the previous training epochs, which requires the
model to compute the predicted score of the entire user-item rating matrix
rather than just the score within the mini-batch data when performing forward
propagation, resulting in exponential time complexity. In summary, in
scenarios with rich side information, we recommend using negative sampling
algorithms that can flexibly combine prior information and model information.
In scenarios where there is no available side information for supervision, we
recommend using the DPL method for loss correction.

##  VI Related Work

###  VI-A Collaborative Filtering

Collaborative filtering was first formalized as a matrix completion problem
for predicting scores [21]. However, users typically only provide positive
feedback by indicating their preferences or interests through interactions,
resulting in binary values of 0 or 1 in the interaction matrix. BPR [9]
introduced pairwise learning from pairwise comparisons of positive and
negative item pairs to predict rankings. The core idea is to optimize the
model to score the positive item higher than the negative item, which is
reflected in the embedding space by pulling the positive item closer to the
user and pushing the negative item further away. Mathematically, BPR and NCE
[18] are equivalent in their formulation [5], but they are interpreted
differently. Based on BPR optimization criterion, a series of recommendation
models have been proposed, such as NGCF [15] and LightGCN [14], which have
achieved state-of-the-art performance. Inspired by the success in CV and NLP,
InfoNCE [7] has also been widely applied in collaborative filtering, which can
be viewed as a generalization of BPR from one negative sample to N negative
samples. Wu et al.[27] prove that optimizing the InfoNCE loss is consistent
with maximizing the Discounted Cumulative Gain (DCG) metric. Both BPR and
InfoNCE face the problem of false negatives in unlabeled data, resulting in
biased user-item representations[16, 28, 29, 30, 31].

To address this issue, negative sampling algorithms have been extensively
studied, such as graph-based negative sampling [32, 33, 34, 35], and side or
prior information-based negative sampling [36, 37, 38, 39]. We group them into
two categories. The first kind is static negative sampling [9, 40, 41, 42,
15], which adopts a fixed sampling distribution, such as uniform sampling. The
second kind is dynamic negative sampling [20, 19, 32, 33]. Algorithms of this
kind favor negative instances with representations more similar to those of
positive instances in the embedding space, for example, by selecting higher
scored or higher ranked instances [20, 19]. However, they are more likely to
suffer from the negative problem [16, 43, 44]. A novel class of methods
generates virtual hard negatives from multiple unlabeled instances. These
methods can be viewed as a generalization of the InfoNCE loss since the scores
of the synthesized virtual negative samples are a function of the embeddings
of unlabeled samples. For example, Huang et al.[29] propose to synthesize
virtual hard negatives by hop mixing embeddings. Jun et al.[45] and Park et
al. [28] design generative adversarial neural networks to generate virtual
hard negatives.

###  VI-B Contrastive Learning

Contrastive learning is based on the "learn-to-compare" paradigm [18, 9],
which discriminates between positive and negative samples to avoid
reconstructing pixel-level information of data [7]. While the representation
encoder $f$ and similarity measure may vary across different domains, such as
collaborative filtering [9, 14, 15] and computer vision tasks [46, 8, 47],
they share the common idea of pulling positive samples closer to the anchor
point while pushing the negative samples apart to train $f$ by optimizing a
contrastive loss [3], such as BPR loss [9], NCE loss [18], InfoNCE loss [7],
Infomax loss [48], asymptotic contrastive loss [3], among others. Supervised
contrastive learning has achieved remarkable success in various domains [49,
50], but it heavily relies on manually labeled datasets [5]. Self-supervised
contrastive learning [13, 51, 8, 49, 52] has been extensively studied for its
advantage in learning representations without requiring supervised data and
has been shown to benefit a wide range of downstream tasks [5, 53, 54, 55, 56,
57]. In self-supervised contrastive learning, positive samples $x^{+}$ are
obtained by applying a semantic-invariant operation on an anchor $x$ with
heavy data augmentation, while negative samples $x^{-}$ are drawn from
unlabeled data, which introduces the false negative problem and can lead to
incorrect encoder training. This problem is related to the classic positive-
unlabeled (PU) learning [10, 58, 11, 59]. Existing empirical risk rewriting
based methods for point-wise losses cannot be directly applied to contrastive
loss. In recent years, several estimators that are consistent with supervised
contrastive loss have been proposed, such as DCL [12], HCL [22], and BCL [60].
In particular, BCL proposes a posterior probability estimation of unlabeled
samples being true negatives.

##  VII Conclusion

In this paper, we focus on addressing the problem of sampling bias from
positive-unlabeled implicit feedback data, but we adopt a different technical
approach from explicit negative sampling. Specifically, we propose a
correction for sampling bias from implicit feedback that yields a modified
loss for pairwise learning called debiased pairwise loss (DPL). The key idea
underlying DPL is to correct the biased probability estimates that result from
false negatives, thereby correcting the gradients to approximate those of
fully supervised data. The proposed objective is easy to implement and does
not require additional side information for supervision or excessive storage
and computational overhead. In our future work, we will further explore the
design of hard negative mining mechanisms on top of debiased pairwise loss.

## References

  * [1] M. U. Gutmann and A. Hyvärinen, “Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.” _Journal of machine learning research_ , vol. 13, no. 2, 2012\.
  * [2] N. Ailon and M. Mohri, “Preference-based learning to rank,” _Machine Learning_ , vol. 80, no. 2-3, pp. 189–211, 2010.
  * [3] T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” in _International Conference on Machine Learning_ , 2020, pp. 9929–9939.
  * [4] D. McFadden, “Conditional logit analysis of qualitative choice behavior,” _Frontiers in Econometrics_ , 1974.
  * [5] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [6]
  * [7] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [8] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _CVPR_ , 2020, pp. 9729–9738.
  * [9] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “Bpr: Bayesian personalized ranking from implicit feedback,” in _UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009_ , 2009, pp. 452–461.
  * [10] B. Jessa and D. Jesse, “Learning from positive and unlabeled data: a survey,” _Machine Learning_ , vol. 109, p. 719–760, 2020.
  * [11] M. C. Du Plessis, G. Niu, and M. Sugiyama, “Analysis of learning from positive and unlabeled data,” in _NeurIPS_ , 2014.
  * [12] C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” in _NeurIPS_ , 2020, pp. 8765–8775.
  * [13] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _ICML_ , 2020, pp. 1597–1607.
  * [14] H. Xiangnan, D. Kuan, W. Xiang, L. Yan, Z. Yongdong, and W. Meng, “Lightgcn: Simplifying and powering graph convolution network for recommendation.” in _SIGIR_ , 2020, p. 10.
  * [15] X. Wang, X. He, M. Wang, F. Feng, and T. S. Chua, “Neural graph collaborative filtering,” in _SIGIR ’2019_ , Proceedings of the 42nd International ACM SIGIR Conference, pp. 2344–2353.
  * [16] J. Ding, Y. Quan, Q. Yao, Y. Li, and D. Jin, “Simplify and robustify negative sampling for implicit collaborative filtering,” in _NeurIPS_ , 2020.
  * [17] B. Liu and B. Wang, “Bayesian negative sampling for recommendation,” in _39th IEEE International Conference on Data Engineering (ICDE)_ , 2023.
  * [18] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , 2010, pp. 297–304.
  * [19] W. Zhang, T. Chen, J. Wang, and Y. Yu, “Optimizing top-n collaborative filtering via dynamic negative item sampling,” in _Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2013, p. 785–788.
  * [20] S. Rendle and C. Freudenthaler, “Improving pairwise learning for item recommendation from implicit feedback,” in _Proceedings of the 7th ACM international conference on Web Search and Data Mining_ , 2014, pp. 273–282.
  * [21] Y. Koren, R. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” _Computer_ , vol. 42, no. 8, pp. 30–37, 2009.
  * [22] J. Robinson, C. Ching-Yao, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” in _ICLR_ , 2021.
  * [23] F. Wang and H. Liu, “Understanding the behaviour of contrastive loss,” in _CVPR_ , 2021, pp. 2495–2504.
  * [24] S. Jain, M. White, and P. Radivojac, “Estimating the class prior and posterior from noisy positives and unlabeled data,” in _Advances in Neural Information Processing Systems_ , 2016.
  * [25] M. Christoffel, G. Niu, and M. Sugiyama, “Class-prior estimation for learning from positive and unlabeled data,” in _Asian Conference on Machine Learning_ , 2016, pp. 221–236.
  * [26] J. Ding, Y. Quan, Q. Yao, Y. Li, and D. Jin, “Simplify and robustify negative sampling for implicit collaborative filtering,” in _NeurIPS_ , 2019.
  * [27] J. Wu, X. Wang, X. Gao, J. Chen, H. Fu, T. Qiu, and X. He, “On the effectiveness of sampled softmax loss for item recommendation,” _arXiv preprint arXiv:2201.02327_ , 2022.
  * [28] D. H. Park and Y. Chang, “Adversarial sampling and training for semi-supervised information retrieval,” in _WWW_ , 2019, p. 1443–1453.
  * [29] T. Huang, Y. Dong, M. Ding, Z. Yang, W. Feng, X. Wang, and J. Tang, “Mixgcf: An improved training method for graph neural network-based recommender systems,” in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, p. 665–674.
  * [30] J. Ding, Y. Quan, X. He, Y. Li, and D. Jin, “Reinforced negative sampling for recommendation with exposure data,” in _IJCAI_ , 2019, pp. 2230–2236.
  * [31] Z. Yang, M. Ding, C. Zhou, H. Yang, J. Zhou, and J. Tang, “Understanding negative sampling in graph representation learning,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1666–1676.
  * [32] X. Wang, Y. Xu, X. He, Y. Cao, M. Wang, and Chua, “Reinforced negative sampling over knowledge graph for recommendation,” in _WWW_ , 2020, pp. 99–109.
  * [33] J. Chen, C. Wang, S. Zhou, Q. Shi, Y. Feng, and C. Chen, “Samwalker: Social recommendation with informative sampling strategy,” in _WWW_ , 2019, pp. 228–239.
  * [34] C. Wang, J. Chen, S. Zhou, Q. Shi, Y. Feng, and C. Chen, “Samwalker++: recommendation with informative sampling strategy,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [35] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in _KDD_ , 2018, pp. 974–983.
  * [36] F. Yuan, J. M. Jose, G. Guo, L. Chen, H. Yu, and R. S. Alkhawaldeh, “Joint geo-spatial preference and pairwise ranking for point-of-interest recommendation,” in _2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI)_ , 2016, pp. 46–53.
  * [37] W. Liu, Z.-J. Wang, B. Yao, and J. Yin, “Geo-alm: Poi recommendation by fusing geographical information and adversarial learning mechanism.” in _IJCAI_ , 2019, pp. 1807–1813.
  * [38] J. Ding, Y. Quan, X. He, Y. Li, and D. Jin, “Reinforced negative sampling for recommendation with exposure data,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 2019, pp. 2230–2236.
  * [39] J. Ding, F. Feng, X. He, G. Yu, Y. Li, and D. Jin, “An improved sampler for bayesian personalized ranking by leveraging view data,” in _Companion Proceedings of the The Web Conference 2018_ , 2018, p. 13–14.
  * [40] T. Chen, Y. Sun, Y. Shi, and L. Hong, “On sampling strategies for neural network-based collaborative filtering,” in _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2017, p. 767–776.
  * [41] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Proceedings of the 26th International Conference on Neural Information Processing Systems_ , 2013, p. 3111–3119.
  * [42] W. Pan and L. Chen, “Gbpr: Group preference based bayesian personalized ranking for one-class collaborative filterin,” in _Twenty-Third International Joint Conference on Artificial Intelligence_ , 2013.
  * [43] X. Qin, N. Sheikh, B. Reinwald, and L. Wu, “Relation-aware graph attention model with adaptive self-adversarial training,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021, pp. 9368–9376.
  * [44] H. Zhao, X. Yang, Z. Wang, E. Yang, and C. Deng, “Graph debiased contrastive learning with joint representation clustering,” in _IJCAI_ , 2021, pp. 3434–3440.
  * [45] J. Wang, L. Yu, W. Zhang, Y. Gong, Y. Xu, B. Wang, P. Zhang, and D. Zhang, “Irgan: A minimax game for unifying generative and discriminative information retrieval models,” in _SIGIR_ , 2017, p. 515–524.
  * [46] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [47] A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with convolutional neural networks,” in _NeurIPS_ , 2014, p. 766–774.
  * [48] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [49] O. Henaff, “Data-efficient image recognition with contrastive predictive coding,” in _ICML_ , 2020, pp. 4182–4192.
  * [50] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” in _NeurIPS_ , 2020, pp. 18 661–18 673.
  * [51] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, “Big self-supervised models are strong semi-supervised learners,” _NeurIPS_ , vol. 33, pp. 22 243–22 255, 2020.
  * [52] L. Xu, J. Lian, W. X. Zhao, M. Gong, L. Shou, D. Jiang, X. Xie, and J.-R. Wen, “Negative sampling for contrastive representation learning: A review,” _arXiv preprint arXiv:2206.00212_ , 2022.
  * [53] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” 2019.
  * [54] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [55] J. Huang, Q. Dong, S. Gong, and X. Zhu, “Unsupervised deep learning by neighbourhood discovery,” in _ICML_ , 2019, pp. 2849–2858.
  * [56] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _CVPR_ , 2018, pp. 3733–3742.
  * [57] C. Zhuang, A. L. Zhai, and D. Yamins, “Local aggregation for unsupervised learning of visual embeddings,” in _CVPR_ , 2019, pp. 6002–6012.
  * [58] M. Du Plessis, G. Niu, and M. Sugiyama, “Convex formulation for learning from positive and unlabeled data,” in _International conference on machine learning_. PMLR, 2015, pp. 1386–1394.
  * [59] R. Kiryo, G. Niu, M. C. Du Plessis, and M. Sugiyama, “Positive-unlabeled learning with non-negative risk estimator,” in _NeurIPS_ , 2017.
  * [60] B. Liu and B. Wang, “Bayesian self-supervised contrastive learning,” _arXiv:2301.11673_ , 2023.