In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources. Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities. We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework. Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets. Recent advances in deep learning have allowed promising improvement in developing various stateof-the-art neural ranking models in the information retrieval (IR) community BID8 BID17 BID12 BID9 BID13 . Similar achievement has been seen in the reading comprehension (RC) community using neural passage ranking (PR) models for answer selection tasks BID19 BID16 BID10 . Most of these neural ranking models, however, require a large amount of training data. As such, we have seen the progress of deep neural ranking models is coming along with the development of several large-scale datasets in both IR and RC communities, e.g. BID0 BID7 BID6 BID3 . Admittedly, creating hand-labeled ranking datasets is very expensive in both human labor and time.To overcome this issue, one strategy is to utilize weak supervision to replace human annotators. Usually we can cheaply obtain large amount of low-quality labels from various sources, such as prior knowledge, domain expertise, human heuristics or even pretrained models. The idea of weak supervision is to extract signals from the noisy labels to train our model. BID4 first applied weak supervision technique to train deep neural ranking models. They show that the neural ranking models trained on labels solely generated from BM25 scores can remarkably outperform the BM25 baseline in IR tasks. BID11 further investigated this approach by using external news corpus for training.In this work, we focus on the setting where queries and their associated candidate passages are given but no relevance judgment is available. Instead of solely relying on the labels from single source (BM25 score), we propose to leverage the weak supervision signals from diverse sources. BID14 proposed a general data programming framework to create data and train models in a weakly supervised manner. To tailor to the ranking tasks, instead of generating a ranked list of passages for each query, we generate binary labels for each query-passage pair. In our neural ranking models, we focus on BERT-based ranking model BID5 (architecture shown in FIG0 ), which achieves new state-of-the-art performance on two public benchmark datasets with full supervision. The contributions of this work are in two fold: (a) we propose a simple data programming framework for ranking tasks; (b) we train a BERT ranking model using our framework, by considering two simple sources of weak supervision signals, unsupervised ranking methods (BM25 and TF-IDF scores) and unsupervised semantic feature representation, we show our model outperforms BM25 baseline by a large margin (around 20% relative improvement in top-1 accuracy on average) and the previous state-of-the-art performance (around 10% relative improvement in top-1 accuracy on average) on three datasets without using ground-truth training labels. In this work, we proposed a simple weak supervision pipeline for neural ranking models based on the data programming paradigm. In particular, we also proposed a new PR model based on BERT, which achieves new SOTA results. In our experiments on different datasets, our weakly supervised BERT-PR model outperforms the BM25 baseline by a large margin and remarkably, even beats the previous SOTA performances with full supervision on two datasets. Further research can be done on how to better aggregate pseudo ranking labels. In our pipeline we reduce the ranking labels into binary labels of relevance of query-passage pairs, which may result in loss of useful information. It would be interesting to design generative models on the ranking labels directly.
We propose a weak supervision training pipeline based on the data programming framework for ranking tasks, in which we train a BERT-base ranking model and establish the new SOTA. The authors propose a combination of BERT and the weak supervision framework to tackle the problem of passage ranking, obtaining results better than the fully supervised state-of-the-art.