Pointwise localization allows more precise localization and accurate interpretability, compared to bounding box, in applications where objects are highly unstructured such as in medical domain. In this work, we focus on  weakly supervised localization (WSL) where a model is trained to classify an image and localize regions of interest at pixel-level using only global image annotation. Typical convolutional attentions maps are prune to high false positive regions. To alleviate this issue, we propose a new deep learning method for WSL, composed of a localizer and a classifier, where the localizer is constrained to determine relevant and irrelevant regions using conditional entropy (CE) with the aim to reduce false positive regions. Experimental results on a public medical dataset and two natural datasets, using Dice index, show that, compared to state of the art WSL methods, our proposal can provide significant improvements in terms of image-level classification and pixel-level localization (low false positive) with robustness to overfitting. A public reproducible PyTorch implementation is provided. Pointwise localization is an important task for image understanding, as it provides crucial clues to challenging visual recognition problems, such as semantic segmentation, besides being an essential and precise visual interpretability tool. Deep learning methods, and particularly convolutional neural networks (CNNs), are driving recent progress in these tasks. Nevertheless, despite their remarkable performance, their training requires large amounts of labeled data, which is time consuming and prone to observer variability. To overcome this limitation, weakly supervised learning (WSL) has emerged recently as a surrogate for extensive annotations of training data (Zhou, 2017) . WSL involves scenarios where training is performed with inexact or uncertain supervision. In the context of pointwise localization or semantic segmentation, weak supervision typically comes in the form of image level tags (Kervadec et al., 2019; Kim et al., 2017; Pathak et al., 2015; Teh et al., 2016; Wei et al., 2017) , scribbles (Lin et al., 2016; Tang et al., 2018) or bounding boxes (Khoreva et al., 2017) . Current state-of-the-art WSL methods rely heavily on pixelwise activation maps produced by a CNN classifier at the image level, thereby localizing regions of interest (Zhou et al., 2016) . Furthermore, this can be used as an interpretation of the model's decision (Zhang & Zhu, 2018) . The recent literature abounds of WSL works that relax the need of dense and prohibitively time consuming pixel-level annotations (Rony et al., 2019) . Bottom-up methods rely on the input signal to locate regions of interest, including spatial pooling techniques over activation maps (Durand et al., 2017; Oquab et al., 2015; Sun et al., 2016; Zhang et al., 2018b; Zhou et al., 2016) , multi-instance learning (Ilse et al., 2018) and attend-and-erase based methods (Kim et al., 2017; Li et al., 2018; Pathak et al., 2015; Singh & Lee, 2017; Wei et al., 2017) . While these methods provide pointwise localization, the models in (Bilen & Vedaldi, 2016; Kantorov et al., 2016; Shen et al., 2018; Tang et al., 2017; Wan et al., 2018 ) predict a bounding box instead, i.e., perform weakly supervised object detection. Inspired by human visual attention, top-down methods rely on the input signal and a selective backward signal to determine the corresponding region of interest. This includes special feedback layers (Cao et al., 2015) , backpropagation error (Zhang et al., 2018a) and Grad-CAM (Chattopadhyay et al., 2018; Selvaraju et al., 2017) . In many applications, such as in medical imaging, region localization may require high precision such as cells, boundaries, and organs localization; regions that have an unstructured shape, and different scale that a bounding box may not be able to localize precisely. In such cases, a pointwise localization can be more suitable. The illustrative example in Fig.1 (bottom row) shows a typical case where using a bounding box to localize the glands is clearly problematic. This motivates us to consider predicting a mask instead of a bounding box. Consequently, our latter choice of evaluation datasets is constrained by the availability of both global image annotation for training and pixel-level annotation for evaluation. In this work, we focus on the case where there is one object of interest in the image. Often, within an agnostic-class setup, input image contains the object of interest among other irrelevant parts (noise, background). Most the aforementioned WSL methods do not consider such prior, and feed the entire image to the model. In such scenario, (Wan et al., 2018) argue that there is an inconsistency between the classification loss and the task of WSL; and that typically the optimization may reach sub-optimal solutions with considerable randomness in them, leading to high false positive localization. False positive localization is aggravated when a class appears in different and random shape/structure, or may have relatively similar texture/color to the irrelevant parts driving the model to confuse between both parts. False positive regions can be problematic in critical domains such as medical applications where interpretability plays a central role in trusting and understanding an algorithm's prediction. To address this important issue, and motivated by the importance of using prior knowledge in learning to alleviate overfitting when training using few samples (Belharbi et al., 2017; Krupka & Tishby, 2007; Mitchell, 1980; Yu et al., 2007) , we propose to use the aforementioned prior in order to favorite models with low false positive localization. To this end, we constrain the model to learn to localize both relevant and irrelevant regions simultaneously in an end-to-end manner within a WSL scenario, where only image-level labels are used for training. We model the relevant (discriminative) regions as the complement of the irrelevant (non-discriminative) regions (Fig.1) . Our model is composed of two sub-models: (1) a localizer that aims to localize both types of regions by predicting a latent mask, (2) and a classifier that aims to classify the visible content of the input image through the latent mask. The localizer is driven through CE (Cover & Thomas, 2006) to simultaneously identify (1) relevant regions where the classifier has high confidence with respect to the image label, (2) and irrelevant regions where the classifier is being unable to decide which image label to assign. This modeling allows the discriminative regions to pop out and be used to assign the corresponding image label, while suppressing non-discriminative areas, leading to more reliable predictions. In order to localize complete discriminative regions, we extend our proposal by training the localizer to recursively erase discriminative parts during training only. To this end, we propose a consistent recursive erasing algorithm that we incorporate within the backpropagation. At each recursion, and within the backpropagation, the algorithm localizes the most discriminative region; stores it; then erases it from the input image. At the end of the final recursion, the model has gathered a large extent of the object of interest that is fed next to the classifier. Thus, our model is driven to localize complete relevant regions while discarding irrelevant regions, resulting in more reliable region localization. Moreover, since the discriminative parts are allowed to be extended over different instances, our proposal handles multi-instances intrinsically. The main contribution of this paper is a new deep learning framework for WSL at pixel level. The framework is composed of two sequential sub-networks where the first one localizes regions of interest, whereas the second classifies them. Based on CE, the end-to-end training of the framework allows to incorporate prior knowledge that, an image is more likely to contain relevant and irrelevant regions. Throughout the CE measured at the classifier level, the localizer is driven to localize relevant regions (with low CE) and irrelevant regions (with high CE). Such localization is achieved with the main goal of providing a more interpretable and reliable regions of interest with low false positive localization. This paper also contributes a consistent recursive erasing algorithm that is incorporated within backpropagation, along with a practical implementation in order to obtain complete discriminative regions. Finally, we conduct an extensive series of experiments on three public image datasets (medical and natural), where the results show the effectiveness of the proposed approach in terms of pointwise localization (measured with Dice index) while maintaining competitive accuracy for image-level classification. In this work, we present a novel approach for WSL at pixel-level where we impose learning relevant and irrelevant regions within the model with the aim to reduce false positive localization. Evaluated on three datasets, and compared to state of the art WSL methods, our approach shows its effectiveness in accurately localizing regions of interest with low false positive while maintaining a competitive classification error. This makes our approach more reliable in term of interpetability. As future work, we consider extending our approach to handle multiple classes within the image. Different constraints can be applied over the predicted mask, such as texture properties, shape, or other region constraints. Predicting bounding boxes instead of heat maps is considered as well since they can be more suitable in some applications where pixel-level accuracy is not required. Our recursive erasing algorithm can be further improved by using a memory-like mechanism that provides spatial information to prevent forgetting the previously spotted regions and promote localizing the entire region (Sec.B.3).
A deep learning method for weakly-supervised pointwise localization that learns using image-level label only. It relies on conditional entropy to localize relevant and irrelevant regions aiming to minimize false positive regions. This work explores the problem of WSL using a novel design of regularization terms and a recursive erasing algorithm. This paper presents a new weakly supervised approach for learning object segmentation with image-level class labels.