Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate over- fitting. This paper considers the related question of “membership inference”, where the goal is to determine if an image was used during training. We con- sider membership tests over either ensembles of samples or individual samples.
 First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time. Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples. To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.
 The widespread adoption of convolutional neural networks (LeCun et al., 1990 ) (ConvNets) for most recognition tasks, was triggered by the work of Krizhevsky et al. (2012) in image classification and subsequent deep architectures BID13 He et al., 2016) . Several works have analyzed these architectures from different perspectives. BID22 have proposed DeconvNet to vizualize filter activations. Lenc & Vedaldi (2015) analyze their equivariance. Mahendran & Vedaldi (2015) show how to invert them and synthetize images maximizing the response of different classes. BID19 analyze the image priors implicitly defined by ConvNets.All these works increase our understanding of ConvNets, but the complex issue of overfitting and its relationship to optimization are still not fully understood. Several strategies are routinely used to avoid overfitting, such as 2 -regularization through weight decay (Krogh & Hertz, 1991) , dropout BID14 , and importantly, data augmentation BID5 Dwibedi et al., 2017; Paulin et al., 2014 ). Yet few works BID21 have analyzed the interplay of overfitting and memorization of training images in high-capacity classification architectures. Specifically, we are not aware of such an analysis for a modern ConvNet such as ResNet-101 learned on Imagenet.In this paper, we consider the privacy issue of membership inference, i.e., we aim at determining if a specific image or group of images was used to train a model. This question is important to protect both the privacy and intellectual property associated with images. For ConvNets, the privacy issue was recently considered by BID21 for the small MNIST and CIFAR datasets. The authors evidence the close relationship between overfitting and privacy of training images. This is reminiscent of prior membership inference attacks, which employ the output of the classifier associated with a particular example to determine whether it was used during training or not (Shokri et al., 2017) . This is related to BID17 , who showed that a classifier can determine with high accuracy if an image comes from a dataset or another by exploiting the bias inherent to datasets. We discuss this relationship and show that we can detect whether a given network has been trained on some of the validation images. This has a concrete application for machine-learning benchmarks: scores are often reported on a validation set with public labels, allowing a malicious or gawky competitor to artificially inflate the accuracy by training on validation images. Our test detects if it is the case, even if only part of the validation set is leaked to the training set.Finally, we propose a new setting for membership inference that only considers intermediary layers of a network, thus extending membership inference to transferred and fine-tuned networks, that have become ubiquitous. Our membership inference does not require the last layer(s) of the original ConvNet to perform the test. This is important because, in many contexts, image recognition systems are built upon a trunk trained on a dataset and then fine-tuned for another task. Examples include Mask-RCNN (He et al., 2017) and models used for fine-grained recognition (Hariharan & Girshick, 2017) . In both cases there are not enough training samples to train a full network: only the last layers of the networks are fine-tuned. In summary, our paper makes the following contributions:• A simple statistical test to detect the "signature" of a dataset in a trained convnet, and to detect if validation images where used to train the model (leakage).• A membership inference test that detects if an image was used to train the trunk of a network. To our knowledge, it is the first work on membership inference that attacks intermediate layers.The paper is organized as follows. Section 2 reviews related work. Section 3 formally introduces the problem. Section 4 considers the problem of determining if a particular dataset, e.g., the validation set, was used during training. Section 5 focuses on detecting if a particular image has been used for training without accessing the network's output layer. We have investigated the memorization capabilities of neural networks from different perspectives. Our experiments show that state-of-the-art networks can remember a large number of images and distinguish them from unseen images. We have analyzed networks specifically trained to remember a set of images and the factors influencing their memorizing and convergence capabilities. It is possible to determine whether an image set was used at training time, even with full data augmentation. On the contrary, the accuracy of determining if a single image was used is low when considering full data augmentation on a large training set such as Imagenet. This implies that data augmentation is an effective privacy-preserving method. Our last contribution is a method that detects training images better than chance even with no access to the last layers, under limited data augmentation.Final remark: The curious reader may have noticed that our title echoes the one of a previous user study BID10 , in which the authors discussed the feasibility of authenticating humans by their capabilities to recognize a set of images.
We analyze the memorization properties by a convnet of the training set and propose several use-cases where we can extract some information about the training set.  Illuminates the generalization/memorization properties of large and deep ConvNets and tries to develop procedures related to identifying whether an input to a trained ConvNet has actually been used to train the network.