Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel- or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain. During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance. However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples. In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation. To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model. To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task. On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of our framework. Recent advances in deep learning have been successful in delivering the state-of-the-art (SOTA) performance in a variety of areas including computer vision, nature language processing, etc. Not only do advanced network architecture designs and better optimization techniques contribute to the success, but the availability of large annotated datasets (e.g. ImageNet (Deng et al., 2009) , MS COCO (Lin et al., 2014) , Cityscapes (Cordts et al., 2016) ) also plays an important role. However, it is never an easy task to curate such datasets. Collecting unlabeled data and the subsequent annotating process are both expensive and time-consuming. In particular, for some applications such as medical imaging, the annotation is limited by the available resources of expert analysts and data protection issues, which makes it even more challenging for curating large datasets. For example, it takes hours for an experienced radiologist to segment the brain tumors on medical images for even just one case. On the contrary to supervised deep learning, human beings are capable of learning a new behaviour or concept through the most typical cases rather than accumulative learning for a lot of cases. Intuitively, we may ask: Is it really necessary to train a deep neural network with massive samples? Are we able to select a subset of most representative samples for network training which can save the annotation cost, improve data efficiency and lead to an at least equivalent or even better model? To the best of our knowledge, this is a less explored domain in deep learning and relevant applications, where a lot of efforts have been put into optimizing the network designs. Rather than improving the performance of a neural network given a curated training set, here we are more interested in how annotated samples can be more efficiently utilized to reach a level of performance. We consider such property as 'data efficiency', namely how efficient a learning paradigm utilizes annotated samples to achieve a pre-defined performance measure. In this paper, we propose a model state-aware framework for data-efficient deep representation learning, illustrated in Figure 1 . The main idea is to mine 'harder' training samples progressively on the data manifold according to the current parameter state of a network until a certain criteria is fulfilled Figure 1 : The general pipeline of proposed framework. The preparation stage is located at the top left corner which represents the training of a variational auto-encoder (VAE) using unannotated samples. The main stage is located within the dashed rectangle, where the decoder (generator) as well as its latent space are used for mining hard training samples according to the error information propagated backward via the target model and decoder (generator). Each proposed sample will be annotated by the labeling tool. (e.g. size of training dataset or performance on validation dataset). The harder samples with respect to a given network state are defined as those yielding higher loss, which are estimated through backpropagation (Hinton et al., 2006) . To be able to select plausible harder samples, a generative model is employed for embedding data into a low-dimensional latent space with better compactness and smoothness. In particular, we investigate two sampling strategies in the latent space, namely sampling by nearest neighbor (SNN) and sampling by interpolation (SI) for different applications. The data efficiency of our framework is evaluated on three datasets, including MNIST and CIFAR-10 for image classification tasks, as well as a medical image set IVUS for biophysical simulation task. There are three major contributions of this work: 1. A general and novel framework is proposed for model state-aware sampling and dataefficient deep representation learning, which can be used in a variety of scenarios with high annotating cost. 2. Unlike previous studies (Sener & Savarese, 2017; Peiyun et al., 2019) , a generative model is introduced to propose informative training samples. Two latent space sampling strategies are investigated and compared. 3. The framework is not only applicable for sampling on an existing dataset, but it also allows suggestive annotation and synthesizing new samples. We demonstrate the latter in a biophysical simulation task, where artificial samples are synthesized from the latent space. 2 RELATED WORK In our framework, an annotating system (i.e. labeling tool or original dataset) is integrated into the training process and used in an active manner. Based on the current model state, more informative samples proposed by a generator are annotated online and appended to the current train set for further training. This closed-loop design makes the most use of the annotating system, which would be very useful in scenarios with high annotation cost, e.g. medical image segmentation. From the performance curves in Figure 3 , we observed an immediate drop when fresh samples were fed into the neural work. But the performance rebounded to a higher level as the neural network learned the information carried by these samples. Compared to the random sampling, our hardness-aware sampling resulted in a deeper drop followed by a higher rebound, indicating that more informative sample were mined. We proposed a model state-aware framework for efficient annotating and learning. Hard samples from the data manifold are mined progressively in the low-dimensional latent space. It is obvious that the proposed framework can not be only generalized to existing machine learning applications, but also those realistic scenarios as long as a labeling tool is available. Imaged-based biomechanical analysis of coronay plaques were performed following the procedure described in (Teng et al., 2014) . The workflow is wrapped into a Phython package named 'VasFEM' as a labeling tool, which is available upon request. The input to the labeling system is a segmentation mask of plaque and the output is the corresponding structural stress map with the same resolution. The material of plaque is assumed to be incompressible and non-linear which is described by the modified Mooney-Rivlin strain energy density function: whereĪ 1 = J −2/3 I 1 with I 1 being the first invariant of the unimodular component of the left Cauchy-Green deformation tensor. J = det(F) and F is the deformation gradient. κ is the Lagrangian multiplier for the incompressibility. c 1 = 0.138 kPa, D 1 = 3.833 kPa and D 2 = 18.803 are material parameters of the blood vessels derived from previous experimental work (Teng et al., 2014) . The finite element method is used to solve the governing equations of plane-strain problem: where [v i ] and [σ ij ] are the displacement vector and stress tensor, respectively, ρ is the material density and t stands for time. A template pulsatile blood pressure waveform is applied on the lumen border. The structural stress map at the systole time point is extracted for analysis. It takes about ten mins to perform a 2D finite element analysis on a segmentation mask with 512x512 pixel IVUS image. As we focus on the data efficiency of our proposed framework, we simplified the simulation by resampling the segmentation mask into 64x64 pixel image size and ignore the different components within the plaque. This reduced the simulation time to two mins. An example of the input image and output stress map is shown in Fig S1 . Figure S1 : An example of the input and output of the labeling tool for IVUS dataset.
This paper introduces a framework for data-efficient representation learning by adaptive sampling in latent space. A method for sequential and adaptive selection of training examples to be presented to the training algorithm, where selection happens in the latent space based on choosing samples in the direction of the gradient of the loss. A method to efficiently select hard samples during neural network training, achieved via a variational auto-encoder that encodes samples into a latent space.