Supervised machine learning models for high-value computer vision applications such as medical image classification often require large datasets labeled by domain experts, which are slow to collect, expensive to maintain, and static with respect to changes in the data distribution. In this context, we assess the utility of observational supervision, where we take advantage of passively-collected signals such as eye tracking or “gaze” data, to reduce the amount of hand-labeled data needed for model training. Specifically, we leverage gaze information to directly supervise a visual attention layer by penalizing disagreement between the spatial regions the human labeler looked at the longest and those that most heavily influence model output. We present evidence that constraining the model in this way can reduce the number of labeled examples required to achieve a given performance level by as much as 50%, and that gaze information is most helpful on more difficult tasks. Medical imaging is a compelling application area for supervised machine learning methods. Convolutional Neural Networks (CNNs) in particular have recently achieved promising results on applications ranging from cancer diagnosis BID5 to radiograph worklist prioritization BID4 ; however, these results rely on massive hand-labeled datasets. This requirement for large hand-labeled datasets -which are expensive, because they require physician time to create -has hampered efforts to deploy these models to improve clinical outcomes.To reduce this labeling cost, we explore rich observational signals that can be passively collected at annotation time, such as eye tracking (or "gaze") data, which describes where a person has looked while performing a task BID2 BID14 . This approach is possible because of recent advances in eye tracking technology, which has quickly transformed from a technique that was intrusive, inaccurate, and expensive into one that is viable for real-time gaze data collection BID6 . Inspired by the success of eye tracking techniques in NLP applications that only use gaze signal at train time BID7 , we examine a straightforward mapping of gaze data to visual attention layer activations in a way that encourages the model to draw influence from the same spatial regions most heavily utilized by the human annotator. While noisy observational signals are challenging to extract useful information from, we show that incorporating them alongside traditional declarative labels can reduce the amount of hand-labeled data required to achieve a given level of performance. We first apply our proposed technique to a simple image classification task, where we show that we can maintain model performance using as few as 50% of the training images when gaze information is incorporated at training time. We then examine how the difficulty of the task impacts this labeled data reduction, and show that observational signals appear to be more helpful for difficult tasks. While we demonstrate our approach on a non-medical dataset for which gaze data was available, similar reductions in required labeled data, particularly for difficult tasks, could improve the feasibility of training useful models for medical imaging tasks. In this study, we introduce a simple method for incorporating observational, passively collected gaze signals into CNN training procedures. We have demonstrated that constraining the model attention to spatial regions deemed relevant to an expert labeler can reduce the amount of labeled data needed to achieve a given level of performance. Additionally, we find that the performance gains from incorporating the observational signal are larger for more difficult tasks. Fully characterizing the circumstances in which we see such gains from gaze-augmented models is a promising avenue for future work. Going forward, we plan to assess the applicability of this technique to medical imaging tasks, and to further investigate how observational signals may improve model robustness.
We explore using passively collected eye-tracking data to reduce the amount of labeled data needed during training. A method to use gaze information to reduce the sample complexity of a model and the needed labeling effort to get a target performance, with improved results in middle-sized samples and harder tasks. A method to incorporate gaze signals into standard CNNs for image classification, adding a loss function term based in the difference between the model's Class Activation Map and the map constructed from eye tracking information.