The knowledge that humans hold about a problem often extends far beyond a set of training data and output labels. While the success of deep learning mostly relies on supervised training, important properties cannot be inferred efficiently from end-to-end annotations alone, for example causal relations or domain-specific invariances. We present a general technique to supplement supervised training with prior knowledge expressed as relations between training instances. We illustrate the method on the task of visual question answering to exploit various auxiliary annotations, including relations of equivalence and of logical entailment between questions. Existing methods to use these annotations, including auxiliary losses and data augmentation, cannot guarantee the strict inclusion of these relations into the model since they require a careful balancing against the end-to-end objective. Our method uses these relations to shape the embedding space of the model, and treats them as strict constraints on its learned representations. %The resulting model encodes relations that better generalize across instances. In the context of VQA, this approach brings significant improvements in accuracy and robustness, in particular over the common practice of incorporating the constraints as a soft regularizer. We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used. It demonstrates the value of an additional training signal that is otherwise difficult to extract from end-to-end annotations alone. The capacity to generalize beyond the training data is one of the central challenges to the practical applicability of deep learning, and grows as the task considered grows more and more complex. Endto-end training provides a weak supervisory signal when the task requires a long chain of reasoning between its input and output (Glasmachers, 2017; Marcus, 2018; Zador, 2019) . A prime example is found in the task of visual question answering (VQA), where a model must predict the answer to a given text question and related image (see Fig. 1 ). Typical VQA models trained with supervision (questions/images and ground truth answers) tend to capture superficial statistical correlations, rather than the underlying reasoning steps required for strong generalization (Goyal et al., 2016; Agrawal et al., 2018; Teney & van den Hengel, 2016) . Prior knowledge that reflects a deeper understanding of the data than the ground truth answers offers an invaluable -although currently ignored -source of information to train models more effectively. We propose a method to incorporate, in deep learning models, prior knowledge that is specified as relations between training examples. Incorporating knowledge beyond end-to-end supervision is an opportunity to improve generalization by providing high-level guidance complementary to ground truth labels. The fact that two data elements are equivalent, e.g. two questions being rephrasings of each other in a question-answering task, provides more information than merely illustrating that they share the same answer. Such a constraint of equivalence exemplifies a high-level, general concept that is much more powerful than a set of examples sharing a label. Prior knowledge has previously been incorporated into network architectures in multiple ways, e.g. by sharing weights spatially in a CNN (Nowlan & Hinton, 1992) . Existing approaches are however often task-specific, and more importantly, usually operate in parameter space (Cohen & Welling, Figure 1: We demonstrate our method on the task of visual question answering (VQA), where we exploit three types of auxiliary annotations expressed as relations between training questions. This task-specific knowledge (for example the equivalence of synonymous questions) provides a training signal complementary to the end-to-end annotations of ground truth answers. 2016; Guttenberg et al., 2016; Laptev & Buhmann, 2015; Teney & Hebert, 2016) . In contrast, our approach uses knowledge expressed in embedding space, which we find far more intuitive for expressing higher-level knowledge. Indeed, in embedding space, one can specify how the network represents data. In parameter space, one can guide how the network processes these representations. While both can be useful, the former can map more directly to a task-or domain expert's knowledge of the data used. The additional knowledge that we use comes as annotations of relations between specific training instances. We do not require all of the training data to be annotated with this additional knowledge. Technically, we propose a novel training method that operates in two phases (see Fig. 2 ). First, we employ the constraints derived from the annotations as soft regularizers. They guide the optimization of the target task to loosely satisfy the constraints. Second, we retrain the earlier layers of the network by distillation (Hinton et al., 2015) using, as targets, embeddings projected on the manifold where the constraints are met perfectly. This second phase is the crux to enforce hard constraints on the learned embeddings. A major finding in our experiments is that these hard constraints are more effective the soft regularizers, in all of our test cases. We present an extensive suite of experiments on synthetic and large-scale datasets (Section 4). In the context of VQA, we apply the method on top of the popular model of to leverage three types of annotations illustrated in Fig. 1 : relations of equivalence between questions (i.e. being rephrasings of one another), of entailment (the answer to a general question being deducible from that of a more specific one), and relations of set membership, where questions are known to share some reasoning steps (e.g. questions referring to similar objects or requiring similar reasoning operations). These annotations are provided with the GQA dataset (Hudson & Manning, 2019) , but have largely been overlooked due to the difficulty of combining this type of knowledge with end-to-end training. hlWe show that imposing hard constraints on linguistic embeddings in this context is superior to the corresponding soft regularizers. We also demonstrate consistent improvements in robustness and accuracy independent from the amount of supervised data, which supports the benefits of such training signals in complement to end-to-end annotations. The contributions of this paper are summarized as follows. 1. We propose a method to exploit prior knowledge expressed as relations between training instances when training a deep learning model. The method enforces hard constraints on the internal representations learned by the model while allowing end-to-end supervised training, which alleviates issues with soft regularizers. 2. We show that the method is applicable to a range of tasks and types of prior knowledge. We describe its application to three generic types of relations (symmetric/equivalence, asymmetric, and set membership) and show that it does not require domain-specific expert knowledge. In many cases, the specification of constraints in embedding space is more intuitive than the alternative practice of designing regularizers in parameter space. 3. We demonstrate the benefits of the method on the task of VQA. We show how to exploit three types of auxiliary annotations about the training questions (equivalence, entailment, and common reasoning steps). This is the first published VQA model to make use of these annotations, which our method allows us to leverage to bring clear improvements in robustness and accuracy. Our results suggest that they provide a training signal complementary to end-to-end annotations. We presented an approach to incorporate prior knowledge in deep learning models in the form of constraints on its internal representations. We then applied the method to the task of VQA to leverage multiple types of relations between training questions. This application is of particular interest because VQA is a prime example of a task where the end-to-end supervised paradigm shows its limits, due to the long chains of reasoning that connect the inputs and outputs. The proposed approach served to shape the space of the internal representations learned by the model. Our experiments with the GQA dataset showed clear benefits in improving the accuracy and robustness of an existing VQA model. Interestingly, these benefits hold regardless of the amount of training data used for end-to-end supervision, suggesting that the type of prior knowledge used cannot otherwise be effectively captured in the model through end-to-end annotations alone. Technically, the proposed method treats the additional knowledge as hard constraints on the model's internal representations. This proved more clearly effective than existing methods to exploit this type of annotations, including soft regularizers and data augmentation. The proposed method can apply to a variety of tasks and domains that we hope to explore in future work. Concrete applications overlap with those considered for topological embeddings such as learning representations for knowledge bases, and other tree-or graph-structured data (Ganea et al., 2018; Nickel & Kiela, 2017; Vendrov et al., 2016) . Enforcing hard constraints on a learned model also allows one to provide guarantees that are otherwise impractical to meet with a purely datadriven approach. In particular, the method could be used to integrate known causal relations in a model, or known outcomes of interventions on specific training examples. This holds the promise of making further steps toward generalizable and trustworthy machine learning models.
Training method to enforce strict constraints on learned embeddings during supervised training. Applied to visual question answering. The authors propose a framework to incorporate additional semantic prior knowledge into the traditional training of deep learning models to regularize the embedding space instead of the parameter space. The paper argues for encoding external knowledge in the linguistic embedding layer of a multimodal neural network, as a set of hard constraints.