Title: An Integrated Multi-Label Multi-Modal Framework in Deep Metric Learning. Abstract: Domains such as healthcare demand machine learning models which provide representations for complex relationships between both heterogeneous modes of data, and multiple co-occurring labels. Previous works have tackled representation learning in the multi-label, multi-modal setting, but have neglected to consider the common requirement of generalization to novel, and unknown, tasks at test-time. In this work, we propose an integrated multi-modal multi-label framework for deep metric learning, which we term 3ML--DML. Our framework extends existing proxy learning losses to the multi-label domain, and provides a novel method for enforcement of label correlations via these proxies. The multi-modal component builds a standard fusion model but draws from deep metric learning criteria in order to incorporate auxiliary, high-dimensional embedding and feature spaces from each mode of data as context to match with the output of the fusion model. We explore our method in a variety of settings, including on healthcare data, and demonstrate improvement over constructed baselines both in the context of multi-label multi-modal learning but most poignantly, in zero-shot generalization to new labels.