the visual domain. Instead, we take a data-driven approach to discover relationships in our data: we assume the network will intrinsically develop connections between different categories when clean examples are offered, and all we have to do is to distill the knowledge out.

We take a simple approach by just testing our network on the training set, and take the confusion matrix as the relationships. Mathematically, for any pair of concepts $i$ and $j$, the relationship $R_{ij}$ is defined as: $$R_{ij} = P(i | j) = \frac{\sum_{k\in C_i}CNN(j|I_k)}{|C_i|},$$ where $C_i$ is the set of indexes for images that belong to concept $i$, $|\cdot|$ is the cardinality function, and given pixel values $I_k$, $CNN(j|I_k)$ is the network’s belief on how likely image $k$ belongs to concept $i$. We want our graph to be sparse, therefore we just used the top $K$ ($K=5$ in our experiments) and re-normalized the probability mass.

After constructing the relationship graph, we put this graph (represented as a matrix) on top of the seventh layer of the network, so that now the soft-max loss function becomes: $$L = \sum_{k}\sum_{i}R_{il_k}\log(CNN(i|I_k)),$$ where $l_k$ is the class label. In this way, the network is trained to predict the context of 