model, our solution can be seen as an early-fusion approach. Nevertheless, it is based on retrieval models, without the need to work directly on feature level. In this sense, we categorize the method as hybrid.

While early-fusion approaches are theoretically able to capture correlations between modalities, often a certain modality produces unsatisfactory performance and leads to biased or over-fitted models [@zadeh2017tensor]. Most early-fusion methods work in a two-step procedure, first extracting features from different modalities, then fusing them by strategies such as concatenation [@kiela2014learning], singular values decomposition [@bruni2014multimodal] or autoencoders [@silberer2014learning]. A few others focus on multimodal features jointly [@kottur2016visual; @hill2014learning], although generally restricted to a pre-defined textual attribute set. Concatenation is a straightforward yet widely used early-fusion approach, which merges vectors obtained by different descriptors. As a drawback, concatenation does not explore inherent correlations between modalities.

Supervised early-fusion optimizes a weighted feature combination, either during or after feature extraction. A common strategy is to build a neural architecture with multiple separate input layers, then including a final supervised layer such as a regressor [@nogueira2018exploiting]. Another approach is the design of a composite loss function, suited for the particular desired task [@park2016image]. Composite loss functions work well in practice, but, 