Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network
Xin Wang Bo Wu Yun Ye Yueqi Zhong
cs.CV cs.LG cs.MM

	
	
	
	
	
	
	
	
	
	
	wangx@mail.dhu.edu.cn
	0000-0002-4315-1867
	
		Donghua University &  JD AI Research
		Shanghai
		China
	

	
	bo.wu@columbia.edu
	
		Columbia University & JD AI Research
		New York
		USA
	

	
	yun.ye@intel.com
	
		JD AI Research
		Beijing
		China
	

	
	zhyq@dhu.edu.cn
	

		Donghua University
		Shanghai
		China
	


	
	
	
	
	
	
	
	
	
		Existing works about fashion outfit compatibility focus on predicting the overall compatibility of a set of fashion items with their information from different modalities. However, there are few works explore how to explain the prediction, which limits the persuasiveness and effectiveness of the model. In this work, we propose an approach to not only predict but also diagnose the outfit compatibility. We introduce an end-to-end framework for this goal, which features for: (1) The overall compatibility is learned from all type-specified pairwise similarities between items, and the backpropagation gradients are used to diagnose the incompatible factors. (2) We leverage the hierarchy of CNN and compare the features at different layers to take into account the compatibilities of different aspects from the low level (such as color, texture) to the high level (such as style). To support the proposed method, we build a new type-specified outfit dataset named Polyvore-T based on Polyvore dataset. We compare our method with the prior state-of-the-art in two tasks: outfit compatibility prediction and fill-in-the-blank. Experiments show that our approach has advantages in both prediction performance and diagnosis ability.
	
	



	
	
	
	
	
	
		<ccs2012>
		<concept>
		<concept_id>10010147.10010178.10010224.10010225.10010231</concept_id>
		<concept_desc>Computing methodologies Visual content-based indexing and retrieval</concept_desc>
		<concept_significance>300</concept_significance>
		</concept>
		<concept>
		<concept_id>10010147.10010257.10010293.10010294</concept_id>
		<concept_desc>Computing methodologies Neural networks</concept_desc>
		<concept_significance>300</concept_significance>
		</concept>
		<concept>
		<concept_id>10010405.10003550.10003555</concept_id>
		<concept_desc>Applied computing Online shopping</concept_desc>
		<concept_significance>300</concept_significance>
		</concept>
		</ccs2012>
	
	
	[300]Computing methodologies Visual content-based indexing and retrieval
	[300]Computing methodologies Neural networks
	[300]Applied computing Online shopping
	
	
	
	
	
	
	
	
	
    Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network
    Yueqi Zhong
    December 30, 2023
===================================================================================






§ INTRODUCTION

Outfit compatibility refers to whether multiple fashion items look good if worn together. Learning outfit compatibility can help consumers compose satisfactory outfits and help enterprises analyze the market. Many works used multimedia information to answer the questions like “Is this outfit compatible?” <cit.>, “How to generate a compatible outfit?” <cit.>. However, they lack explanations about the prediction and generation results , “Why is the outfit compatible?”. 






Outfit diagnosis means pointing out the incompatible factors in the fashion outfit. It provides not only the compatibility prediction but also the factors lead to incompatibility as the explanation. There are several reasons to diagnose the outfit: 1)  For researchers, it helps to understand the essence of compatibility. 2) For consumers, the explanations make the prediction and generation result more convincing. 3) For companies and designers, it provides hints for adjusting the incompatible outfit to be a more compatible and popular one. What does the diagnosis result look like? In this work, it has a format like “the outfit is incompatible due to the incompatible color of the trousers and shoes”. As illustrated in fig:task, Imaging a person is given an outfit for evaluation, what he or she would do is comparing each item with others and the outfit compatibility is a holistic feeling about all of the comparisons. The primitive unit in this thinking process is a single pairwise compared compatibility. A single item does not have the compatibility problem. On the other hand, fashion compatibility is evaluated in different aspects ranging from low level (such as color, texture) to high level (such as style). <cit.> has shown that low-level features may directly determine the aesthetic feeling of fashion. 

Outfit compatibility prediction is a binary classification problem. To compute the compatibility from variable lengths of fashion items in the outfit, previous works explored the pooling operation <cit.>, concatenation <cit.> or LSTM <cit.> to consume the CNN features and output a score. Even the deeply extracted features can achieve impressive performance, it is hard to explain the meaning of each feature dimension. Different from the classification problem, the compatibility problem is irrelevant to the feature contents but lie at the comparisons between features. Besides, in previous works, only the CNN features of the last layer are used which contain strong semantic information. As mentioned before, the low-level feature is of great importance since it not only provides different aspects for diagnosis but also affects the compatibility.




In this work, we introduce an end-to-end framework called Multi-Layered Comparison Network (MCN) to take both prediction and diagnosis into consideration. First, the overall compatibility is thought as a holistic feeling after comparing each fashion item with others, so we introduce the comparison module, which compares enumerated pairwise similarities between the input feature vectors as the output. The compatibility score is learned from all of these similarities with a non-linear model. To diagnose the outfit, we use the backpropagation gradient of each input similarity to approximate its importance for the incompatibility <cit.>. The similarity with the largest gradient value has the most critical problem. Second, to construct features in different aspects, we leverage the hierarchical architecture of CNN, which enables the model to learn semantically stronger information as the layers going deeper. We construct the comparison modules at different layers of the backend CNN. In the early layers, the comparisons concern more about the low-level features (such as color, shape), in the later layers they concern more about the high-level features (such as category, style).


To effectively implement the comparison module, we follow the ideas from the projected embedding <cit.> and type-aware embedding <cit.>. To leverage both image and text information of fashion items, we adopt visual semantic embedding  <cit.> to learn a common representation between them. We provide a cleaner outfit dataset named Polyvore-T for the experiment, which is constructed by adding type labels and removing items that are not relevant to fashion based on <cit.>. Our approach can also be generalized to other sources of dataset. For example, to handle the fashion blogs with richer textual components, named entity recognition can be firstly used to extract the descriptions of each part in outfits, then these descriptions can be fed to our framework. We show that the diagnosis result of our method is corresponding to human cognition of the outfit compatibility. The prediction performance is tested in two fashion tasks: outfit compatibility prediction and fill-in-the-blank, in which our approach shows superiority comparing to previous works. In summary, the main contributions of this work include:


	
  * We propose to diagnose the compatibility of the outfit, which is implemented by using the gradient values to approximate the importance of input similarities.
	
  * We propose to incorporate comparison modules at different layers of a convolutional neural network to account for different levels of semantic information.
	
  * We build a cleaner Polyvore-T dataset for experiments and show that our framework outperforms other baselines in outfit compatibility prediction and fill-in-the-blank tasks.







§ RELATED WORK



 §.§ Fashion Recognition and Understanding

Fashion recognition is the pioneer and basis of fashion compatibility learning. A significant amount of researches has been made to push the margin of this area in the deep learning community.  Kiapout <cit.> use a supervised method to learn 5 different fashion style with the support of human pose estimation. Liu <cit.> propose a large-scale fashion dataset called DeepFashion and a deep network jointly training the fashion categories, attributes, and landmarks. Edgar <cit.> propose to learn fashion embedding using weakly-labeled data from massive online product descriptions. Lee <cit.> leverage the context in fashion outfits and use a skip-gram model to learn a better fashion representation. The recognition of fashion is also affected by context information such as the city <cit.>, location <cit.>, scenery <cit.> and trend <cit.>. For unsupervised learning, Hisao <cit.> use the topic model to map fashion attributes to several fashion styles by considering fashion attributes as words, outfits as documents and styles as topics. Al-Halah <cit.> introduce a non-negative matrix factorization method to project the deeply extracted features into a fashion style space. 




 §.§ Visual Compatibility Learning


Even though compatibility is a subjective aesthetic standard, massive user feedbacks like co-view and co-purchase records <cit.> or proposals from the 	fashion community <cit.> can be used to learn the rule of compatibility. Recent works about outfit compatibility can be divided into two groups. The first group focuses on learning the pairwise compatibility, another group learns outfit compatibility in an end-to-end manner.


Pairwise compatibility can be learned as a metric by mapping item features into a compatibility space then compute their euclidean distance <cit.>. Further researches relief this metric since compatibility is not as strict as retrieval tasks. He <cit.> mix multiple metrics with different confidence to achieve more robust performance. Shih <cit.> propose to make multiple projection points of a query image. Vasileva <cit.> learn different metric spaces for different type combinations. While evaluating the outfit compatibility, these methods take the average of all pairwise compatibility as the output and neglect the relationship between the pairwise compatibility and the overall compatibility. 

For end-to-end training outfits, pooling <cit.> or concatenation <cit.> operations can be used to aggregate multiple item features.
Then the multilayer perceptron (MLP) can be used to compute a compatibility score. However, the addition operation in MLP is not an efficient way to learn interactions between each input <cit.>. Han <cit.> propose to use BiLSTM to construct dependency between several fashion items. Based on that, Nakamura <cit.> build an autoencoder upon the BiLSTM to learn the style embedding of outfits, which has similar idea as controlling the style of image captions <cit.>. However, an outfit is more like a set rather than a sequence because it is disordered.







§ APPROACH

Our approach aims to diagnose the outfit in different aspects, which is implemented by an end-to-end framework called Multi-layered Comparison Network (MCN). There are four parts in MCN: it starts from the multi-layered feature extractor (sec:multilayer) which extract features in different aspects. Then the comparison modules (sec:comparison) compare enumerated pairwise similarities between features along with multiple layers and the MLP predictor compute a score from them all. Visual semantic embedding (sec:vse) is used to handle multimodal information. The workflow of MCN is first predicting the outfit compatibility then backpropagating a gradient from the output score till the input similarities. Similar works are <cit.>, which visualize the response of each pixel to different classes for interpreting the CNN classification, but our goal is to visualize the response of each similarity to the compatibility score. 

In the following contents of this section, we first show the diagnosis process (sec:diagnosis) then introduce each part in the framework. The overall scheme of MCN is presented in fig:overall.




 §.§ Outfit Diagnosis by Gradients
 
Outfit compatibility can be considered as a summary after comparing each item with others in different aspects, , color, texture and style . To learn the relationship between the holistic compatibility and pairwise similarities, a simple linear model can be used and has great interpretability, because the weight of each input dimension indicate the importance for the output. The drawback of the linear model is the limited capacity. In contrast, multilayer perceptron (MLP) model has better capacity but the hidden unit is hard to explain. To leverage the advantage of MLP while maintaining the interpretability for diagnosis, we use the backpropagation gradients to approximate the importance of each similarity concerning the incompatibility.

Given an outfit containing N items, their features in a specific aspect, the color, can be denoted as a set X={x_1, x_2, …, x_N} where x_i is the vector for the i-th item. The enumerated pairwise similarities among the features can be represented as a matrix form:


    R=[
    r_11    r_12   ⋯    r_1N
    
    r_21    r_22   ⋯    r_2N
    ⋮   ⋮   ⋱   ⋮
    
    r_N1    r_N2   ⋯    r_NN]


Here, r_ij is the similarity between x_i and x_j, which is an undirected relationship, formally r_ij = r_ji. We call R the comparison matrix. For comparisons in K different aspects, there are K different comparison matrices {R^1, R^2, …, R^K}. All elements in these comparison matrices will be fed into a 2-layer MLP to compute the score of outfit compatibility:


    s = 𝐖_2 ReLU(𝐖_1[R^1; R^2; …; R^K] + 𝐛)


Here, [.,.] denotes concatenating multiple matrices and flattening them into a vector. The non-linear relationship between s and multiple R can be helpful for a better prediction performance but make it hard to estimate the importance of each input to the compatibility score like a simple linear model.  To solve this, we use the derivative of s to approximate the importance of each input similarities. One interpretation of this operation is if we compute the first-order Taylor expansion of the MLP model:


    s ≈𝐖^† [R^1; R^2; …; R^K] + 𝐛


the approximated equation is a linear model and the weight W^† can be easily interpreted as the importance of each input similarity. The element of W^† is the derivative of s with respect to the point of input similarities [R_0^1; R_0^2; …; R_0^K]:


    w_ij^k = ∂ s/∂ r_ij^k|_[R_0^1; R_0^2; …; R_0^K] .


Suppose incompatible outfits are labeled as 1, w_ij^k can be interpreted as the importance of each similarity concerning the incompatibility (If incompatible outfits are 0, just use the opposite number of w_ij^k). The importance of each item can be computed by summing up the gradients of all similarities containing it:


    ω_q = ∑_1 ≤ k ≤ K∑_i = q, j ≠ q w_ij^k


where ω_q is the diagnosed importance of the q-th item. By substituting the most problematic item, we can revise the outfit to be a more compatible one with very little change to the original composition. During the training, we use the sigmoid function σ(s) = 1/1+exp(-s) to model the output score as the compatibility probability and use the binary cross entropy as the loss function: 


    ℒ_clf = y ·logσ(s)
    + (1 - y) ·log (1 - σ(s))




 §.§ Comparison with Projected Embedding
 
An important problem is how to compute each pairwise similarity r_ij in the comparison matrix. The simplest way is adopting cosine similarity to measure the distance of features in a common space. However, a common space will induce some undesired consequences: (1) The variation of compatibility is compressed. For example, all trousers compatible with a shirt will be forced to close with each other, which means these trousers must be compatible with each other even they are not. (2) The triangle inequality will limit the position of each embedding. The triangle inequality states that for any three items, the sum of any two pairwise embedding distances should be greater than or equal to the remaining pairwise embedding distance. This will leads to an improper situation e.g., if a shirt matches a trouser, the trouser then matches a shoe, the consequence is the shoe is forced to match the shirt. Therefore, we use the projected embedding with respect to different fashion type combinations to address the above problems, which refers to <cit.>.

A fashion outfit often contains items from different types such as top, bottom and shoe . Different kinds of pairwise type combinations can be used as conditions for projecting the embedding to different subspaces. Let r_ij = f(x_i, x_j) as the similarity between x_i and x_j. We compute the similarity following a projection process:


    f(x_i, x_j) = d(P^i → (i,j) x_i, P^j → (i,j) x_j)


Here, P^i → (i,j) is the projection of i-th item conditioned on combination (i,j). d(.,.) is the cosine similarity. The projection in this work is implemented as:


    P^i → (i,j) x_i = ReLU(x_i ⊗ m_(i,j))


where m_(i,j) is a learnable mask vector with the same dimension as the features x_i, ⊗ denotes the element-wise product operation. The mask m_(i,j) works as an element-wise gating function which selects relevant elements in the feature vector for different compatibility conditions. We add two additional loss to penalize the training <cit.>: ℒ_mask aims to encourage the masks to be sparse and ℒ_emb encourages the CNN to encode normalized representations in the latent space:


    ℒ_mask(m) = ||m||_1 ,  ℒ_emb(x) = ||x||_2






















































 §.§ Multi-Layered Representation
 
The representation of fashion items in different aspects not only enables the model to diagnose the outfit in a detailed manner but also feed rich information for compatibility prediction. To construct different representations, one way is to predefine several concepts such as color, texture and shape . However, these definitions may not completely cover the fashion descriptions. Predefinition is not the only way for representations in different aspects. Modern deep learning use CNN which deeply connects multiple convolutional layers. Even though the convolutional kernel in each layer has a small size but with the architecture going deeper, the receptive field of the model can become larger. Therefore, the former layers in CNN capture low-level features such as color, texture, the later layers capture high-level features such as fashion style and compatibility. This characteristic of CNN can be used for building different representations of fashion items from low level to high level.

Leveraging this fact, we construct comparison modules at different layers of the backend CNN model. In detail, we use Global Average Pooling (GAP) to reduce the features maps at different CNN middle layers into vectors <cit.>. GAP has two benefits in this situation. First, GAP transforms feature maps into vectors to satisfy the comparison operation in eq:proj. Second, fashion features like color, texture are irrelevant to location and GAP can effectively get rid of the spatial information. Feature vectors of K layers are fed into K different comparison module to compute enumerated pairwise similarities {R^1, R^2, …, R^K}, which is the input for computing overall compatibility in eq:predictor. In experiment, we use ResNet-50 <cit.> as the backend network, 4 different layers are chosen for multi-layered representation including “conv2_x", “conv3_x", “conv4_x", “conv5_x". 




 §.§ Visual Semantic Embedding
 
In real-life situations, fashion items are often described by multimodal information such as image, tags . Visual Semantic Embedding (VSE) <cit.> is a method to take full advantage of information from different modalities by providing a common expression between them<cit.>.

For one fashion item in Polyvore dataset, its associated text description is like “classic skinny jeans”, which can be denoted as S = {w_1, w_2, ⋯, w_n}, where w_i is i-th word and can be represented as one-hot vector e_i. The word embedding of e_i is v_i = W_T e_i and the semantic embedding of an outfit is v = 1/M∑_i=1^M v_i, where W_T is the weight of word embedding model. A similar process is made to project the visual features x into the embedding space as u = W_I x. 

The object of VSE is to make u and v close to each other in the joint space when they are from the same item. Minimizing the following contrastive loss can achieve that object:


    ℒ_vse(v, u;W_T, W_I) =    ∑_u ∑_k max(0, m - d(u, v) + d(u, v_k)) 
        + ∑_v ∑_k max(0, m - d(v, u) + d(v, u_k))


where d(u, v) is the function to compute the distance between two embeddings. For one matching (from the same item) pair v and u, v_k denotes the semantic embeddings of all possible non-matching items, u_k denotes the visual embeddings of all possible non-matching items. This loss function expects the distances of all matching pairs u and v are smaller than all of the other non-matching pairs with a margin m. In practice, we can use the mini-batch as the set to enumerate all u_k and v_k.

To the end, our framework can be jointly trained in an end-to-end manner with a total loss ℒ_total and λ_{1, 2, 3} is the weights for each additional losses. The trainable parameters includes {Θ, W_T, W_I, W_2, W_1, b, M} where Θ is the parameters for the backend CNN model, W_2, W_1, b are the parameters of MLP in eq:predictor, M denotes the masks for different conditions in eq:mask.


    ℒ_total = ℒ_clf + λ_1 ℒ_emb + λ_2  ℒ_mask + λ_3 ℒ_vse








§ EXPERIMENT

To overcome the subjective nature of fashion sense, a dataset containing the compatible outfits can be constructed by fashion-sensitive people. The randomly combined outfits are highly possible incompatible. In this section, we first introduce our method to build the Polyvore-T dataset. Then we compare the prediction performance of our MCN framework with several baselines. Finally, we show the result of automatically diagnosing and revising the fashion outfit. Code and dataset are released at <https://github.com/WangXin93/fashion_compatibility_mcn>.



 §.§ Polyvore-T Dataset

We build a type-labeled fashion outfit dataset upon the Polyvore dataset <cit.>. Original Polyvore dataset contains 21889 expertise-selected outfits. Graph segmentation is used to split train, validation and test dataset. Each item has the corresponding image, text description, votes, and category (like jeans, skirts, sports, totally 381 kinds of category). However, there are overlaps between these categories such as “shoulder bags“ and “bags“ . At the same time, some categories do not have insufficient samples for training. So we label type information for each item by grouping 381 categories into 5 types with the following method:  (1) Unrelated categories such as “lipsticks”, “mirrors” are filtered and there are 158 categories retained after filtering. (2) The filtered categories are classified into 5 types (top, bottom, shoes, bag, accessory) by hand. (3) The item not in the category list will be dropped and the left items can be labeled according to their categories. When there are multiple items of the same type in one outfit, only the first item will be taken. We filtered out outfits in the dataset with less than 3 types to make sure each outfit in our dataset has at least 3 items and up to 5 items. The statistics of the filtered dataset are shown in tab:stats. Type information enables the comparisons are conditioned on the type combination as discussed in sec:comparison. It also prevents repeating items such as two pairs of trousers in one outfit. 





 §.§ Experiment Settings


Training.
In all experiments, we use ImageNet<cit.> pre-trained ResNet-50 <cit.> model as the backbone model unless specified. The spatial size of the input images is 224 × 224. The input outfit has a variable length from 3 to 5.  The empty parts are handled by padding a mean image of that type. Batch normalization layer <cit.> is added after the comparison modules to normalize the similarities of the same type combination. All experiments are trained on a single Tesla P40 GPU, each input mini-batch has 32 outfits. It takes about 9 hours to train 50 epochs. The initial learning rate is 1e^-2 and decay by a factor of 0.2 every 10 epochs. The optimizing strategy is SGD with a momentum of 0.9. We set the weights of additional losses λ_{1, 2, 3} are 5e^-3, 5e^-4 and 1 respectively. Only model parameters with the best performance on the validation set will be saved. 


Negative Sample.
During the training, the positive samples or compatible outfits come from the ground truth dataset. Negative samples are generated by substituting each item in the positive outfit samples with a random item of the same type from the dataset. It is because in real life fashion designers focus on composing a compatible outfit, but seldom think how to build an incompatible outfit. The subjectivity in incompatibility also makes the dataset labeling unreliable. Since fashion experts in communities like Polyvore have composed the outfits with the different aesthetic rules, randomly composed outfits are highly possible incompatible. 




 §.§ Tasks
 

The performance of outfit compatibility prediction is evaluated in two tasks: outfit compatibility prediction AUC and fill-in-the-blank accuracy. fig:eval gives an illustration of the two tasks. During the test, each model is evaluated 5 times with different dynamically generated test sets.

Outfit Compatibility Prediction (AUC): 
	The goal of the outfit compatibility prediction task is to compute a score as overall compatibility. MCN predicts the score from multiple input images in an end-to-end way. We generate test set following subsec:Details with a positive and negative ratio of 1:1. The area under the ROC curve is used to compare different methods.
	
Fill-in-the-blank (FITB): 
	Fill-in-the-blank task aims to select the most compatible item with the remainder of the outfit as the question. It is evaluated by the accuracy of correctly answering the question. Each question has 4 options in our experiment. MCN and other baselines do this task by predicting scores of 4 different outfits in which only substituting the blank part with different options, then choose the option with the highest score as the answer.









 §.§ Baselines


We compare our method with several recent fashion outfit compatibility works. Their brief descriptions are as follows:

Pooling <cit.>: 
It uses average-pooling operation to aggregate the variable length of feature inputs to predict the compatibility.
	
Concatenation <cit.>: 
It concatenates 5 item features into a long vector with a length of 1000 × 5, then uses MLP as the binary classifier. We set the size of the hidden layer as 1000.
	
CSN <cit.>:
It is a metric learning method for pairwise compatibility. The compatibility is computed as the distance of projected embeddings conditioned on different type combination. Outfit compatibility is the average of all pairwise compatibility.
	
BiLSTM+VSE <cit.>:
Each time step LSTM consumes one CNN encoded features and outputs a hidden state and a prediction of the next item. The cross entropy between the predicted items and the ground-truth is the compatibility score. It jointly optimizes the forward LSTM loss, backward LSTM loss, and VSE loss. 
	
Self-Attention <cit.>:
It uses the self-attention mechanism to relate different items in an outfit to compute a representation of the outfit. We use the scaled dot-product attention <cit.> where the query, key, and value are the item features in the same outfit.






 §.§ Prediction Performance


The prediction performance of MCN. The results of outfit compatibility prediction AUC and FITB accuracy are shown in tab:comparsion. It can be observed that there is a clear margin between MCN and other baselines. FITB is a harder task because substituting only one part may have little effect on global estimation. BiLSTM performs not well when there is no repeating item of the same type in the outfit, this result is similar to experiments in <cit.>. Self-Attention computes the relationships between item features as the attention weights, but it does not perform well in the compatibility task because the compatibility cares more about the feature similarities than the feature contents. The pooling method performs better than several complex methods because it is an interaction between all item features. MCN has the same spirit that learns overall compatibility from all pairwise feature similarities.



Ablation Study To analyze the efficiency of each part in the MCN framework, we conduct an ablation study. The results are shown in tab:ablation1 where CM stands for the comparison module, VSE is the Visual Semantic Embedding and PE is the Projected Embedding. It can be observed that concatenating item features is the most inefficient method. The comparison module without projection embeddings can already achieve competitive performance and the comparison operation uses fewer parameters because of the removal of MLP with long feature vectors. VSE regularizes the training so there is no additional parameter while evaluation; the parameters of PE have the same magnitude with feature vectors.


What is the relationship between the overall compatibility and pairwise similarities? As illustrated in tab:ablation2, end-to-end training with comparison module and metric learning method CSN perform similarly. However, CSN has already sampled from the training dataset, so it can not use the same dataset to learn this relationship between the overall compatibility and pairwise similarities. However, the depth of the MLP predictor in MCN can be finetuned to study this problem. We explore 0 to 3 layers and find there is a plateau on 2-layer MLP. It indicates that there is a non-linear relationship between them.


Multi-layered representations benefit the performance. fig:scales is an example of retrieval  with representation at different layers It can be seen that the results retrieved by the low-level features such as layer 1 are similar at color, texture. The results retrieved by high-level features such as layer 4 have similarity in visual style and compatibility. This observation proves the multi-layered representation in MCN ranges from the low level to the high level. tab:scales analyzes the effects of input from different layers. It can be seen that each layer has its contribution to the performance, which supports that the low-level features have an important effect on fashion compatibility.
































 §.§ Outfit Compatibility Diagnosis

Diagnosis from Different Layers
The diagnosis process computes the importance of each pairwise similarity and item concerning the incompatibility as described in sec:diagnosis We show several diagnosis results in sec:diagnosis. All diagnosis values are normalized to let margin between the maximum and minimum as 1. The edges with the 3 greatest values are marked as red dash lines. It can be observed that the diagnosis result is corresponding to human. For instance, in fig:diagnosis3 the green dress is pointed out mainly because its comparison between the black sweater is incompatible.

Automatically Revise the Outfit
One benefit of outfit diagnosis is we can revise the outfit based on the diagnosis result, which lets us make very little change to the original composition. We try a simple strategy to substitute the problematic items in the outfit as described in Algorithm <ref>. Several results are shown in fig:revise. It can be seen that the revised outfit has a better aesthetic feeling while most items are the same as that in the original outfit.








§ CONCLUSION

In this paper, we introduce an approach to not only predict but also diagnose the fashion outfit. We introduce an end-to-end framework that incorporates the comparison module, multi-layered representation, and visual semantic embedding. The diagnosis process is implemented by first predicting the compatibility score then use the backpropagation gradient to approximate the importance of each compared similarities to the incompatibility. Experiments show that the comparison module is an efficient way to aggregate multiple features for learning outfit compatibility. Representations from different layers also boost the performance which indicates that both low-level features and high-level features have impacts on fashion compatibility. We show that our framework can diagnose the outfit by pointing out the problematic similarities and items, which can be used to interpret the prediction and automatically revise the outfit. For future work, we are planning to explore how to explain different aspects in a better way and quantitively evaluate the diagnosis results.


This work is supported by the National Natural Science Foundation of China (Grant No.61572124). Thanks JD AI Research led by Dr. Tao Mei and thanks Wei Zhang for hist insightful discussions.





ACM-Reference-Format


