Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrieval
Binghui Chen Weihong Deng
cs.CV




















Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrieval
    Binghui Chen^1, ^2, Weihong Deng^1Corresponding author

^1Beijing University of Posts and Telecommunications

^2AI Labs, Didi Chuxing, Beijing 100193, China

chenbinghui@bupt.edu.cn, whdeng@bupt.edu.cn

    December 30, 2023
==============================================================================================================================================================================================================

empty




   In zero-shot image retrieval (ZSIR) task, embedding learning becomes more attractive, however, many methods follow the traditional metric learning idea and omit the problems behind zero-shot settings. In this paper, we first emphasize the importance of learning visual discriminative metric and preventing the partial/selective learning behavior of learner in ZSIR, and then propose the Decoupled Metric Learning (DeML) framework to achieve these individually. Instead of coarsely optimizing an unified metric, we decouple it into multiple attention-specific parts so as to recurrently induce the discrimination and explicitly enhance the generalization. And they are mainly achieved by our object-attention module based on random walk graph propagation and the channel-attention module based on the adversary constraint, respectively. We demonstrate the necessity of addressing the vital problems in ZSIR on the popular benchmarks, outperforming the state-of-the-art methods by a significant margin. Code is available at http://www.bhchen.cn






§ INTRODUCTION




In zero-shot image retrieval (ZSIR), the model is required to learn embedding from the seen classes and then to be capable of utilizing the learned knowledge to distinguish the unseen classes without any attributes or semantic information.

Most existing works adopt deep embedding (metric) learning and dedicate to exploring powerful loss functions, such as pair-based methods <cit.>, and hard-example mining strategies <cit.>, etc. These ideas can be summarised as: learn a good matric on the seen categories so as to boost the performance on the unseen ones. However, in fact, they neglect the existence and significance of some vital problems to ZSIR, thus are easily stuck in the particular data distribution and knowledge zone of the seen training set, some helpful knowledge for unseen classes may have been left out with a high probability, as a result, their performances are almost on par with each other and unsatisfactory.





Specifically, in ZSIR, the ideas above are actually unreasonable as the discrimination and generalization abilities of the learned metric are highly influenced by the following two aspects: (1) the undiscriminating visual input, which contains much noisy information (it is hard to know where should be focused on), resulting in poor descriptors and inducing difficulty in distinguishing the unseen classes. In another word, undiscriminating visual input deteriorates the discrimination of features. (2) partial/selective learning behavior of deep models<cit.>, specifically, for a functional learner parameterized by CNN, when given an arbitrary input, it will selectively learn partial attributes knowledge that are the easiest ones to decrease the current training empirical risk over the seen categories, instead of learning all-sided details and information, thus yielding over-fitting on seen categories and generalizing worse to the unseen ones. For example on birds, if the head knowledge is enough to distinguish the training seen classes, the deep model will only focus on the head and ignore the other body parts like back, foot and wing, while when testing on the unseen classes (which have similar head features but different wing features), the trained model will easily fail to distinguish them[Since in zero-shot settings, the unseen classes have no intersection with the seen classes, this partial learning behavior is common.]. This partial learning behavior sometimes is one reason for neural redundancy, since all the neurons only focus on the certain easy knowledge, causing that the performance doesn't increase much with the feature dimension. Moreover, when optimizing an unified metric as in the aforementioned methods (as in Fig.<ref>.(a)), these two issues are commonly mixed with each other and jointly impede the learning of representation, and without explicit and helpful guidance, the poor discrimination and generalization characteristics of features are greatly exacerbated regardless of the efficacy of objective function and hard-sample mining strategy. And most ZSIR works ignore the importance of learning both discriminative and robust discriminators. To this end, proposing knowledge-preserving and discriminative metric learning framework remains important.




In this paper, inspired by the idea of “divide and rule”, we propose the Decoupled Metric Learning (DeML), a novel and generally applicable framework, to alleviate the above problems by decoupling the embedding representation into multiple attention-specific learners so as to encourage its discrimination and generalization in an explicit manner, as shown in Fig.<ref>.(b). Moreover, equipping each learners with the particular attention tasks will further alleviate the neural redundancy and give chances of performance improvement.

Our DeML is a model-agnostic learning framework, and different from the existing metric learning works, it has no need to mine hard samples or construct elaborate sample-pairs. The main contributions are as follows:


∙ We provide insights of two vital problems to ZSIR that undiscriminating visual input and partial/selective learning behavior of learner will impede the discrimination and generalization of the metric, resp. And propose DeML, a hybrid-attention based decoupling method, to address these challenges individually.

∙ DeML decouples the commonly used unified metric into object-attention root-learners and channel-attention sub-learners, aiming at recurrently inducing the discrimination and explicitly promoting the diversity together with the generalization, resp.

∙ The decoupled learners are supported by the corresponding random walk inferred object-attention modules (OAM)[Some works call it spatial-attention.] and the adversarial channel-attention modules (CAM). They can be easily performed as the drop-in parts and are model–agnostic.

∙ We demonstrate the necessity to learn discriminative and robust metric via our decoupling idea. Extensive experiments have been performed on several challenging datasets for ZSIR, including CUB <cit.>, CARS <cit.>, Stanford Online Products <cit.> and In-Shop <cit.>. Our DeML achieves state-of-the-art performances and surpasses other methods by a significant margin.






§ RELATED WORK




Zero-shot learning: ZSL has been widely studied in many tasks, such as image classification <cit.>, hashing<cit.>, video recognition <cit.> etc. However, different from these ZSL tasks that are capable of exploiting the extra auxiliary supervision signals of unseen classes (e.g. semantic word embedding of class name and explicit attribute information), our DeML concentrates on a more general version of zero-shot image retrieval (ZSIR), where there are only similarity labels s_pq∈{0,1} available. Therefore, how to capture the discriminative and all-sided information only from the input image is the mainstay of this task.

Deep embedding/metric learning: With only s_pq available, many works seek to exploring deep metric learning for ZSIR task. For example, sampling-Matters <cit.> proposes distance weighted sampling strategy. Proxy-NCA <cit.> explains why popular classification loss works from a proxy-agent view, and its implementation is very similar with Softmax. ALMN <cit.> proposes to optimize an adaptive large margin objective via the generated virtual points instead of mining hard-samples. However, all the above methods are to tackle with the unified metric by designing losses and exploring sample-mining strategies, thus suffer from the aforementioned issues easily. Additionally, HDC <cit.> employs the cascaded models and selects hard-samples from different levels and models. BIER loss <cit.> adopts the online gradients boosting methods. Although these two methods try to improve the performances by resorting to the ensemble idea, they can only learn from the undiscriminating input and also suffer from the partial learning behavior, as a result, the discrimination and generalization of holistic features are still limited.

Attention mechanism: Attention serves as a tool to bias the allocation of available resources towards the most informative parts of an input. Many are implemented in combination with a gating mechanism (e.g. softmax or sigmoid) and are widely applied in many tasks, e.g. image captioning <cit.>, lip reading <cit.>, image classification <cit.>. Schwartz et at. <cit.> adopt high order attention modules for VQA. Fu et at. <cit.> propose to learn an attention network so as to produce the attention proposals, while, it is optimized in a two-stage manner, i.e. iteratively train the embedding model and attention model. We emphasize that our DeML is a hybrid-attention system. First, it adopts a parameter-free object-attention module, which is performed by random walk graph propagation and can be directly plugged into any bedrock CNNs without training. Second, different from <cit.> where the channel-attention operations are cascaded, i.e. only the most informative signal will be captured, our DeML is equipped with a series of channel-attention modules in parallelism, which are optimized by the adversary module and insist to learn various combinations of attributes simultaneously and to capture knowledge as much as possible.




§ DECOUPLED METRIC LEARNING



As discussed in Sec.<ref>, in zero-shot setting, simply optimizing an unified representation like in most existing works wouldn't guide the learning of discriminative and robust metric as a joint result of (1) undiscriminating visual input and (2) the partial/selective learning behavior[Notably, these shortcomings also reside in conventional full-shot tasks, while they heavily exacerbate the performances in zero-shot settings.].

Naturally motivated by “divide and rule”, the intuitive solution is to decouple the single representation into a series of artificially defined learners so as to explicitly encourage different learners to attend to different latent features and improve the holistic feature's discrimination and diversity. Moreover, many research works point out that attention mechanism is a crucial component connecting data embeddings with decision making modules, and diverse attentional cues allows the learning of various knowledge. We follow this inspiring idea and recast the above solution as hybrid-attention learning by decoupling the original unified metric into several attention-specific parts, each part is followed by the corresponding fc learner, i.e. object-attention (OA) root-learner and channel-attention (CA) sub-learner. The overall framework of DeML is illustrated in Fig.<ref> (for clarity, we depict the network with only two OA root-learners and three CA sub-learners under each root-learner as an example).

Specifically, for the i-th scale, the embedding part consists of three different components: (1) the bedrock CNN, i.e. Googlenet-V1 <cit.> (we decompose it into (Conv1-Pool3)/FNet(Incep4-Pool5)/GNet), (2) the object-attention module (OAM) is to incrementally pinpoint the informative object-based region proposals for discriminative feature learning, and (3) the channel-attention module (CAM) is to concurrently produce diverse attribute-based channel-attention proposal cap_(i,j). When given an image x_n, we will directly use it for the first scale input and employ its cropped&zoomed version for the next scale input. At the i-th scale, the input will first be encoded by FNet^i and then fed into multiple CAMs. The produced channel-attention proposals cap_(i,j) will then be encoded by the corresponding subsequent GNet^i and CA sub-learners. All the CA sub-learners at the same scale constitute one OA root-learner. We now define OA_i,CA_(i,j) as the corresponding learners and OA_i(·),CA_(i,j)(·) as the corresponding encoding functions, then give the holistic feature representation φ[x_n] as:





    φ[x_n]={OA_1[x_n^1]; OA_2[x_n^2]; ⋯}
    
    w   here OA_i[·]={⋯; CA_(i,j)[·]; ⋯}





where the input x_n∈𝒳_seen≡{x_n}_n=1^N, x_n^i is the i-th scale image which is cropped and zoomed from x_n^i-1 by OAM_i-1, and x_n^1=x_n is the original scale image. Notably, we use square brackets “[]” to show the conditional relation, e.g. CA_(i,j)[x_n^i] indicates the CA learner CA_(i,j) is conditional on input x_n^i, and this relation can also be formulated as CA_(i,j)[x_n^i]=CA_(i,j)(g^i(CAM_(i,j)(f^i(x_n^i)))) (shows the encoding process from image to feature space), where f^i and g^i indicate the encoding functions of FNet^i and GNet^i, resp, CAM_(i,j) denotes the corresponding channel-attention module (CAM), i∈ [1⋯ I], j∈ [1⋯ J] (in Fig.<ref>, I=2,J=3). The total size of φ[x_n] is d-dim, thus for I OA learners and I× J CA learners, OA_i(·)∈ℝ^d/I and CA_(i,j)(·)∈ℝ^d/I× J.

Below, we will introduce our OAM and CAM in Sec.<ref> and Sec.<ref> resp, and then show the objective metric loss together with the regularization term in Sec.<ref>.






 §.§ Random Walk Inferred OAM




As we discussed above, in order to learn discriminative OA learner, we should first provide the discriminative visual inputs. However, unlike the detection methods that have precise bounding-box annotations about object locations(e.g. RPN <cit.>), there are no auxiliary annotations for training in ZSIR. We thus present the random walk inferred object-attention module(OAM), a parameter-free and spatial attention method, by performing graph propagation over the receptive fields of deep responses in Conv maps, so as to produce the object-attention proposals.

As illustrated in Fig.<ref>, our OAM takes as input many convolutional response maps (U^l_k∈ℝ^C_l_k×H_l_k×W_l_k, k∈[1,2,⋯], C_l_k, H_l_k and W_l_k denote the number of channels, height and width of the l_k-th layer, respectively), and then gives the final object-attention proposal. For clarity, we give a toy example and omit the subscript k for written convenience (if not specified), each spatial location (h,w) on U^l has a feature vector u_h,w^l∈ℝ^C_l. To produce the spatial object-attention proposal M^l∈ℝ^H_l×W_l, a fully connected directed graph G^l is first constructed by connecting every location on U^l with the weight matrix D^l∈ℝ^H_lW_l×H_lW_l, where D_hW_l+w,h^'W_l+w^'^l indicates the weight of the edge from node (h,w) to (h^',w^') and is defined as:





    D_hW_l+w,h^'W_l+w^'^l   ≜u_h,w^l-u_h^',w^'^l





Before performing propagation, the weights of the outbound edges of each node are normalized to 1, i.e. D_a,b^l←D_a,b^l/∑_b^'=1^H_lW_lD_a,b^'^l. Then, a random walk graph propagation algorithm <cit.> is employed to produce the object-attention proposal M^l. As shown in Fig.<ref>.(a), the node (h,w) receives and diffuses mass via the inbound and outbound directed edges respectively. Then this random walk propagation iteratively collects mass at the nodes that have high dissimilarity with their surroundings (i.e. it will highlight the saliency foreground object region), since the edge weight (i.e. transition probability) is proportional to the dissimilarity between two nodes as defined by Eq.<ref>, transitions into such subgraphs is likely, and unlikely if nodes have similar deep features u^l. By the globally objectness flow, it can not only collects object evidence but also depresses noise regions. For the convenience of random walk calculation, the proposal map M^l is first reshaped to a H_lW_l-dim vector, and then it is updated by the iterative multiplication with the weight matrix D^l, i.e. M←D×M.

Moreover, as a common knowledge that the response maps at different layers are of different information, e.g. the bottom layers have better predictive power for shape while the top layers are robust to noisy regions <cit.>, our OAM takes as input multiple Conv. layers U^l_k,k∈[1,2,⋯] so as to avoid the influence of noise surroundings (e.g. trees around birds), and each object-attention proposal M^l_k can be obtained by performing random walk graph propagation over U^l_k as follows:



    M^l_k←(∏_t=1^TD^l_k)M^l_k




M^l_k is first initialized with value 1/H_l_kW_l_k, and the weight matrix D^l_k is conditional on the response map U^l_k, then the iterative update of M^l_k like M←D×M can be rewritten as the above equation and each M^l_k can reach its stable state with T=10. After Eq.<ref>, all M^l_k are reshaped to 2D maps and resized to the same size as the input image, then they are averaged to a single map, i.e. the final object-attention proposal. In experiments, we find using {k∈[1,2], and  l_1=Incep4e, l_2=Incep5b} is enough, As in Fig.<ref>, with the help of this proposal, the coarse scale image is processed into the finer scale by operation , and then the finer scale input will benefit the discriminative learning of the corresponding OA learner.

Remark: The procedure in Eq.<ref> can be viewed as a variant of the eigenvector centrality measure <cit.>, which outputs a proposal map to indicate the mass of each location on the deep response maps. It can also be regarded as a Markov chain that can reach unique stable state since the chain is ergodic, a property which emerges from the fact that the graph G is by construction strongly connected. <cit.>.
Moreover, the random-walk based informative-region search is computationally lightweight and parameter-free, thus it can be easily employed in any bedrock CNNs and has no need to be additionally optimized, i.e. when given the well trained convolutional maps U, it can directly infer the important object region in an unsupervised manner. Finally, recurrently executing OAMs will incrementally improve the discrimination of the corresponding OA learners.




 §.§ Adversarial CAM



Although the OA learners can produce discriminative representations, each single OA learner will still be likely to suffer from the partial/selective learning behavior, i.e. only focuses on certain attributes (e.g. heads of birds, bellows of car, etc.) which are the easiest ones to reduce the current training risk. Therefore, in order to improve the diversity of each OA root-learner, we further decouple it into J CA sub-learners which insist to capture various attributes information coming from diverse channel-attention proposals cap_(i,j) as in Fig.<ref>, and each cap_(i,j) is produced by a particular channel-attention mudule (CAM_(i,j)). For clarity, we take one CAM for example, as shown in Fig.<ref>.(b), it takes as input U^l∈ℝ^C_l×H_l×W_l of the l-th layer (omit subscript k for convenience) and outputs a channel-weighted counterpart of U^l, i.e. cap_(i,j). We formulate it as:


    cap_(i,j)=CAM_(i,j)(U^l)=U^l⊡σ[W_(i,j)^2δ(W_(i,j)^1Ψ(U^l))]


where U^l is first aggregated to a channel descriptor through spatial average pooling Ψ, then is passed through two fully-connected layers (parameterized by W_(i,j)^1∈ℝ^64×C_l, W_(i,j)^2∈ℝ^C_l×64) that intend to capture the interactions among channels and are followed by ReLU (δ) and Sigmoid (σ) respectively. Finally, U^l is reweighted by the channel-wise multiplication ⊡ between 2D map U^l_c and scalar σ[W_2δ(W_1Ψ(U^l))]_c, where c is the channel index.

Since the filters in certain convolutional layers serve as the local attribute detectors to some extend, the resulting deep response maps will reserve the attribute information in channels. In another word, different attributes reside in different channels. Therefore, the reweighting operation in our CAM can be regarded as an attributes selector(e.g. in Fig.<ref>.(b) it selects 'foot' and 'wing' yet ignores 'body'), and the concerned cues(within cap_(i,j)) will be passed to the corresponding CA sub-learner. If there are diverse selectors, i.e. multiple CAMs, capable of capturing different combinations of attributes information in parallel, the corresponding CA learners will focus on different attributes knowledge, and the richness and diversity of each OA root-learner will be significantly improved.

While, to explicitly encourage the diversity of CAMs, directly adding diversity constraints after CAMs might be detrimental, since in fact it is hard to know which channel contains what attribute signal and which kind of channel combination is beneficial to the embedding learning. To this end, as shown in Fig.<ref>, all cap_(i,j) are first fed into the corresponding subsequent shared embedding layers, and the diversity of CAMs are naturally transferred to that of the CA learners. And in order to explicitly encourage the diversity of CAMs, we can instead impose diversity constraints on the CA learners. Then, an adversarial optimization strategy is introduced as the diversity constraint by playing a maxmin game, i.e. the adversary network tries to minimize the discrepancies among CA learners while CAMs try to maximize these discrepancies as follows:




    max_f^i,g^i,CAMsmin_ϝ^i L_adv(x_n^i;ϝ^i;f^i;g^i ;CAMs)=




    λ_0∑_j,j^'^Jϝ^i[CA_(i,j)[x_n^i]]-ϝ^i[CA_(i,j^')[x_n^i]]^2_2





where f^i, g^i and ϝ^i indicate the mapping functions of FNet^i, GNet^i and the adversary network respectively, λ_0 is the loss weight. The above equation is a discrepancy measurement over all J CA sub-learners that belong to the i-th OA root-learner. Moreover, to simplify the two-stage optimization in this maxmin game, a gradients reverse layer (GRL) <cit.> is introduced as shown in Fig.<ref>.(c), so as to make Eq.<ref> be equivalent to min_ϝ^iL_adv and min_(f^i,g^i,CAMs) (-L_adv).

Remark: Since at each scale i the params of GNet^i (g^i) in different channel-attention branches are shared as shown in Fig.<ref>, when optimizing min_(f^i,g^i,CAMs) (-L_adv), the only differentiating parts are CAMs, in other words, only diverse CAMs will make (-L_adv) small. Moreover, the reason of using adversary objective instead of directly constrain CA learners to be different with each other is because the definition of difference between learners is likely to be hard to be artificially made than the similarity, we thus play the above adversarial game so as to allow the automatic learning of the optimal discrepancies between CAMs.




 §.§ Objective Loss and Regularization Term

Employing multiple OAMs and CAMs, the holistic feature representation φ[x_n] can be decoupled into a series of attention-specific parts. Then, the binomial deviance loss <cit.> is utilized and the empirical metric loss over all CA sub-learners can be formulated as:



    L_metric=1/IJ∑_i=1^I∑_j=1^J∑_p, q1/w_pqlog(1+e^-(2s_pq-1)α(D^(i,j)_pq-β)γ_pq)

where α=2, β=0.5 are the scaling and translation parameters resp. γ_pq is the penalty coefficient and is set to 1 if s_pq=1(i.e. images x_p,x_q are from the same class), otherwise γ_pq=35. w_pq is the number of positive pairs(negative pairs) if s_pq=1(s_pq=0). D_pq^(i,j)=<CA_(i,j)[x_p^i],CA_(i,j)[x_q^i]>/CA_(i,j)[x_p^i]CA_(i,j)[x_q^i] measures the distance between inputs x_p^i and x_q^i at learner CA_(i,j). Notably, if J=1, the OA root-learner has no need to be decoupled, thus CA_(i,j)=OA_i. Additionally, the learners are trained on the relatively small datasets (compared to ImageNet <cit.>) from scratch and thus are easier to be over-fitting, following the commonly used weight decay strategy which regularizes the value of parameters not to be too large so as to reduce over-fitting, we present a regularization term for optimizing each learner, called activation decay, to constraint the representation value as follows:





    L_act=λ_1/2IJN∑_i=1^I∑_j=1^J∑_n=1^NCA_(i,j)[x_n^i]^2_2+L_ntri
    
       where L_ntri=λ_2∑_i=1^I∑_j=1^Jtr[(ω_ijω_ij^T-I)⊗(ω_ijω_ij^T-I)]





where L_ntri is used to avoid the trivial solution(i.e. all parameters are optimized to 0) when minimizing L_act, λ_1,λ_2 are trade-off hyper-parameters. ω_ij∈ℝ^d/IJ×d_1 are the parameters in learner CA_(i,j), d_1 indicates its input channel number. ⊗ refers to the Hadamard-Product. Finally, DeML can be trained by jointly optimizing Eq.<ref>, <ref> & <ref>.






 §.§ Discussion



Why decoupling? In ZSIR, the traditional idea of devising loss functions for optimizing an unified metric omits the issues of (1) undiscriminating visual input regions and (2) the partial/selective learning behavior of learner, and thus exhibits limitations in discriminative and knowledge-preserving feature learning that is crucial for ZSIR. However, in our DeML, decoupling this unified metric into multiple separate learners gives us chances and flexibilities to mitigate the above problems individually, i.e. recurrently improve the discrimination via OA root-learners and explicitly encourage the diversity and richness via CA sub-learners, and as a result, the redundancy in features is reduced and the representation ability is encouraged as well. Additionally, in Sec.<ref>, we show that our decoupling idea outperforms the existing methods in ZSIR by a significant margin even without hard-samples mining.

Relation to other ensemble methods: 
In deep metric learning community, there are some other ensemble works related to us. Yuan et al <cit.> employ multiple layers at different depths for hard-aware samples mining and then cascade the learned embeddings together. Opitz et at <cit.> adopt the online gradients boosting and optimize different learners with the reweighted data. Kim et at <cit.> try to increase the feature diversity via contrastive loss but ignore the importance of learning discriminative metric in ZSIR task. In summary, although these methods aim at learning ensemble metric, they easily suffer from the undiscriminating inputs, and the partial learning behaviors are not effectively constrained as well. We emphasize that our DeML recasts the ensemble problem as a hybrid-attention model that has a clear object of addressing the aforementioned issues individually, and decouples the unified metric into multiple attention-specific parts, so as to explicitly intensify both the discrimination and diversity within each part of features, thus capturing rich knowledge and being capable of generalizing to unseen categories.

Relation to other attention mechanism: Many other attention works, such as spatial-attention <cit.>, channel-attention <cit.> and semantic-attention <cit.>, employ the task-specific attention modules to mine the most informative signals. While, we emphasize that our DeML is a hybrid-attention system, which dedicates to capturing not only the informative object-attention regions via OAMs but also the diversity of attributes via CAMs. Moreover, OAM differs from works <cit.> in that it is performed by the random walk graph propagation, which is in a parameter-free manner and can be directly employed without training. And different from <cit.> where the channel-attention modules are to extract the most informative channel-signals, our parallel CAMs are constrained by the adversary task such that they are different with each other and capable of extracting rich and complementary attributes knowledge. In summary, CAMs can work in conjunction with OAMs, and the generalization and discrimination of the holistic metric are explicitly and naturally ensured.




§ EXPERIMENTS



Notation: For clarity, the conventional unified metric learning(i.e. directly train d-dim features with Eq.<ref>) is denoted by Ud, e.g. U512, and we set it as our baseline. The proposed method is denoted by DeML(I,J), where there are I OA root-learners and I×J CA sub-learners in total, d is set to 512 by default and each OA(CA) learner is 512/I(512/IJ)-dim, if not divisible, operation floor is performed.

Implementation details: Following the codes released by <cit.>, we choose the pretrained GooglenetV1 <cit.> as our bedrock network and adopt the same data preprocess method so as to make fair comparisons with other works. The learners, CAMs and adversary networks are initialized with random weights. CAMs are placed after pool3 layer, OAM takes as inputs incep4e_output and incep5b_output.

Training: The optimizer is Adam <cit.> with learning rate 1e-5 and weight decay 2e-4. The training iterations are 15k(CUB), 25k(CARS), 100k(Stanford Online Products and In-shop) resp. We set λ_0=1,λ_1=0.014,λ_2=0.25 and 10 times learning rate for the learners.

Evaluation and datasets: For fair comparison, we follow <cit.> to evaluate the retrieval performance by Recall@K metric. For testing unseen images, the outputs of I×J learners will be concatenated into the holistic feature after L2 normalization. Then we use the simple cosine distance rule for similarity measurement. The datasets include CUB<cit.>, CARS<cit.>, Stanford Online Products<cit.> and In-Shop<cit.>. There is no intersection of classes between training and testing sets.


 §.§ Results





Effect of activation decay: From Tab.<ref>, one can observe that with the help of our activation decay L_act, the performances of unified metric learning under different feature sizes are consistently improved over both CUB
rt0.45

  1.2!
    
    Method     CUB (R@1)     CARS (R@1) 

    U64     50.0     65.4

    U64+L_act     50.9 ↑     67.1 ↑

    U192      52.4      68.3 

    U192+L_act     54.9 ↑      74.6 ↑ 

    U384      53.0        69.1 

    U384+L_act     55.6 ↑      77.4 ↑ 

    U512      52.9      68.4 

    U512+L_act     56.2 ↑      77.6 ↑ 

    
    
font=small
      Effect of L_act.
      

  

and CARs datasets, e.g. (U512+L_act) vs. U512, moreover, from Fig.<ref> one can observe that the training curve of U512 rises fast to a higher level, i.e. 1, but its testing curve first rises a bit and then drops continuously to 0.2, indicating that U512 suffers from serious over-fitting over the seen categories, while the training curve of (U512+L_act) rises slower than U512 and its testing curve has a rising trend, implying that L_act indeed serves as a regularization term and can improve the generalization ability on unseen categories by reducing over-fitting. Additionally, from the results of (U64+L_act) to (U512+L_act), one can observe that the performances increase with dimension d, but when d is large enough (e.g. from 384 to 512), the performance gain is small (55.6% vs. 56.2% on CUB, 77.4% vs. 77.6% on CARS), and the similar phenomenon can be observed from U64 to U512 as well, indicating that the neural redundancy indeed exists and limits the further improvement, and highlighting the necessity of our decoupling idea laterally.


Attention visualization: In order to intuitively understand our attention modules, we provide clear visualizations of the corresponding attention proposals for qualitative analysis, i.e. the attended regions at multiple scales by the proposed OAMs and the diverse attribute proposals by CAMs, Fig.<ref>. Firstly, for object-attention regions, one can observe that these localized regions at second or third scales are discriminative to the corresponding categories, and are easier to be recognized than the first scale. Secondly, for channel-attention proposals, it can be observed that different CAMs process the same input U^l into diverse proposals, which contain different attributes combinations, by different weighting strategies. Thus forcing the following CA learners to concentrate on different knowledge, the partial learning behavior can be avoided. These above results are consistent with human perception that it would be helpful to look closer for better decisions and to learn rich knowledge for handling unseen categories.























































Effect of attention modules: We also conduct quantitative comparisons on attention modules in Tab.<ref>. By default, dimension d is set to 512. The model DeML(I=1,J=1) is very similar to model (U512+L_act) with only a small difference of an extra single CAM, and from Tab.<ref> and Tab.<ref>, one can observe that their performances are almost the same(56.2% vs. 56.1% on CUB, 77.6% vs. 77.9% on CARS), implying that capturing the single attentional knowledge via attention module might not be helpful for ZSIR. Moreover, in Tab.<ref> one can observe that, by fixing J=1, the holistic performance (H) will significantly increase with I (56.1%→60.9%→64.9% on CUB, 77.9%→83.4%→85.6% on CARS), and the OA learner at finer scale will be more discriminative than that at coarse scale, e.g. in DeML(I=3,J=1) on CUB, the third OA learner achieves 60.1%, the second gets 58% and the first gets 54%, indicating that the more discernible the input is, the more discriminative embedding is. Furthermore, when fixing I to a certain value, decoupling each OA root-learner into multiple CA sub-learners will consistently improve the performances over both CUB and CARS, e.g. when I=1, the result (H) increases with J from 56.1% to 59.0% (77.9% to 82.5%) on CUB (CARS), and the similar improvements can be also observed when I=2 or 3, revealing that explicitly encouraging the diversity and capturing rich knowledge indeed improve the generalization of metric in ZSIR. This conclusion can also be demonstrated by Fig.<ref>, i.e. the training curve of DeML(I=1,J=8) is much lower than that of U512 or (U512+L_act), showing that the learned knowledge are not specially prepared for the training seen categories and the partial learning behavior is mitigated, while its testing curve outperforms both U512 and (U512+L_act), implying that learning diverse knowledge will further improve the generalization ability by reducing over-fitting. Worthy of mention is that in theory bigger J will be more helpful(e.g. J=8), however due to the limitation of GPU memory, during training when I=3, J is limited to 3.
In summary, the appropriate combination of (I,J) will significantly improve both the discrimination and generalization of the deep metric in an interpretable attention-based decoupling manner.





































Effect of adversary loss: Unlike OAMs that can automatically extract the object-attention regions via random-walk graph propagation, the CAMs should be extra constrained so as to explicitly encourage the learning of complementary attributes knowledge instead of the partial one. From Tab.<ref>, one can observe that without the constraint of L_adv, the performances of DeML(I=1,J=8) are drastically decreased over both CUB and CARS (i.e. from 59.0% to 52.0% on CUB and 82.5% to 68.2% on CARS), and the similarities between different CA learners are obviously increased from 0 to nearly 1, verifying that our adversary loss L_adv indeed encourages the discrepancies between CA learners (and CAMs) and reduces the neural redundancy, as a result, the diverse and rich information are captured such that the generalization ability is enhanced. Moreover, one can observe that even if there are 8 64-dim learners in (DeML(I=1,J=8) without L_adv), the performances are close to (U64+L_act), since these 8 learners are supervised independently and thus are all likely to focus on the same attributes knowledge that are the easiest one to reduce the training risk, and then have little differences with each other('sim' is close to 1), implying the redundant learners will not contribute much and the partial learning behavior will turn up easily without the diversity constraint.


Comparison with State-of-the-art: In order to highlight the significance of our decoupling idea for zero-shot retrieval, we compare DeML with some remarkable embedding works
[For fair comparison, we report the reproducing results of <cit.> when using the same training settings as us, e.g. batchsize and loss function.], over the popular used datasets CUB, CARS, Stanford Online Products and In-Shop Clothes. The comparison results are listed in Tab.<ref>. From these tables, one can observe that our baseline(U512) can only achieve the general performances, and the performances of traditional ideas of loss-designing and samples-mining are almost on par with each other, while, by explicitly intensifying both the discrimination and diversity within the deep metric, our DeML can significantly improve the performances over baseline model and outperforms other state-of-the-art methods by a noteworthy margin, demonstrating the necessity of our explicit enhancement for discrimination and generalization via the decoupling idea. Moreover, different from the listed ensemble methods <cit.>, DeML has clear objects of jointly mitigating the aforementioned issues that are vital to ZSIR, and thus can easily surpass them. Worthy of mention is that, we find DeML(I=2,J=4) is enough for Stanford Online Products dataset, since as in Fig.<ref> the second scale has been capable of localizing the discriminative regions.











































§ CONCLUSION



In this paper, we propose an interpretable DeML, a hybrid-attention based decoupling method, for zero-shot image retrieval task by explicitly intensifying the discrimination and generalization within the learned metric with the proposed OAMs and CAMs. Extensive experiments validate the necessity and effectiveness of our decoupling idea. 

Acknowledgments: This work was partially supported by the National Natural Science Foundation of China under Grant Nos. 61871052, 61573068, 61471048, and 61375031, and by the Beijing Nova Program under Grant No. Z161100004916088, DiDi GAIA Research Collaboration Initiative and BUPT Excellent Ph.D. Students Foundation CX2019307.
ieee



