Improving Generalization via Attribute Selection on Out-of-the-box Data
Xiaofeng Xu Ivor W. Tsang Chuancai Liu
cs.CV


1

 



Improving Generalization via Attribute Selection on Out-of-the-box Data

 

Xiaofeng Xu^ 1,  2, Ivor W. Tsang^ 2 and Chuancai Liu^ 1,  3

^ 1School of Computer Science and Engineering, Nanjing University of Science and Technology.

^ 2Centre for Artificial Intelligence, University of Technology Sydney.

^ 3Collaborative Innovation Center of IoT Technology and Intelligent Systems, Minjiang University.




Keywords: Zero-shot learning, attribute selection, out-of-the-box data, generalization error bound

empty
NC instructions

 





 Abstract 

Zero-shot learning (ZSL) aims to recognize unseen objects (test classes) given some other seen objects (training classes), by sharing information of attributes between different objects. Attributes are artificially annotated for objects and treated equally in recent ZSL tasks. However, some inferior attributes with poor predictability or poor discriminability may have negative impacts on the ZSL system performance. This paper first derives a generalization error bound for ZSL tasks. Our theoretical analysis verifies that selecting the subset of key attributes can improve the generalization performance of the original ZSL model, which utilizes all the attributes. Unfortunately, previous attribute selection methods are conducted based on the seen data, and their selected attributes have poor generalization capability to the unseen data, which is unavailable in the training stage of ZSL tasks. Inspired by learning from pseudo relevance feedback, this paper introduces the out-of-the-box data, which is pseudo data generated by an attribute-guided generative model, to mimic the unseen data. After that, we present an iterative attribute selection (IAS) strategy which iteratively selects key attributes based on the out-of-the-box data. Since the distribution of the generated out-of-the-box data is similar to the test data, the key attributes selected by IAS can be effectively generalized to test data. Extensive experiments demonstrate that IAS can significantly improve existing attribute-based ZSL methods and achieve state-of-the-art performance.




§ INTRODUCTION

With the rapid development of machine learning technologies, especially the rise of deep neural network, visual object recognition has made tremendous progress in recent years <cit.>. These recognition systems even outperform humans when provided with a massive amount of labeled data. However, it is expensive to collect sufficient labeled samples for all natural objects, especially for the new concepts and many more fine-grained subordinate categories <cit.>. Therefore, how to achieve an acceptable recognition performance for objects with limited or even no training samples is a challenging but practical problem <cit.>. Inspired by human cognition system that can identify new objects when provided with a description in advance <cit.>, zero-shot learning (ZSL) has been proposed to recognize unseen objects with no training samples <cit.>. Since labeled sample is not given for the target classes, we need to collect some source classes with sufficient labeled samples and find the connection between the target classes and the source classes.

As a kind of semantic representation, attributes are widely used to transfer knowledge from the seen classes (source) to the unseen classes (target) <cit.>. Attributes play a key role in sharing information between classes and govern the performance of zero-shot classification. In previous ZSL works, all the attributes are assumed to be effective and treated equally. However, as pointed out in <cit.>, different attributes have different properties, such as the distributive entropy and the predictability. The attributes with poor predictability or poor discriminability may have negative impacts on the ZSL system performance. The poor predictability means that the attributes are hard to be correctly recognized from the feature space, and the poor discriminability means that the attributes are weak in distinguishing different objects. Hence, it is obvious that not all the attributes are necessary and effective for zero-shot classification.

Based on these observations, selecting the key attributes, instead of using all the attributes, is significant and necessary for constructing ZSL models. <cit.> proposed the zero-shot learning with attribute selection (ZSLAS) model, which selects attributes by measuring the distributive entropy and the predictability of attributes based on the training data. ZSLAS can improve the performance of attribute-based ZSL methods, while it suffers from the drawback of generalization. Since the training classes and the test classes are disjoint in ZSL tasks, the training data is bounded by the box cut by attributes (illustrated in Figure <ref>). Therefore, the attributes selected based on the training data have poor generalization capability to the unseen test data.



To address the drawback, this paper derives a generalization error bound for ZSL problem. Since attributes for ZSL task is literally like the codewords in the error correcting output code (ECOC) model <cit.>, we analyze the bound from the perspective of ECOC. Our analyses reveal that the key attributes need to be selected based on the data which is out of the box (i.e. the distribution of the training classes). Considering that test data is unavailable during the training stage for ZSL tasks, inspired by learning from pseudo relevance feedback <cit.>, we introduce the out-of-the-box[The out-of-the-box data is generated based on the training data and the attribute representation without extra information, which follows the standard zero-shot learning setting.] data to mimic the unseen test classes. The out-of-the-box data is generated by an attribute-guided generative model using the same attribute representation as the test classes. Therefore, the out-of-the-box data has a similar distribution to the test data. 

Guided by the performance of ZSL model on the out-of-the-box data, we propose a novel iterative attribute selection (IAS) model to select the key attributes in an iterative manner. 
Figure <ref> illustrates the procedures of the proposed ZSL with iterative attribute selection (ZSLIAS).
Unlike the previous ZSLAS that uses training data to select attributes at once, our IAS first generates out-of-the-box data to mimic the unseen classes, and subsequently iteratively selects key attributes based on the generated out-of-the-box data. During the test stage, selected attributes are employed as a more efficient semantic representation to improve the original ZSL model. By adopting the proposed IAS, the improved attribute embedding space is more discriminative for the test data, and hence improves the performance of the original ZSL model.



The main contributions of this paper are summarized as follows: 


	
  * We present a generalization error analysis for ZSL problem. Our theoretical analyses prove that selecting the subset of key attributes can improve the generalization performance of the original ZSL model which utilizes all the attributes.
	
  * Based on our theoretical findings, we propose a novel iterative attribute selection strategy to select key attributes for ZSL tasks.
	
  * Since test data is unseen during the training stage for ZSL tasks, we introduce the out-of-the-box data to mimic test data for attribute selection. Such data generated by a designed generative model has a similar distribution to the test data. Therefore, attributes selected based on the out-of-the-box data can be effectively generalized to the unseen test data.
	
  * Extensive experiments demonstrate that IAS can effectively improve the attribute-based ZSL model and achieve state-of-the-art performance. 


The rest of the paper is organized as follows. Section <ref> reviews related works. Section <ref> gives the preliminary and motivation. Section <ref> presents the theoretical analyses on generalization bound for attribute selection. Section <ref> proposes the iterative attribute selection model.  Experimental results are reported in Section <ref>. Conclusion is drawn in Section <ref>.




§ RELATED WORKS


In this section, we review some related works on zero-shot learning, attribute selection and deep generative models.



 §.§ Zero-shot Learning

ZSL can recognize new objects using attributes as the intermediate semantic representation. Some researchers adopt the probability-prediction strategy to transfer information. <cit.> proposed a popular baseline, i.e. direct attribute prediction (DAP). DAP learns probabilistic attribute classifiers using the seen data and infers the label of the unseen data by combining the results of pre-trained classifiers. 

Most recent works adopt the label-embedding strategy that directly learns a mapping function from the input features space to the semantic embedding space. 
One line of works is to learn linear compatibility functions. For example, <cit.> presented an attribute label embedding (ALE) model which learns a compatibility function combined with ranking loss. <cit.> proposed an approach that models the relationships among features, attributes and classes as a two linear layers network. Another direction is to learn nonlinear compatibility functions. <cit.> presented a nonlinear embedding model that augments bilinear compatibility model by incorporating latent variables. <cit.> proposed a first general Kronecker product kernel-based learning model for ZSL tasks. In addition to the classification task, <cit.> proposed an attribute network for zero-shot hashing retrieval task.



 §.§ Attribute Selection

Attributes, as a kind of popular semantic representation of visual objects, can be the appearance, a part or a property of objects <cit.>. For example, object elephant has the attribute big and long nose, object zebra has the attribute striped. Attributes are widely used to transfer information to recognize new objects in ZSL tasks <cit.>. As shown in Figure <ref>, using attributes as the semantic representation, data of different categories locates in different boxes bounded by the attributes. Since the attribute representation of the seen classes and the unseen class are different, the boxes with respect to the seen data and the unseen data are disjoint.

In previous ZSL works, all the attributes are assumed to be effective and treated equally. However, as pointed out in <cit.>, not all the attributes are effective for recognizing new objects. Therefore, we should select the key attributes to improve the semantic presentation. <cit.> proposed a novel greedy algorithm which selects attributes based on their discriminating power and reliability. <cit.> proposed to select attributes by measuring the distributive entropy and the predictability of attributes based on the training data. In short, previous attribute selection models are conducted based on the training data, which makes the selected attributes have poor generalization capability to the unseen test data. While our IAS iteratively selects attributes based on the out-of-the-box data which has a similar distribution to the test data, and thus the key attributes selected by our model can be more effectively generalized to the unseen test data.



 §.§ Attribute-guided Generative Models

Deep generative models <cit.> aim to estimate the joint distribution p (y; x) of samples and labels, by learning the class prior probability p (y) and the class-conditional density p (x | y) separately. The generative model can be extended to a conditional generative model if the generator is conditioned on some extra information, such as attributes in the proposed method. <cit.> introduced a conditional version of generative adversarial nets, i.e. CGAN, which can be constructed by simply feeding the data label. CGAN is conditioned on both the generator and discriminator and can generate samples conditioned on class labels. Conditional Variational Autoencoder (CVAE) <cit.>, as an extension of Variational Autoencoder, is a deep conditional generative model for structured output prediction using Gaussian latent variables. We modify CVAE with the attribute representation to generate out-of-the-box data for the attribute selection.




§ PRELIMINARY AND MOTIVATION




 §.§ ZSL Task Formulation

We consider zero-shot learning as a task that recognizes unseen classes which have no labeled samples available. Given a training set D_s={ ( x_n,y_n ),n= 1,...,N_s}, the task of traditional ZSL is to learn a mapping f:𝒳→𝒴 from the image feature space to the label embedding space, by minimizing the following regularized empirical risk:

    L ( y,f ( x;𝐖 )  ) = 1/N_s∑_n=1^N_sl ( y_n,f ( x_n;𝐖 )  )+Ω ( 𝐖 ),

where l ( · ) is the loss function, which can be square loss 1/2(f(x)-y)^2, logistic loss log(1+exp(-yf(x))) or hinge loss max(0,1-yf(x)). 𝐖 is the parameter of mapping f, and Ω ( · ) is the regularization term.

The mapping function f is defined as follows:

    f ( x;𝐖 )=max_y∈𝒴 F ( x,y;𝐖 ),

where the function F:𝒳×𝒴→ℛ is the bilinear compatibility function to associate image features and label embeddings defined as follows:

    F ( x,y;𝐖 )=θ ( x  )^T𝐖φ ( y  ),

where  θ ( x  ) is the image features, φ ( y  ) is the label embedding (i.e. attribute representation).

We summarize some frequently used notations in Table <ref>.





 §.§ Interpretation of ZSL Task

In traditional ZSL models, all the attributes are assumed to be effective and treated equally. While in previous works, some researchers pointed out that not all the attributes are useful and significant for zero-shot classification <cit.>. To the best of our knowledge, there is no theoretical analysis for the generalization performance of ZSL tasks, let alone selecting informative attributes for unseen classes. To fill in this gap, we first derive the generalization error bound for ZSL models. 

The intuition of our theoretical analysis is to simply treat the attributes as a kind of error correcting output codes, then the prediction of ZSL tasks can be deemed as the assignment of class labels with respective pre-defined ECOC, which is the closest to the predicted ECOC problem <cit.>. Based on this novel interpretation, we derive a theoretical generalization error bound of ZSL model as shown in Section <ref>. From the generalization bound analyses, we find that the discriminating power of attributes governs the performance of the ZSL model.



 §.§ Deficiency of ZSLAS

Some attribute selection works have been proposed in recent years. <cit.> proposed the ZSLAS model that selects attributes based on the distributive entropy and the predictability of attributes using training data. Simultaneously considering the ZSL model loss function and attribute properties in a joint optimization framework, they selected attributes by minimizing the following loss function:

    L( y,f( x;𝐬,𝐖 ) ) =    1/N_s∑_n=1^N_s{l_ZSL ( y_n,f ( x_n;𝐬,𝐖 )  ) 
        + α l_p(θ(x_n),φ(y_n);𝐬) - β l_v(θ(x_n),μ;𝐬)},

where 𝐬 is the weight vector of the attributes which will be further used for attribute selection. θ(·) is the attribute classifier, φ(y_n) is the attribute representation, μ is an auxiliary parameter. l_ZSL is the model based loss function for ZSL, i.e. l(·) as defined in Eq. (<ref>). l_p is the attribute prediction loss which can be defined based on specific ZSL models and l_v is the loss of variance which measures the distributive entropy of attributes <cit.>. After getting the weight vector 𝐬 by optimizing Eq. (<ref>), attributes can be selected according to 𝐬 and then be used to construct ZSL model.

From our theoretical analyses in Section <ref>, ZSLAS can improve the original ZSL model to some extent <cit.>. However, ZSLAS suffers from a drawback that the attributes are selected based on the training data.
Since the training and test classes are disjoint in ZSL tasks, 
it is difficult to measure the quality and contribution of attributes regarding discriminating the unseen test classes. Thus, the selected attributes by ZSLAS have poor generalization capability to the test data due to the domain shift problem.




 §.§ Definition of Out-of-the-box

Since previous attribute selection models are conducted based on the bounded in-the-box data, the selected attributes have poor generalization capability to the test data. However, the test data is unavailable during the training stage.
Inspired by learning from pseudo relevance feedback <cit.>,
we introduce the pseudo data, which is outside the box of the training data, to mimic test classes to guide the attribute selection. Considering that the training data is bounded in the box by attributes, we generate the out-of-the-box data using an attribute-guided generative model. Since the out-of-the-box data is generated based on the same attribute representation as test classes, the box of the generated data will overlap with the box of the test data. And consequently, the key attributes selected by the proposed IAS model based on the out-of-the-box data can be effectively generalized to the unseen test data.




§ GENERALIZATION BOUND ANALYSIS


In this section, we first derive the generalization error bound of the original ZSL model and then analyze the bound changes after attribute selection. In previous works, some generalization error bounds have been presented for the ZSL task. <cit.> transformed ZSL problem to the domain adaptation problem and then analyzed the risk bounds for domain adaptation. <cit.> considered ZSL problem as a specific setting of pairwise learning and analyzed the bound by the kernel ridge regression model. However, these bound analysis are not suitable for ZSL model due to their assumptions. In this work, we derive the generalization bound from the perspective of ECOC model, which is more similar to the ZSL problem.



 §.§ Generalization Error Bound of ZSL

Zero-shot classification is an effective way to recognize new objects which have no training samples available. The basic framework of ZSL model is using attribute representation as the bridge to transfer knowledge from seen objects to unseen objects. To simplify the analysis, we consider ZSL as a multi-class classification problem. Therefore, ZSL task can be addressed via an ensemble method which combines many binary attribute classifiers. Specifically, we pre-trained a binary classifier for each attribute separately in the training stage. To classify a new sample, all the attribute classifiers are evaluated to obtain an attribute codeword (a vector in which each element represents the output of an attribute classifier). Then we compare the predicted codeword to the attribute representations of all the test classes to retrieve the label of the test sample.

To analyze the generalization error bound of ZSL, we first define some distances in the attribute space, and then present a proposition of the error correcting ability of attributes.


	
	Given the attribute matrix 𝐀 for associating labels and attributes, let 𝐚_i, 𝐚_j denote the attribute representation of label y_i and y_j in matrix 𝐀 with length N_a, respectively. Then the generalized attribute distance between 𝐚_i and 𝐚_j can be defined as 
	
    d(𝐚_i,𝐚_j) = ∑_m=1^N_aΔ (𝐚_i^(m),𝐚_j^(m)),

	where N_a is the number of attributes, 𝐚_i^(m) is the m^th element in the attribute representation 𝐚_i of the label y_i. Δ (𝐚_i^(m),𝐚_j^(m)) is equal to 1 if 𝐚_i^(m)≠𝐚_j^(m), 0 otherwise.


We further define the minimum distance between any two attribute representations in the attribute space.


	
	The minimum attribute distance τ of matrix 𝐀 is the minimum distance between any two attribute representations 𝐚_i and 𝐚_j as follows:
	
    τ = min_i≠ j d(𝐚_i,𝐚_j),     ∀  1≤ i,j ≤ N_a.



Given the definition of distance in the attribute space, we can prove the following proposition.


	
	Given the label-attribute correlation matrix 𝐀 and a vector of predicted attribute representation f(x) for an unseen test sample x with known label y. If x is incorrectly classified, then the distance between the predicted attribute representation f(x) and the correct attribute representation 𝐚_y is greater than half of the minimum attribute distance τ, i.e.
	
    d(f(x),𝐚_y) ≥τ/2.




	Suppose that the predicted attribute representation for test sample x with correct attribute representation 𝐚_y is f(x), and the sample x is incorrectly classified to the mismatched attribute representation 𝐚_r, where r∈𝒴_u ∖{y}. Then the distance between f(x) and 𝐚_y is greater than the distance between f(x) and 𝐚_r, i.e.,
	
    d(f(x),𝐚_y) ≥ d(f(x),𝐚_r).

	
	Here, the distance between attribute representation can be expanded as the element-wise summation based on Eq. (<ref>) as follows:
	
    ∑_m=1^N_aΔ (f^(m)(x),𝐚_y^(m)) ≥∑_m=1^N_aΔ (f^(m)(x),𝐚_r^(m)).

	
	Then, we have:
	
    d(f(x),𝐚_y)     = ∑_m=1^N_aΔ (f^(m)(x),𝐚_y^(m)) 
        = 1/2∑_m=1^N_a{Δ (f^(m)(x),𝐚_y^(m)) + Δ (f^(m)(x),𝐚_y^(m))}
       (1)≥1/2∑_m=1^N_a{Δ (f^(m)(x),𝐚_y^(m))  + Δ (f^(m)(x),𝐚_r^(m))}
       (2)≥1/2∑_m=1^N_aΔ (𝐚_y^(m),𝐚_r^(m))
        =  1/2d(𝐚_y,𝐚_r) (3)≥τ/2,

	where (1) follows Eq. (<ref>), (2) is based on the triangle inequality of distance metric <cit.> and (3) follows Eq. (<ref>).


From Proposition <ref>, we can find that, the predicted attribute representation is not required to be exactly the same as the ground truth for each unseen test sample. As long as the distance is less than τ/2, ZSL models can correct the error committed by some attribute classifiers and make an accurate prediction.

Based on the Proposition of error correcting ability of attributes, we can derive the theorem of generalization error bound for ZSL.


	
	Given N_a attribute classifiers, f^(1), f^(2), ...,f^(N_a), trained on training set D_s with label-attribute matrix 𝐀, the generalization error rate for the attribute-based ZSL model is upper bounded by
	
    2N_aB̅/τ,

	where B̅=1/N_a∑_m=1^N_aB_m and B_m is the upper bound of the prediction loss for the m^th attribute classifier f^(m).



	According to Proposition <ref>, for any incorrectly classified test sample x with label y, the distance between the predicted attribute representation f(x) and the true attribute representation 𝐚_y is greater than τ/2, i.e.,
	
    d(f(x),𝐚_y) = ∑_m=1^N_aΔ (f^(m)(x),𝐚_y^(m)) ≥τ/2.

	
	Let k be the number of incorrect image classifications for unseen test dataset D_u= {( x_i,y_i  ),i= 1,...,N_u}, we can obtain:
	
    kτ/2   ≤∑_i=1^N_u∑_m=1^N_aΔ(f^(m)(x_i),𝐚_y_i^(m)) 
       ≤∑_i=1^N_u∑_m=1^N_aB_m = N_u N_a B̅,

	where B̅=1/N_a∑_m=1^N_aB_m and B_m is the upper bound of attribute prediction loss.
	
	Hence, the generalized error rate k/N_u is bounded by 2N_aB̅/τ.



	
	From Theorem <ref>, we can find that the generalization error bound of the attribute-based ZSL model depends on the number of attributes N_a, minimum attribute distance τ and average prediction loss B̅ for all the attribute classifiers. According to the Definition <ref> and <ref>, the minimum attribute distance τ is positively correlated to the number of attributes N_a. Therefore, the generalization error bound is mainly affected by the average prediction loss B̅. Intuitively, the inferior attributes with poor predictability cause greater prediction loss B̅, and consequently, these attributes will have negative effect on the ZSL performance and increase the generalization error rate.




 §.§ Improvement of Generalization after Attribute Selection

It has been proven that the generalization error bound of ZSL model is affected by the average prediction loss B̅ in the previous section. In this section, we will prove that attribute selection can reduce the average prediction loss B̅, and consequently reduce the generalization error bound of ZSL from the perspective of PAC-style <cit.> analysis.

	
	Given N_a attribute classifiers, to obtain an attribute classifier with (1-δ) probability that has at most k_a incorrect predicted attributes, the PAC bound D of the attribute-based ZSL model is:
	
    D ∝N_a/k_a[4 log(2/δ)+8(d+1)log(13N_a/k_a)],

	where d is the dimension of the image features.




	
	Here, k_a/N_a is the tolerable prediction error rate of attribute classifiers. According to the definition of the average attribute prediction loss B̅, it is obvious that the ZSL model with smaller B̅ could tolerate a greater k_a/N_a. From Lemma <ref>, we can find that the PAC bound D is monotonically increasing with respect to N_a/k_a. Hence, the PAC bound D decreases when the N_a/k_a decreases, and consequently the average prediction loss B̅ decreases.



	
	Suppose that the PAC bound of the attribute-based ZSL model is D. The probability of the test error distancing from an upper bound is given by:
	
    p(e_ts≤ e_tr +√(1/N_s[D(log(2N_s/D)+1)-log(η/4)])) = 1-η,

	where N_s is the size of the training set, 0 ≤η≤ 1, and e_ts,  e_tr are the test error and the training error respectively.



	
	From Lemma <ref>, we can find that the PAC bound can affect the probabilistic upper bound on the test error. Specifically, to obtain a high probability with small test error, the PAC bound should be small. In other words, the model with smaller PAC bound would have a smaller test error bound.



	
	For the attribute-based ZSL model, attribute selection can decrease the generalization error bound.



	In attribute selection, the key attributes are selected by minimizing the loss function in Eq. (<ref>) on the out-of-the-box data. Since the generated out-of-the-box data has a similar distribution to the test data, the test error of ZSL will decrease after attribute selection, i.e. ZSLIAS has a smaller test error bound than the original ZSL model. Therefore, we can infer that ZSLIAS has a smaller PAC bound based on Remark <ref>. According to Remark <ref>, we can infer that the average prediction error B̅ decreases after attribute selection. As a consequence, the generalization error bound of ZSLIAS is smaller than the original ZSL model based on Remark <ref>.


From Proposition <ref>, we can observe that the generalization error of ZSL model will decrease after adopting the proposed IAS. In other words, ZSLIAS have a smaller classification error rate comparing to the original ZSL method when generalizing to the unseen test data.




§ IAS WITH OUT-OF-THE-BOX DATA


Motivated by the generalization bound analyses, we select the key attributes based on the out-of-the-box data. In this section, we first present the proposed iterative attribute selection model. Then, we introduce the attribute-guided generative model designed to generate the out-of-the-box data. The complexity analysis of IAS is given at last.



 §.§ Iterative Attribute Selection Model

Inspired by the idea of iterative machine teaching <cit.>, we propose a novel iterative attribute selection model that iteratively selects attributes based on the generated out-of-the-box data. Firstly, we generate the out-of-the-box data to mimic test classes by an attribute-based generative model. Then, the key attributes are selected in an iterative manner based on the out-of-the-box data. After obtaining the selected attributes, we can consider them as a more efficient semantic representation to improve the original ZSL model. 

Suppose given the generated out-of-the-box data D_g = {( x_n,y_n ), n= 1,...,N_g}, we can combine the empirical risk in Eq. (<ref>) with the attribute selection model. Then the loss function is  rewritten as follows:

    L ( y,f ( x;𝐬,𝐖 )  )  = 1/N_g∑_n=1^N_gl ( y_n,f ( x_n;𝐬,𝐖 )  )+Ω ( 𝐖 ),

where 𝐬∈ ( 0,1  )^N_a is the indicator vector for the attribute selection, in which s_i=1 if the i^th attribute is selected or 0 otherwise. N_a is the number of all the attributes.

Correspondingly, the mapping function f in Eq. (<ref>) and the compatibility function F in Eq. (<ref>) can be rewritten as follows:

    f ( x;𝐬,𝐖 )=max_y∈𝒴 F ( x,y;𝐬,𝐖 ),


    F ( x,y;𝐬,𝐖 )=θ ( x  )^T𝐖 ( 𝐬∘φ ( y  )  ),

where ∘ is element-wise product operator (Hadamard product), 𝐬 is the selection vector defined in Eq. (<ref>).

To solve the optimization problem in Eq. (<ref>), we need to specify the choice of the loss function l ( · ). The loss function in Eq. (<ref>) for single sample (x_n,y_n) is expressed as follows <cit.>:

    l ( y_n,f (  ( x_n;𝐬,𝐖  )  )  )
       = ∑_y∈𝒴_gr_ny [  ( y_n,y  )+F ( x_n,y;𝐬,𝐖  ) -F ( x_n,y_n;𝐬,𝐖  ) ]_+
       = ∑_y∈𝒴_gr_ny [ ( y_n,y )+θ ( x_n )^T𝐖( 𝐬∘φ (y) )-θ ( x_n )^T𝐖( 𝐬∘φ ( y_n ) )   ]_+,

where 𝒴_g is the label of generated out-of-the-box data, which is the same as  𝒴_u. 
 (y_n;y)=0 if y_n = y, 1 otherwise. r_ny∈ [0,1] is the weight defined in specific ZSL methods. 

Since the dimension of the optimal attribute subset (i.e. l_0-norm of 𝐬) is agnostic, finding the optimal 𝐬 is a NP-Complete <cit.> problem. Therefore, inspired by the idea of iterative machine teaching <cit.>, we adopt the greedy algorithm <cit.> to optimize the loss function in an iterative manner. Eq. (<ref>) gets updated during each iteration as follows:

    L^t+1=1/N_g∑_n=1^N_g    l^t+1( y_n,f ( x_n;𝐬^t+1,𝐖^t+1  )  )+Ω  ( 𝐖^t+1  ), 
        s.t. ∑_s_i∈𝐬^t+1s_i=t+1 ,
         ∑_s_j∈ ( 𝐬^t+1-𝐬^t )s_j=1.

The constraints on 𝐬 ensure that 𝐬^t updates one element (from 0 updates to 1) during each iteration, which indicates that only one attribute is selected each time. 𝐬^0 is the initial vector of all 0's.

Correspondingly, the loss function in Eq. (<ref>) for single sample (x_n,y_n) gets updated during each iteration as follows: 

    l^t+1 = ∑_y∈𝒴_g r_ny [     ( y_n,y  )+θ  ( x_n  )^T𝐖^t+1 ( 𝐬^t+1∘φ  ( y  )  )  
          -θ  ( x_n  )^T𝐖^t+1 ( 𝐬^t+1∘φ  ( y_n  )  )   ]_+.

Here l^t+1 subjects to the same constrains as Eq. (<ref>).

To minimize the loss function in Eq. (<ref>), we can alternatively optimize 𝐖^t+1 and 𝐬^t+1 by optimizing one variable while fixing the other one. In each iteration, we firstly optimize 𝐖^t+1 via the gradient descent algorithm <cit.>. The gradient of Eq. (<ref>) is calculated as follows:

    ∂ L^t+1/∂𝐖^t+1 = 1/N_g∑_n=1^N_g∂ l^t+1/∂𝐖^t+1 + 1/2α𝐖^t+1,

where

    ∂ l^t+1/∂𝐖^t+1=∑_y∈𝒴_gr_nyθ (x_n)^T(𝐬^t∘ (φ (y)-φ (y_n))),

where α is the regularization parameter.

After updating 𝐖^t+1, we can traverse all the elements equal to 0 in 𝐬^t, and turn them into 1 respectively. Then 𝐬^t+1 is updated by the optimal 𝐬^t+1 which achieves the minimal loss of Eq. (<ref>):

    𝐬^t+1 = min_𝐬^t+11/N_g∑_n=1^N_g  l^t+1( y_n,f ( x_n;𝐬^t+1,𝐖^t+1  )  )+Ω  ( 𝐖^t+1  ),



When iterations end and 𝐬 is obtained, we can easily get the subset of key attributes by selecting the attributes corresponding to the elements equal to 1 in the selection vector 𝐬. 

The procedure of the proposed IAS model is given in Algorithm <ref>.





 §.§ Generation of Out-of-the-box Data

In order to select the discriminative attributes for test classes, we should do attribute selection on the test data. Since the training data and the test data are located in the different boxes bounded by the attributes, we adopt an attribute-based generative model <cit.> to generate out-of-the-box data to mimic test classes. Comparing to the ZSLAS, the key attributes selected by IAS based on the out-of-the-box data can be more efficiently generalized to test data.

Conditional variational autoencoder (CVAE) <cit.> is a conditional generative model in which the latent codes and generated data are both conditioned on some extra information. In this work, we propose the attribute-based variational autoencoder (AVAE), a special version of CVAE with tailor-made attributes, to generate the out-of-the-box data.

VAE <cit.> is a directed graphical model with certain types of latent variables. The generative process of VAE is as follows: a set of latent codes z is generated from the prior distribution p(z), and the data x is generated by the generative distribution p(x|z) conditioned on z: z ∼  p(z), x ∼ p(x|z). The empirical objective of VAE is expressed as follows <cit.>:

    L_VAE(x )=-KL(q(z|x)∥ p(z))+1/L∑_l=1^Llog p(x|z^(l)),

where z^(l)=g(x,ϵ ^(l)), ϵ ^(l)∼𝒩(0,𝐈). q(z|x) is the recognition distribution which is reparameterized with a deterministic and differentiable function g(·,·) <cit.> . KL denotes the Kullback-Leibler divergence <cit.> between the incorporated distributions. L is the number of samples.

Combining with the condition, i.e. the attribute representation of labels, the empirical objective of the AVAE is defined as follows:

    L_AVAE(x,φ (y))=-KL(q(z|x,φ (y))∥ p(z|φ (y)))  +1/L∑_l=1^Llog p(x|φ (y),z^(l)),

where z^(l)=g(x,φ (y),ϵ ^(l)), φ ( y  ) is the attribute representation of label y.

In the encoding stage, for each training data point x^(i), we estimate the q(z^(i)|x^(i), 
φ (y^(i))) = Q(z) using the encoder. In the decoding stage, after inputting the concatenation of the z̃ sampled from the Q(z) and the attribute representation φ (y_u), the decoder will generate a new sample x_g with the same attribute representation as the unseen class φ (y_u).

The procedure of AVAE is illustrated in Figure <ref>. At training time, the attribute representation (of training classes) whose image is being fed in is provided to the encoder and decoder. To generate an image of a particular attribute representation (of test classes), we can just feed this attribute vector along with a random point in the latent space sampled from a standard normal distribution. The system no longer relies on the latent space to encode what object you are dealing with. Instead, the latent space encodes attribute information. Since the attribute representations of test classes are fed into the decoder at generating stage, the generated out-of-the-box data D_g has a similar distribution to the test data.






 §.§ Complexity Analysis

Suppose that there are N_u unseen samples belonging to L test classes, and the number of all the attributes is N_a. The complexity of original ZSL model is 𝒪_ZSL∼𝒪(N_uN_aL^2). For the proposed ZSLIAS, the complexity of training stage is 𝒪_ZSLIAS∼ N_a(N_a+1)/2·𝒪_ZSL, i.e. 𝒪(N_uN_a^3L^2), and the complexity of test stage is equal to  𝒪_ZSL, i.e. 𝒪(N_uN_aL^2).




§ EXPERIMENTS


To evaluate the performance of the proposed iterative attribute selection model, extensive experiments are conducted on four standard datasets with ZSL setting. In this section, we first compare the proposed approach with the state-of-the-art, and then give detailed analyses.



 §.§ Experimental Settings



  §.§.§ Dataset
 We conduct experiments on four standard ZSL datasets: (1) Animal with Attribute (AwA) <cit.>, (2) attribute-Pascal-Yahoo (aPY) <cit.>, (3) Caltech-UCSD Bird 200-2011 (CUB) <cit.>, and (4) SUN Attribute Database (SUN) <cit.>. The overall statistic information of these datasets is summarized in Table <ref>.





  §.§.§ Dataset Split
 Zero-shot learning assumes that training classes and test classes are disjoint. Actually, ImageNet, the dataset exploited to extract image features via deep neural networks, may include some test classes. Therefore, <cit.> proposed a new dataset split (PS) ensuring that none of the test classes appears in the dataset used to train the extractor model. In this paper, we evaluate the proposed model using both splits, i.e., the original standard split (SS) and the proposed split (PS).



  §.§.§ Image Feature
 Deep neural network feature is extracted for the experiments. Image features are extracted from the entire images for AwA, CUB and SUN datasets, and from bounding boxes mentioned in <cit.> for aPY dataset, respectively. The original ResNet-101 <cit.> pre-trained on ImageNet with 1K classes is used to calculate 2048-dimensional top-layer pooling units as image features.



  §.§.§ Attribute Representation
 Attributes are used as the semantic representation to transfer information from training classes to test classes. We use 85, 64, 312 and 102-dimensional continuous value attributes for AwA, aPY, CUB and SUN datasets, respectively.



  §.§.§ Evaluation protocol
 
Unified dataset splits shown in Table <ref> are used for all the compared methods to get fair comparison results. Since the dataset is not well balanced with respect to the number of images per class <cit.>, we use the mean class accuracy, i.e. per-class averaged top-1 accuracy, as the criterion of assessment. Mean class accuracy is calculated as follows: 

    acc=1/L∑_y∈𝒴_u#correct  predictions  in  y/#samples  in  y,

where L is the number of test classes, 𝒴_u is the set comprised of all the test labels.




 §.§ Comparison with the State-of-the-Art

To evaluate the efficiency of the proposed iterative attribute selection model, we modify several latest ZSL baselines by the proposed IAS and compare them with the state-of-the-art.

We modify seven representative ZSL baselines to evaluate the IAS model, including three popular ZSL baselines (i.e. DAP <cit.>, LatEm <cit.> and SAE <cit.>) and four latest ZSL baselines (i.e. MFMR <cit.>, GANZrl <cit.>, fVG <cit.> and LLAE <cit.>).

The improvement achieved on these ZSL baselines is summarized in Table <ref>. It can be observed that IAS can significantly improve the performance of attribute-based ZSL methods. Specifically, the mean accuracies of these ZSL methods on four datasets (i.e. AwA, aPY, CUB and SUN) are increased by 11.09%, 15.97%, 9.10%, 5.11%, respectively (10.29% on average) after using IAS. For DAP on AwA and aPY datasets, LatEm on AwA dataset, IAS can improve their accuracy by greater than 20%, which demonstrates that IAS can significantly improve the performance of ZSL models. Interestingly, SAE performs badly on aPY and CUB datasets, while the accuracy rises to an acceptable level (from 8.33% to 38.53%, and from 24.65% to 42.85%, respectively) by using IAS. Even though the performance of state-of-the-art baselines is pretty well, IAS can still improve them to some extent (5.48%, 3.24%, 2.80% and 3.64% on average for MFMR, GANZrl, fVG and LLAE respectively). These results demonstrate that the proposed iterative attribute selection model makes sense and can effectively improve existing attribute-based ZSL methods. This also proves the necessity and effectiveness of attribute selection for ZSL tasks.



As a similar work to ours, ZSLAS selects attributes based on the distributive entropy and the predictability of attributes. Thus, we compare the improvement of IAS and ZSLAS on DAP and LatEm, respectively. In Table <ref>, it can be observed that ZSLAS can improve existing ZSL methods, while IAS can improve them by a greater level (2.15% vs 10.61% on average). Compared to ZSLAS, the advantages of ZSLIAS can be interpreted in two aspects. Firstly, ZSLIAS selects attributes in an iterative manner, hence it can select a more optimal subset of key attributes than ZSLAS that selects attributes at once. Secondly, ZSLAS is conducted based on the training data, while ZSLIAS is conducted based on the out-of-the-box data which has a similar distribution to the test data. Therefore, attributes selected by ZSLIAS is more applicable and discriminative for test data. Experimental results demonstrate the significant superiority of the proposed IAS model over previous attribute selection models.




 §.§ Detailed Analysis

In order to further understand the promising performance, we analyze the following experimental results in detail.




  §.§.§ Evaluation on the Out-of-the-box Data
 
In the first experiment, we evaluate the out-of-the-box data generated by a tailor-made attribute-based deep generative model. Figure <ref> shows the distribution of the out-of-the-box data and the real test data sampled from AwA dataset using t-SNE. Note that the out-of-the-box data in Figure <ref>(b) is generated only based on the attribute representation of unseen classes, and without extra information of any test images. It can be observed that the generated out-of-the-box data can capture a similar distribution to the real test data, which guarantees that the selected attributes can be effectively generalized to test data.



We also quantitatively evaluate the out-of-the-box data by calculating various distances between three distributions, i.e. the generated out-of-the-box data (𝒳_g), unseen test data (𝒳_u) and seen training data (𝒳_s), in pairs. Table <ref> shows the distribution distances measured by Wasserstein Distance <cit.>, KL Divergence <cit.>, Hellinger Distance <cit.> and Bhattacharyya Distance <cit.>, respectively. It is obvious that the distance between 𝒳_g and 𝒳_u is much less than the distance between 𝒳_u and 𝒳_s, which means that the generated out-of-the-box data has a similar distribution to the unseen test data compared to the seen data. Therefore, attributes selected based on the out-of-the-box data are more discriminative for test data comparing to attributes selected based on training data.



We illustrate some generated images of unseen classes (i.e. panda and seal) and annotate them the corresponding attribute representations as shown in Figure <ref>. Numbers in black indicate the attribute representations of the labels of real test images. Numbers in red and green are the correct and the incorrect attribute values of generated images, respectively. We can see that the generated images have the similar attribute representation as test images. Therefore, the tailor-made attribute-based deep generative model can generate the out-of-the-box data which captures a similar distribution to the unseen data.






  §.§.§ Effectiveness of IAS
 
In the second experiment, we compare the performance of three ZSL methods (i.e. DAP, LatEm and SAE) after using IAS on four datasets, respectively. The accuracies with respect to the number of selected attributes are shown in Figure <ref>. On AwA, aPY and SUN datasets, we can see that the performance of these three ZSL methods increases sharply when the number of selected attributes grows from 0 to about 20%, and then reaches the peak. These results suggest that only about a quarter of attributes are the key attributes which are necessary and effective to classify test objects. In Figure <ref>(b) and <ref>(f), there is an interesting result that SAE performs badly on aPY dataset with both SS and PS (the accuracy is less than 10%), while the performance is acceptable after using IAS (the accuracy is about 40%). These results demonstrate the effectiveness and robustness of IAS for ZSL tasks.



Furthermore, we modify DAP by using all the attributes (#84), using the selected attributes (#20) and using the remaining attributes (#64) after attribute selection, respectively. The resulting confusion matrices of these three variants evaluated on AwA dataset with proposed split setting are illustrated in Figure <ref>. The numbers in the diagonal area (yellow patches) of confusion matrices indicate the classification accuracy per class. It is obvious that IAS can significantly improve DAP performance on most of the test classes, and the accuracies on some classes nearly doubled after using IAS, such as horse, seal, and giraffe. Even though some objects are hard to be recognized by DAP, like dolphin (the accuracy of DAP is 1.6%), we can get an acceptable performance after using IAS (the accuracy of DAPIAS is 72.7%). The original DAP only performs better than IAS with regard to the object blue whale, this is because in the original DAP, most of the marine creatures (such as blue whale, walrus and dolphin) are classified as the blue whale, which increases the classification accuracy while also increasing the false positive rate. More importantly, the confusion matrix of DAPIAS contains less noise (i.e. smaller numbers in the side regions (white patches) of confusion matrices apart from the diagonal area) than DAP, which suggests that DAPIAS has less prediction uncertainties. In other words, adopting IAS can improve the robustness of attribute-based ZSL methods.



In Figure <ref>, the accuracy of using the selected attributes (71.88% on average) is significantly improved comparing to the accuracy of using all the attributes (46.23% on average), and the accuracy of using the remaining attributes (31.32% on average) is extremely terrible. These results suggest that the selected attributes are the key attributes for discriminating test data. The missing attributes are useless and even have a negative impact on the ZSL system. Therefore, it is obvious that not all the attributes are effective for ZSL tasks, and we should select the key attributes to improve performance.





  §.§.§ Interpretability of Selected Attributes
 
In the third experiment, we present the visualization results of attribute selection. We find that ZSL methods obtain the best performance when selecting about 20% attributes as shown in Figure <ref>. Therefore, we illustrate the top 20% key attributes selected by DAP, LatEm and SAE on four datasets in Figure <ref>. Three rows in each figure are DAP, LatEm and SAE from top to bottom, and yellow bars indicate the attributes which are selected by the corresponding methods. We can see that the attribute subsets selected by different ZSL methods are highly coincident for the same dataset, which demonstrates that the selected attributes are the key attributes for discriminating test data. Specifically, we enumerate the key attributes selected by three ZSL methods on AwA dataset in Table <ref>. Attributes in boldface indicate that they are simultaneously selected by all the three ZSL methods, and attributes in italics indicate that they are selected by any two of these three methods. It can be observed that 13 attributes (65%) are selected by all the three ZSL methods. These three attribute subsets selected by diverse ZSL models are very similar, which is another evidence that IAS is reasonable and useful for zero-shot classification.






§ CONCLUSION


We present a novel and effective iterative attribute selection model to improve existing attribute-based ZSL methods. In most of the previous ZSL works, all the attributes are assumed to be effective and treated equally. However, we notice that attributes have different predictability and discriminability for diverse objects. Motivated by this observation, we propose to select the key attributes to build ZSL model. Since training classes and test classes are disjoint in ZSL tasks, we introduce the out-of-the-box data to mimic test data to guide the progress of attribute selection. The out-of-the-box data generated by a tailor-made attribute-based deep generative model has a similar distribution to the test data. Hence, the attributes selected by IAS based on the out-of-the-box data can be effectively generalized to the test data. To evaluate the effectiveness of IAS, we conduct extensive experiments on four standard ZSL datasets. Experimental results demonstrate that IAS can effectively select the key attributes for ZSL tasks and significantly improve state-of-the-art ZSL methods.

In this work, we select the same attributes for all the unseen test classes. Obviously, this is not the global optimal solution to select attributes for diverse categories. In the future, we will consider a tailor-made attribute selection model that can select the special subset of key attributes for each test class.




§ ACKNOWLEDGMENTS

This work is supported in part by ARC under Grant LP150100671 and Grant DP180100106, in part by NSFC under Grant 61373063 and Grant 61872188, in part by the Project of MIIT under Grant E0310/1112/02-1, in part by the Collaborative Innovation Center of IoT Technology and Intelligent Systems of Minjiang University under Grant IIC1701, and in part by China Scholarship Council.


APA
100

urlstyle


[Airola et al.(2017)Airola, & Pahikkala]airola2018fast
Airola, A., & Pahikkala, T. (2017).
Fast Kronecker product kernel methods via generalized vec trick.
IEEE Transactions on Neural Networks and Learning Systems, 29(8), 3374 – 3387.

[Akata et al.(2015)Akata, Perronnin, Harchaoui, & Schmid]akata2016label
Akata, Z., Perronnin, F., Harchaoui, Z., & Schmid, C. (2015).
Label-embedding for image classification.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(7), 1425 – 1438.

[Beran(1977)]beran1977minimum
Beran, R. (1977).
Minimum Hellinger distance estimates for parametric models.
The Annals of Statistics, 5(3), 445 – 463.

[Bucher et al.(2017)Bucher, Herbin, & Jurie]bucher2017generating
Bucher, M., Herbin, S., & Jurie, F. (2017).
Generating visual representations for zero-shot classification.
In Proceedings of the IEEE International Conference on Computer Vision (pp. 2666-2673).

[Burges et al.(2005)Burges, Shaked, Renshaw, Lazier, Deeds, Hamilton, & Hullender]burges2005learning
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005).
Learning to rank using gradient descent.
In Proceedings of the 22nd International Conference on Machine learning (ICML-05) (pp. 89 – 96).

[Cheng et al.(2017)Cheng, Qiao, Wang, & Yu]cheng2018random
Cheng, Y., Qiao, X., Wang, X., & Yu, Q. (2017).
Random forest classifier for zero-shot learning based on relative attribute.
IEEE Transactions on Neural Networks and Learning Systems, 29(5), 1662 – 1674.

[Cormen et al.(2009)Cormen, Leiserson, Rivest, & Stein]cormen2009introduction
Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2009).
Introduction to Algorithms.
MIT press.

[Dietterich et al.(1994)Dietterich, & Bakiri]dietterich1994solving
Dietterich, T. G., & Bakiri, G. (1994).
Solving multiclass learning problems via error-correcting output codes.
Journal of Artificial Intelligence Research, 2), 263 – 286.

[Farhadi et al.(2009)Farhadi, Endres, Hoiem, & Forsyth]farhadi2009describing
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009, June).
Describing objects by their attributes.
In 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 1778 – 1785). IEEE.

[Garey et al.(1974)Garey, Johnson, & Stockmeyer]garey1974some
Garey, M. R., Johnson, D. S., & Stockmeyer, L. (1974, April). 
Some simplified NP-complete problems.
In Proceedings of the Sixth Annual ACM Symposium on Theory of Computing (pp. 47 – 63). ACM.

[Guo et al.(2018)Guo, Ding, Han, & Tang]guo2018zero
Guo, Y., Ding, G., Han, J., & Tang, S. (2018, April).
Zero-shot learning with attribute selection.
In Thirty-Second AAAI Conference on Artificial Intelligence.

[He et al.(2016)He, Zhang, Ren, & Sun]he2016deep
He, K., Zhang, X., Ren, S., & Sun, J. (2016).
Deep residual learning for image recognition.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770 – 778).

[Ji et al.(2019)Ji, Sun, Yu, Pang, & Han]ji2019attribute
Ji, Z., Sun, Y., Yu, Y., Pang, Y., & Han, J. (2019).
Attribute-guided network for cross-modal zero-shot hashing.
IEEE Transactions on Neural Networks and Learning Systems.

[Jiang et al.(2017)Jiang, Wang, Shan, Yang, & Chen]jiang2017learning
Jiang, H., Wang, R., Shan, S., Yang, Y., & Chen, X. (2017).
Learning Discriminative Latent Attributes for Zero-Shot Classification.
In Proceedings of the IEEE International Conference on Computer Vision (pp. 4223 – 4232).

[Kailath(1967)]kailath1967divergence
Kailath, T. (1967).
The divergence and Bhattacharyya distance measures in signal selection.
IEEE Transactions on Communication Technology, 15(1), 52 – 60.

[Kingma et al.(2013)Kingma, & Welling]kingma2013auto
Kingma, D. P., & Welling, M. (2013).
Auto-encoding variational bayes.
arXiv preprint arXiv:1312.6114.

[Kodirov et al.(2017)Kodirov, Xiang, & Gong]kodirov2017semantic
Kodirov, E., Xiang, T., & Gong, S. (2017).
Semantic autoencoder for zero-shot learning.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3174-3183).

[Kullback(1987)]kullback1987letter
Kullback, S. (1987).
Letter to the editor: The Kullback-Leibler distance.

[Lampert et al.(2013)Lampert, Nickisch, & Harmeling]lampert2014attribute
Lampert, C. H., Nickisch, H., & Harmeling, S. (2013).
Attribute-based classification for zero-shot visual object categorization.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3), 453 – 465.

[Li et al.(2019)Li, Jing, Lu, Zhu, Yang, & Huang]li2019zero
Li, J., Jing, M., Lu, K., Zhu, L., Yang, Y., & Huang, Z. (2019).
From Zero-Shot Learning to Cold-Start Recommendation.
In Thirty-Third AAAI Conference on Artificial Intelligence.

[Liu et al.(2014)Liu, Wiliem, Chen, & Lovell]liu2014automatic
Liu, L., Wiliem, A., Chen, S., & Lovell, B. C. (2014, August).
Automatic image attribute selection for zero-shot learning of object categories.
In 2014 22nd International Conference on Pattern Recognition (pp. 2619 – 2624). IEEE.

[Liu et al.(2017)Liu, Dai, Humayun, Tay, Yu, Smith, Rehg, & Song]liu2017iterative
Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., Rehg, J. M., & Song, L. (2017, August).
Iterative machine teaching.
In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 2149 – 2158). JMLR. org.

[Ma et al.(2017)Ma, Jia, Sun, Schiele, Tuytelaars, & Van Gool]ma2017pose
Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., & Van Gool, L. (2017).
Pose guided person image generation.
In Advances in Neural Information Processing Systems (pp. 406 – 416).

[Ma et al.(2017)Ma, Chang, Xu, Sebe, & Hauptmann]ma2017joint
Ma, Z., Chang, X., Xu, Z., Sebe, N., & Hauptmann, A. G. (2017).
Joint attributes and event analysis for multimedia event detection.
IEEE Transactions on Neural Networks and Learning Systems, 29(7), 2921 – 2930.

[Miao et al.(2016)Miao, Huang, & Zhao]miao2016topprf
Miao, J., Huang, J. X., & Zhao, J. (2016).
TopPRF: A probabilistic framework for integrating topic space into pseudo relevance feedback.
ACM Transactions on Information Systems (TOIS), 34(4), 22.

[Murphy(2004)]murphy2004big
Murphy, G. (2004).
The big book of concepts.
MIT press.

[Odena et al.(2017)Odena, Olah, & Shlens]odena2016conditional
Odena, A., Olah, C., & Shlens, J. (2017, August).
Conditional image synthesis with auxiliary classifier gans.
In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 2642 – 2651). JMLR. org.

[Palatucci et al.(2009)Palatucci, M., Pomerleau, Hinton, & Mitchell]palatucci2009zero
Palatucci, M., Pomerleau, D., Hinton, G. E., & Mitchell, T. M. (2009).
Zero-shot learning with semantic output codes.
In Advances in Neural Information Processing Systems (pp. 1410 – 1418).

[Patterson et al.(2012)Patterson, & Hays]patterson2012sun
Patterson, G., & Hays, J. (2012, June).
Sun attribute database: Discovering, annotating, and recognizing scene attributes.
In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 2751 – 2758). IEEE.

[Rocha et al.(2014)Rocha, & Goldenstein]rocha2014multiclass
Rocha, A., & Goldenstein, S. K. (2014).
Multiclass from binary: Expanding one-versus-all, one-versus-one and ecoc-based approaches.
IEEE Transactions on Neural Networks and Learning Systems, 25(2), 289 – 302.

[Romera-Paredes et al.(2015)Romera-Paredes, & Torr]romera2015embarrassingly
Romera-Paredes, B., & Torr, P. (2015, June).
An embarrassingly simple approach to zero-shot learning.
In International Conference on Machine Learning (pp. 2152 – 2161).

[Shen et al.(2018)Shen, Ji, Wang, Li, & Li]shen2018weakly
Shen, Y., Ji, R., Wang, C., Li, X., & Li, X. (2018).
Weakly supervised object detection via object-specific pixel gradient.
IEEE Transactions on Neural Networks and Learning Systems, 29(12), 5960 – 5970.

[Stock et al.(2018)Stock, Pahikkala, Airola, De Baets, & Waegeman]stock2018comparative
Stock, M., Pahikkala, T., Airola, A., De Baets, B., & Waegeman, W. (2018).
A comparative study of pairwise learning methods based on kernel ridge regression.
Neural Computation, 30(8), 2245 – 2283.

[Sohn et al.(2015)Sohn, Lee, & Yan]sohn2015learning
Sohn, K., Lee, H., & Yan, X. (2015).
Learning structured output representation using deep conditional generative models.
In Advances in Neural Information Processing Systems (pp. 3483 – 3491).

[Sun et al.(2017)Sun, Schiele, & Fritz]sun2017domain
Sun, Q., Schiele, B., & Fritz, M. (2017).
A domain based approach to social relation recognition.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3481 – 3490).

[Tong et al.(2018)Tong, Klinkigt, Chen, Cui, Kong, Murakami, & Kobayashi]tong2018adversarial
Tong, B., Klinkigt, M., Chen, J., Cui, X., Kong, Q., Murakami, T., & Kobayashi, Y. (2018, April).
Adversarial zero-shot learning with semantic augmentation.
In Thirty-Second AAAI Conference on Artificial Intelligence. (pp. 3483 – 3491).

[Valiant(1984)]valiant1984theory
Valiant, L. G. (1984, December).
A theory of the learnable.
In Proceedings of the Sixteenth Annual ACM Symposium on Theory of Computing (pp. 436 – 445). ACM.

[Vallender(1974)]vallender1974calculation
Vallender, S. (1974).
Calculation of the Wasserstein distance between probability distributions on the line.
Theory of Probability & Its Applications, 18(4), 784 – 786.

[Vapnik(2013)]vapnik2013nature
Vapnik, V. (2013).
The nature of statistical learning theory.
Springer Science & Business Media.

[Wah et al.(2011)Wah, Branson, Welinder, Perona, & Belongie]welinder2010caltech
Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011).
The caltech-ucsd birds-200-2011 dataset.

[Xian et al.(2016)Xian, Akata, Sharma, Nguyen, Hein, & Schiele]xian2016latent
Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., & Schiele, B. (2016).
Latent embeddings for zero-shot classification.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 69 – 77).

[Xian et al.(2018)Xian, Lampert, Schiele, & Akata]xian2018zero
Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2018).
Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly.
IEEE Transactions on Pattern Analysis and Machine Intelligence.

[Xian et al.(2019)Xian, Sharma, Schiele, & Akata]xian2019f
Xian, Y., Sharma, S., Schiele, B., & Akata, Z. (2019).
f-VAEGAN-D2: A feature generating framework for any-shot learning.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 10275 – 10284).

[Xu et al.(2017)Xu, Shen, Yang, Zhang, Shen, & Song]xu2017matrix
Xu, X., Shen, F., Yang, Y., Zhang, D., Shen, H., & Song, J. (2017).
Matrix tri-factorization with manifold regularizations for zero-shot learning.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3798-3807).

[Xu et al.(2019)Xu, Tsang, & Liu]xu2019complementary
Xu, X., Tsang, I. W., & Liu, C. (2019).
Complementary Attributes: A New Clue to Zero-Shot Learning.
IEEE Transactions on Cybernetics.

[Zheng et al.(2018)Zheng, Li, Yan, Tang, & Tan]zheng2018sparse
Zheng, Y., Li, S., Yan, R., Tang, H., & Tan, K. C. (2018).
Sparse temporal encoding of visual features for robust object recognition by spiking neurons.
IEEE Transactions on Neural Networks and Learning Systems, 29(12), 5823 – 5833.

[Zhou et al.(2019)Zhou, Fang, Zhang, Gong, Peng, Cao, & Goh]zhou2019learning
Zhou, J. T., Fang, M., Zhang, H., Gong, C., Peng, X., Cao, Z., & Goh, R. S. M. (2019).
Learning with annotation of various degrees.
IEEE Transactions on Neural Networks and Learning Systems..

[Zhou et al.(2019)Zhou, Tsang, Ho, & Muller]zhou2019n
Zhou, J. T., Tsang, I. W., Ho, S., & Muller, K. (2019).
N-ary decomposition for multi-class classification.
Machine Learning, 108(5), 809 – 830.


