Attribute Aware Pooling for Pedestrian Attribute Recognition
Kai Han Yunhe Wang Han Shu Chuanjian Liu Chunjing Xu Chang Xu
cs.CV cs.LG eess.IV


[
    
Hui Yu and Hailiang Liu

    December 30, 2023
=============================




This paper expands the strength of deep convolutional neural networks (CNNs) to the pedestrian attribute recognition problem by devising a novel attribute aware pooling algorithm. Existing vanilla CNNs cannot be straightforwardly applied to handle multi-attribute data because of the larger label space as well as the attribute entanglement and correlations. We tackle these challenges that hampers the development of CNNs for multi-attribute classification by fully exploiting the correlation between different attributes. The multi-branch architecture is adopted for fucusing on attributes at different regions. Besides the prediction based on each branch itself, context information of each branch are employed for decision as well. The attribute aware pooling is developed to integrate both kinds of information. Therefore, attributes which are indistinct or tangled with others can be accurately recognized by exploiting the context information. Experiments on benchmark datasets demonstrate that the proposed pooling method appropriately explores and exploits the correlations between attributes for the pedestrian attribute recognition.




§ INTRODUCTION

Pedestrian attribute recognition has appealed much research effort due to its continuing demands for intelligent video surveillance <cit.>. It is a challenging task because of the large variance in pedestrian images, such as the viewpoint, lighting or pose changes. Thanks to the development of deep learning, pedestrian attribute recognition based on convolutional neural networks (CNNs) has made tremendous progress on the benchmark datasets,  PETA <cit.>, RAP <cit.>, and PA-100K <cit.>. However, existing methods still have a long way to the practical application where the scenario is pretty complex.





Extraordinary capability of CNN to accomplish the single-label image classification task has been comprehensively validated <cit.>. An example in the ImageNet Dataset <cit.> has only one label, such as “dog", “cat" or “car". However, pedestrian often contains multiple attributes, “age", “gender” and “clothing". Moreover, the fact that these attributes usually do not correspond to some certain objects will bring in hidden semantic recognition problem. Multi-label pedestrian attribute classification is thus a more practical and challenging problem to be thoroughly investigated. The superiority of CNNs in the standard single-label natural image classification cannot be straightforwardly extended to the attribute classification scenario due to the difference in input image domains and problem settings. Specifically, in multi-attribute recognition, candidate attributes may locate in different regions of the given image and the label space of a k-category classification task becomes 2^k, which requires more training data and parameters for establishing a deep neural network to achieve a comparable accuracy. Current CNN solutions for the pedestrian attribute classification problem mainly focus on extracting high-level features from the entire image <cit.>, and multi-branch architecture for different attribute groups <cit.>, to accomplish the subsequent classification.

The multi-branch manner for pedestrian attribute recognition achieves better performance via sharing the bottom l layers then breakuping into tree-like architecture, resulting in task-specific sub-networks for similar attribute groups. This is inspired by the common observation that bottom layers in CNNs mainly extract low level visual features which can be shared across different tasks, while the top layers capture high level semantic features that are more task specific. <cit.> proposed a multi-task deep convolutional neural network (MCNN) for facial attribute classification. <cit.> dynamically created a tree-like deep architecture where similar tasks reside in the same branch until the top layers. <cit.> utilized a CNN-RNN model to take advantage of the intragroup mutual exclusion and inter-group correlation. These methods mine the correlations of attributes, but ignore the prior knowledge underlying the attribute data.

In this paper, we propose to develop a new CNN architecture for pedestrian multi-attribute recognition by exploring the correlation between different attributes, attribute co-occurrence priori <cit.>. In particular, there is a high probability that some attributes that often co-occur in the training set will also co-occur in the testing set. For example, given a pedestrian annotated with “long hair", “skirt" and “high-heeled shoes", we can easily deduce its gender attribute as “female”. Based on this insightful observation, we develop a novel attribute aware pooling method (AAP) for integrating the information from different predictions, namely CoCNN. More specifically, the base network follows the multi-branch architecture, and then the context information in these branch is collected to estimate the attribute probabilities, which is then combined with the individual estimations of each branch for improving the resulting decision. The diagram of the proposed scheme is shown in Fig. <ref>. Experimental results on benchmarks demonstrate the superiority of the proposed algorithm over the state-of-the-art methods for pedestrian attribute classification.



§ PRELIMINARIES

We first briefly introduce some related works on CNN and then build the relationship between the output of a conventional CNN and the proposed attribute aware pooling method.

Let 𝒳 and 𝒴 denote instance space and k-label space, respectively. Given a labeled training set with n instances, {(𝐱^1,𝐲^1),(𝐱^2,𝐲^2),...,(𝐱^n,𝐲^n)}, where 𝐱^i is the i-th instance and 𝐲^i denotes its label vector which is a k-dimensional binary vector, 𝐲^i = [𝐲_1^i,...,𝐲_k^i] and 𝐲_k^i ∈{0, 1}. Denote the feature of 𝐱_i calculated by a given CNN 𝒩 as x^i = 𝒩(𝐱^i) ∈ℝ^d, the conventional probability hypothesis for the j-th attribute is

    𝐲̂^i_j = (𝐚^j |x^i) = 1/1+e^-θ_j^Tx^i,

where 𝐚^j is the j-th attribute in the dataset corresponding to the j-th dimensionality of the label space 𝒴, θ_j∈ℝ^d is the parameter of linear classification layer. 






In practice, for a given image 𝐱 (image index i is omitted to simplify notation), it is difficult to accurately estimate its conditional probabilities all attributes labels simultaneously, due to the tangle between attributes and the deformation of objects,  occlusions, illumination changes, rotations, and scale transforms. To address this problem, the multi-branch architecture is utilized in this work,  we have m branches {b_1,...,b_m}. The lower layers are shared across branches and the top layers are separated to focus on the attributes at different positions, as shown in Fig. <ref>. The detailed multi-branch architecture is given in the experiments section. Formally for each branch, we can construct the following m× k matrix, 

    P = [
    [ (𝐚^1 |b_1) (𝐚^2 |b_1)            (𝐚^k |b_1); (𝐚^1 |b_2) (𝐚^2 |b_2)          ⋱          ⋮;          ⋮          ⋱          ⋱           ; (𝐚^1 |b_m)                       (𝐚^k |b_m);            ]],

where (𝐚^j |b_l) stands for the conditional probability of 𝐚^j given the feature of l-th branch, and each row in P corresponds to the softmax output of the branch b_l. Moreover, conditional probabilities of the whole image 𝐱 different attributes can be estimated through a max-pooling operation: 

    (𝐚^j |𝐱) ≈max_l∈[1,m](𝐚^j |b_l) = max_l∈[1,m] P_l,j.


It is instructive to note that the accuracy of Eq. <ref> strongly depends on the prediction result of different branches. These branches are extracted from input images by exploiting empirical knowledge such as position information. However, attributes tangled with each other are difficult to be separated by branches accurately.  Hence, the estimated conditional probability vector using Eq. <ref> could be biased when the j-th attribute in 𝐱 has not be well separated. Elements in the j-th column of P will be extremely small, which leads to the (𝐚^j |𝐱) after the max-pooling being small as well.

A widely accepted fact in the filed of multi-label classification <cit.> is that correlated labels have a strong probability to simultaneously occur in real-world images. In practice, for an attribute 𝐚^j, we can use the other attributes to estimate the conditional probability the j-th attribute. The auxiliary estimation is generated as follow: 

    P_l,j^+ = 1/s∑_i=1^kS_l^i(𝐚^j|𝐚^i),

where S_l^i∈{0,1} is an indicator and S_l^i=1 denotes 𝐚^i appears in the image, and vice versa, and s = ∑_i=1^kS_l^i is the number of the positive attributes. (𝐚^j | 𝐚^i) is the conditional probability of two attributes or labels which can be pre-calculated by accounting the number of their co-occurrence in the dataset. P^+ is an m× k matrix, which can be applied for refining the prediction result of conventional CNNs for the multi-label pedestrian attribute classification problem. Since the co-occurance information S_l^j of any branch b_l is not given in the dataset, Eq. <ref> is hard to calculate directly. We therefore propose a novel attribute aware pooling method which first integrates the co-occurance information of every branch {b_1,...,b_m} and the priori knowledge, and then generates the estimation of the whole image by exploiting the co-occurrence priori over different attributes.




§ ATTRIBUTE AWARE POOLING

To further explore the correlation between attributes using the priori knowledge in the real data, in this section, we first build conditional probability matrices different attributes using the co-occurrence priori and then develop the new attribute aware pooling method.



 §.§ Co-occurrence Priori Embedding

Given a training set with k labels and n instances, let N_i denote the number of the i-th label occur in the dataset, and p_i = (𝐚^i)=N_i/n be the probability of the i-th label. By denoting the co-occurrence number of the i-th label and the j-th label in the dataset as N_i,j, we can construct a matrix J∈ℝ^k× k, where J_i,j =(𝐚^i,𝐚^j)= N_i,j/n is the joint probability of 𝐚^i and 𝐚^j. Subsequently, we can obtain the conditional probability of an attribute given another attribute as

    C_i,j = (𝐚^i|𝐚^j) = (𝐚^i, 𝐚^j)/(𝐚^j) = J_ij/p_j.

After constructing this matrix, we can further estimate the auxiliary probability P^+_l,j by exploiting the context information of the l-th branch 𝐱_l. Fig. <ref> shows the matrix C discovered on PA-100K dataset <cit.>, we can see that some attributes frequently co-occur in the dataset, “Female” and “Skirt&Dress”, “LongSleeve” and “Trousers”.



In the k-label classification task, given an instance with m branches {b_1,...,b_m}, the softmax classifier outputs m k-dimensional vectors, which are stacked into an m× k matrix P as shown in Eq. <ref>. For any branch b_l, there are m-1 branches in 𝐱 that surround with b_l and we denote these branches as b̃_l. The probability of the occurrence of attribute 𝐚^j in b̃_l can be calculated by

    (𝐚^j_l)=(𝐚^j|b_1,...,b_l-1,b_l+1,...,b_m).

However, this high-order posterior probability cannot be accurately calculated. Alternatively, we use the following locally max-pooling as an approximation: 

    Q_l,j = (𝐚^j_l) ≈max_i≠ l(𝐚^j|b_i).

Attributes identification in b̃_l can be easily accomplished through the following threshold function with the parameter τ:

    S_l^j  = {[ 1,   if   Q_l,j >τ,;     0,   otherwise.;                     ]

and we can use this indicator matrix for calculating Eq. <ref>. Compared with the output conditional probability P_l,j of the l-th branch b_l given the j-th attribute derived from CNNs, P_l,j^+ is calculated by exploiting outputs of other branches, and can be taken as another important auxiliary predictor. 

However, given the biased recognition of the branches, this hard threshold strategy may bring in inaccurate classification result and the introduction of the hyper-parameter τ will make s various for different branches. In addition, since attributes have some conflicts, absence of an attribute may lead to the appearance of another one. Therefore, we use the total probability theorem to modify P_l,j^+ as follow:

    P_l,j^+ = 1/k∑_i=1^k   (𝐚^i_l)(𝐚^j|𝐚^i)
       +1/k∑_i=1^k(𝐚̃^i_l)(𝐚^j|𝐚̃^i)

where 𝐚̃^i indicates that 𝐚^i does not occur, and the second term can be calculated as

    (𝐚̃^i_l)   (𝐚^j|𝐚̃^i) = (𝐚̃^i_l)(𝐚^j,𝐚̃^i)/(𝐚̃^i)
       =(𝐚̃^i_l)(𝐚^j)-(𝐚^j,𝐚^i)/1-(𝐚^i)
        = (1-(𝐚^i_l))p_j-J_i,j/1-p_i,

and we can first calculate the matrix 

    C̃_i,j =(𝐚^i|𝐚̃^j)=p_i-J_i,j/1-p_j

based on the training set in advance for convenience.

By exploiting the pre-defined two matrices C and C̃, the additional estimated probability matrix P^+ can be simply calculated as

    P^+ = 1/k[QC+(1-Q)C̃],

where 1 is a k× k full one matrix.

Eq. <ref> is estimated by exploiting context information and co-occurrence tables, which provides another approach to estimate the classification result of the input instance 𝐱. The attribute aware pooling method is thus developed to combine the two decision matrices of different properties. Formally, the output of the attribute aware pooling layer is

    P̂ = P+λ P^+,

where λ is the parameter for balancing the prediction results of the branch itself and context information from the other branches. We will further test its impact on the classification accuracy in the following section experimentally.

Finally, we assemble predictions of every branch to form a k-dimensional vector. Since every element in P̂ is greater than zero, for a given image 𝐱^i, the output of the proposed method is 

    𝐩̂_j^i = (𝐚^j|𝐱^i)=max_l∈[1,m]P̂_l,j/∑_j=1^kmax_l∈[1,m]P̂_l,j,

where M∈{0,1}^m× k is the max-pooling projection which has only one 1 per column. Let ℰ^i = diag(P̂^TM), and we have

    𝐩̂^i=ℰ^i/||ℰ^i||_1,

where ||·||_1 is the conventional ℓ_1 norm for vectors. The ground-truth conditional probabilities of 𝐱^i is 𝐩^i=𝐲^i/||𝐲^i||_1. Therefore, the loss function of the network using the proposed method is

    J =   1/2n∑_i=1^n∑_j=1^k(𝐩̂_j^i -𝐩_j^i )^2 
    =   1/2n∑_i=1^n||𝐩̂^i-𝐩^i||_2^2.


By optimizing the above loss function using the mini-batch SGD approach, we can fine-tune filters in the neural network 𝒩 and obtain a new CNN with improvement in attribute recognition accuracy. The diagram of the proposed attribute aware pooling architecture is shown in Fig. <ref>.



 §.§ Back Propagation


We proceed to introduce the optimization method for CNNs with the proposed attribute aware pooling method. Stochastic gradient descent (SGD) needs to calculate the gradient of the loss function and then utilizes the back propagation strategy for updating parameters. We next detail the gradient and the back propagation of the proposed method.

In fact, the proposed attribute aware pooling method is attached after the softmax layer of a general CNN on m branches and followed by some pooling and probability calculations as shown in Fig. <ref>. Therefore, we do not have to modify any additional convolution layer and filters, and we only need to calculate gradient of the loss function in the last layer the individual branch estimation matrix P in Eq. <ref>. Commonly, this gradient can be computed using the standard chair rule, 
    ∂ J/∂ P=∂ J/∂𝐩̂·∂𝐩̂/∂ℰ·∂ℰ/∂P̂·∂P̂/∂ P.

We first calculate the gradient of the first two terms for a given instance 𝐱 as follow:

    ∂ J/∂ℰ = 𝐩̂-𝐩/||ℰ||_1-(𝐩̂-𝐩)^Tℰ/||ℰ||_1^2,

where ||ℰ||_1 = ∑_j=1^k |ℰ_j|, and ℰ∈ℝ^k×1. Subsequently, the gradient of P̂ is calculated as follow: 

    ∂ J/∂P̂ = M diag(𝐩̂-𝐩/||ℰ||_1-(𝐩̂-𝐩)^Tℰ/||ℰ||_1^2),

where the variables in diag(·) make up a k-dimensional vector and M is the max-pooling projection which maps the gradient into a m× k sparse matrix. 

As for the gradient of P, consider Q_l = diag(P^T S_l) where S_l∈{0,1}^m× k is the l-th mask the locally max-pooling (Eq. <ref>) for generating the l-th row of Q. Elements in the l-th row of S_l are zeros. The l-th row of P̂ can be reformulated as

    P̂_l    = P_l+λ P_l^+ = P_l+λ (Q_lC+(1-Q_l)C̃)
    
     =    P_l+λ/k[ diag(P^TS_l)C+(1-diag(P^TS_l))C̃].

Removing terms irrelevant to P, the j-th element in P_l^+ can be simplified as

    P_l,j^+    =diag(P^TS_l)C_j-diag(P^TS_l)C̃_j
       = Tr(P^TS_l diag(C_j))-Tr(P^TS_l diag(C̃_j)),

where C_j and C̃_j are the j-th columns in C and C̃, respectively. The gradient of P_l,j^+ to P is

    ∂ P_l,j^+/∂ P   = S_l diag(C_j)-S_l diag(C̃_j)
       =S_l diag(C_j-C̃_j).

Therefore, the gradient of the classification loss P is 

    ∂ J/∂ P= ∂ J/∂P̂+λ/k∑_l=1^m∑_j=1^k∂ J/∂P̂_l,jS_l diag(C_j-C̃_j),

and P can be updated as

    P = P-η∂ J/∂ P,

where η is the learning rate. Alg. <ref> summarizes the feed forward and the back propagation of the proposed method.






§ EXPERIMENTS

Here we will present the setting of experiments for evaluating the proposed method and compare it with the state-of-the-art methods on benchmark pedestrian attribute recognition datasets.



 §.§ Datasets and Metrics
 
The evaluation is conducted on three largest publicly available pedestrian attribute datasets: 

	
  * PETA <cit.>: The PEdesTrian Attribute dataset consists of 19,000 pedestrian images with 65 attributes (61 binary and 4 multi-class). Following <cit.>, we randomly divide the whole dataset into three partitions: 9,500 images for training, 1,900 for validation, and 7,600 for testing, and focus on the 35 attributes whose positive proportions are bigger than 1/20. 
	
  * RAP <cit.>: The Richly Annotated Pedestrian attribute dataset contains 41,585 pedestrian images with 72 attributes (69 binary and 3 multi-class). It is split into 33,268 images for training and the remaining 8,317 for testing. The same 51 binary attributes are evaluated for fair comparison following <cit.>.
	
  * PA-100K <cit.>: The PA-100K dataset has 100,000 images with 26 commonly used binary attributes. The whole dataset is randomly split into training, validation and test sets with a ratio of 8:1:1.


We use five evaluation metrics including a class-centric metric mean accuracy (mA), and four instance-centric metrics, accuracy, precision, recall and F1-score, following <cit.>



 §.§ Implementation Details

In the proposed method, there are m branches for attribute prediction. We use ResNet-50 <cit.> as the backbone network for building our multi-branch architecture. The main body of ResNet-50 except the last residual block is shared and the the last block is adopted as the architecutre of each branch. For a 448×224 input image, the feature maps F from the shared CNN is of size 2048×14×7. Specially, we design 4 branches in our model and the whole feature F are fed into the first branch. Inspired by <cit.>, we spatially partition the feature maps F into 3 parts with sizes of 4×7, 6×7 and 6×7, serving for the other three branches, as shown in Fig <ref>. Each part focuses on different human position, head, upper body and lower body, for position-aware attribute recognition.





In our method, we first counted occurrence and co-occurrence numbers of different categories among train images and built two matrices C and C̃ for each dataset. The backbone model parameters are initialized by directly using the pre-trained models on ILSVRC 2012 <cit.>. Adam optimizer <cit.> with a batch size of 32 is used to fine-tune the entire model with an initial learning rate of 0.01. The images are resized to 512×256 and during training randomly cropped to the size of 448×224 with random horizontal flippling. All methods were implemented using PyTorch <cit.> and run on NVIDIA V100 graphics cards. We denote the multi-branch architecture with the proposed attribute aware pooling method as CoCNN. 

The proposed CNN architecture with the attribute aware pooling method has only one hyper-parameter λ whose functionality is to combine the original CNN output and the estimation from the context information. We tested the impact of λ by tuning it from 0 to 0.5 with the step of 0.05 on the mA metric using the PETA dataset. In fact, a larger λ makes the result 𝐩̂ more inclined to the auxiliary estimation P^+ and vice versa. As a result, we obtained a 86.97% mA value at λ = 0.2 which is the best trade-off between estimations in Eq. <ref>. Moreover, for RAP and PA-100K datasets, we keep λ = 0.2 in the following experiments.




 §.§ Results and Analysis



  §.§.§ Effect of Multi-branch Architecture and AAP

The improvement of CoCNN mainly comes from two aspects, multi-branch architecture and attribute aware pooling. In this section, we will measure how much improvement these two aspects bring. The vanilla ResNet-50 with binary cross entropy loss is adopted as baseline model, and the multi-branch ResNet-50 without attribute aware pooling is denoted as multi-branch model. From the results in Tab. <ref>, we can see that multi-branch architecture improve the metrics generally in the three datasets by focusing on different parts of the human body. With attribute aware pooling, CoCNN further improves the accuracy to higher level.

Two examples from the PA-100K dataset are given in Fig. <ref> for qualitative analysis. From the results, we found that multi-branch model made wrong predictions “Skirt&Dress” and “AgeLess18” for the two pedestrians, respectively. By exploiting their correlation with other attributes, especially the “Female” in the left person, and the “UpperPlaid” and “Backpack” in the right example, CoCNN well corrected the wrong predictions in multi-branch model.







  §.§.§ Comparison with State-of-the-art Methods
 
After verifing the effect of the proposed method, we compared the proposed approach with several state-of-the-art approaches, DeepMAR <cit.>, HP-net <cit.>, SR C-RNN <cit.>, VAA <cit.>, and GRL <cit.>. Results of comparison models are mostly reported from their papers directly. The results are listed in Tab. <ref>.

From results on PETA dataset, we found that, CoCNN clearly outperformed other state-of-the-art methods and achieved the highest mA, acc, prec and F1 values (86.97, 79.95, 87.65). The recall of CoCNN is the second best one. This improvement mainly comes from that the proposed method takes multiple attributes into account when generating the classification annotation.

In addition, similar observation also can be seen on RAP and PA-100K datasets. This phenomenon further illustrates that the proposed attribute aware pooling method is a general auxiliary regularization for enhancing pedestrian attribute recognition and can be embedded into any similar task.



§ CONCLUSIONS

Existing vanilla CNNs cannot be effectively applied to handle the pedestrian attributes recognition task. Therefore, we propose using the co-occurrence priori for improving performance of CNNs, namely, attribute aware pooling. With a multi-branch CNN as base architecture, we first construct two co-occurrence tables through training sets and use context information of every branch to complete the decision derived from each branch itself, which not only excavates properties of different human parts but also investigates the relationship between different branch. Experiments conducted on several benchmark datasets show that the performance of the proposed method is better than those of state-of-the-art methods in terms of the commonly used metrics.

wang2018learning
han2018autoencoder
luo2018heterogeneous
luo2019transferring

named


