Gender classification is a fundamental and important application in computer vision, and it has become a research hotspot. Real-world applications require gender classification in unconstrained conditions where traditional methods are not appropriate. This paper proposes a Deep Convolutional Neural Network for feature extraction together with fully-connected layers for metric learning. A Siamese network is built for similarity measuring to promote the performance of classification. Extensive experiments on several databases demonstrate that a significant improvement can be obtained for gender classification tasks in both constrained and unconstrained conditions.