The softmax function is widely used to train deep neural networks for multi-class classification. Despite its outstanding performance in classification tasks, the features derived from the supervision of softmax are usually sub-optimal in some scenarios where Euclidean distances apply in feature spaces. To address this issue, we propose a new loss, dubbed the isotropic loss, in the sense that the overall distribution of data points is regularized to approach the isotropic normal one. Combined with the vanilla softmax, we formalize a novel criterion called the isotropic softmax, or isomax for short, for supervised learning of deep neural networks. By virtue of the isomax, the intra-class features are penalized by the isotropic loss while inter-class distances are well kept by the original softmax loss. Moreover, the isomax loss does not require any additional modifications to the network, mini-batches or the training process. Extensive experiments on classification and clustering are performed to demonstrate the superiority and robustness of the isomax loss. Recent years have witnessed significant progress in image classification tasks with convolution neural networks (CNN) BID13 ; . For classification problems, the softmax is a suitable criterion for supervised learning since it is capable of training network parameters to generate discriminative features for hyperplanes to distinguish different classes. Due to its end-to-end characteristic, CNN is amenable to learning such that we only need to feed the network with plenty of training samples. Therefore, the softmax is the most fundamental classifier applied in architectures of deep learning.However, there are still defects of the softmax loss 1 . Features extracted by convolution layers work best only with softmax classifier. When we apply these feature vectors in other tasks such as image retrieval with k-nearest neighbors (k-NN) or clustering with K-means, the results are usually suboptimal, as shown in FIG0 . To separate different classes is the sole purpose of softmax classifier, and it does not ensure that the distances (generally Euclidean distances) within the same class are smaller than inter-class ones. In order to extract better features not only for classification powered by the softmax classifier but for other tasks using distances of feature vectors, many approaches have been proposed in the past years.One way is adding some new loss terms to the original softmax loss. The center loss penalizes distances of training samples to their corresponding class centers, thus enhancing the compactness of each class. Since this loss cannot extend the distances between class centers, the result depends heavily on the center initialization of all classes. Also, a relatively small batch size will seriously affect the performance of the algorithm, because centers cannot be accurately calculated with limited examples in a batch, especially for datasets with plenty of classes. The contrastive-center loss BID17 combines the center loss and the contrastive loss BID22 together to penalize distances of samples in the same classes and enlarge inter-class distances. It works well on the CIFAR-10 classification task BID10 and the LFW verification task BID7 . To gain good performance, however, this loss needs to carefully select data batches for training too. Some other strategies aim at solving the problem in a different way. The triplet loss BID18 comes up with a novel method. Instead of exploiting the softmax loss, they remove the final logits layer of network and directly minimize Euclidean distances between anchors and positive samples while maximizing distances of anchors to negative samples in triplets. However, the number of different triplets are much more than that of training samples, leading to that selecting proper triplets is crucial for the triplet loss. Or inappropriate triplets will result in slow convergence. What's more, the scalar margin in the triplet loss and the learning strategy influence the final performance of the model as well. To sum up, the principle of the triplet loss is straightforward and plausible, but the parameters and the "semi-hard triplets mining" approach make this algorithm hard to implement.Methods mentioned above are all based on supervised optimization, meaning that they all depend on sample labels to learn discriminative features. In this paper, we propose a simple approach, named isotropic normalization, that reshapes data distribution towards easy classification without the aid of labels. We firstly analyze the distribution of features extracted by CNNs supervised by the softmax loss. Elliptical shapes of feature distributions lead to intra-class distances even greater than inter-class distances, indicating that the softmax loss needs to be improved for tasks using feature distances. Then we attempt to modify the feature distribution according to the global distribution itself rather than information from sample labels. Combined with the vanilla softmax, we propose a new loss, called the isotropic softmax (isomax for short) loss. For the isomax, the intra-class distances of the features can be well minimized by the isotropic loss, and at the same time, the vanilla softmax loss ensures the inter-class separability. We perform extensive experiments with different networks and different datasets to illustrate the effectiveness, simplicity and portability of our method. In this paper, we propose a new isotropic loss together with the original softmax loss named isomax loss. With the joint supervision signal, CNNs generate more isotropic feature distribution for each class, and the Euclidean distance in the class decreases. The 2-D visualization and extensive experiments on different datasets for different tasks show the effectiveness of our approach. Comparison to other related works illustrates the advantage of our method on tasks using feature distance.
The discriminative capability of softmax for learning feature vectors of objects is effectively enhanced by virture of isotropic normalization on global distribution of data points.