In probabilistic classification, a discriminative model based on Gaussian mixture exhibits flexible fitting capability. Nevertheless, it is difficult to determine the number of components. We propose a sparse classifier based on a discriminative Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning. This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components. The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and can be trained in an end-to-end manner. Experimental results indicated that the proposed method prevented overfitting by obtaining sparsity. Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of a deep NN. In supervised classification, probabilistic classification is an approach that assigns a class label c to an input sample x by estimating the posterior probability P (c|x). This approach is primarily categorized into two types of models: discriminative model and generative model. The former optimizes the posterior distribution P (c|x) directly on a training set, whereas the latter finds the class conditional distribution P (x|c) and class prior P (c) and subsequently derives the posterior distribution P (c|x) using Bayes' rule. The discriminative model and generative model are mutually related (Lasserre et al., 2006; Minka, 2005) . According to Lasserre et al. (2006) , the only difference between these models is their statistical parameter constraints. Therefore, given a certain generative model, we can derive a corresponding discriminative model. For example, the discriminative model corresponding to a unimodal Gaussian distribution is logistic regression (see Appendix A for derivation). Several discriminative models corresponding to the Gaussian mixture model (GMM) have been proposed (Axelrod et al., 2006; Bahl et al., 1996; Klautau et al., 2003; Tsai & Chang, 2002; Tsuji et al., 1999; Tüske et al., 2015; Wang, 2007) . They indicate more flexible fitting capability than the generative GMM and have been applied successfully in fields such as speech recognition (Axelrod et al., 2006; Tüske et al., 2015; Wang, 2007) . The problem to address in mixture models such as the GMM is the determination of the number of components M . Classically, Akaike's information criterion and the Bayesian information criterion have been used; nevertheless, they require a considerable computational cost because a likelihood must be calculated for every candidate component number. In the generative GMM, methods that optimize M during learning exist (Crouse et al., 2011; Štepánová & Vavrečka, 2018) . However, in a discriminative GMM, a method to optimize M simultaneously during learning has not been clearly formulated. In this paper, we propose a novel GMM having two important properties: sparsity and discriminability, which is named sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning. This learning algorithm improves the generalization capability by obtaining a sparse solution and determines the number of components automatically by removing redundant components. Furthermore, the SDGM can be embedded into neural networks (NNs) such as convolutional NNs and trained in an end-to-end manner with an NN. To the authors best knowledge, there is no GMM that has both of sparsity and discriminability. The contributions of this study are as follows: • We propose a novel sparse classifier based on a discriminative GMM. The proposed SDGM has both sparsity and discriminability, and determines the number of components automatically. The SDGM can be considered as the theoretical extension of the discriminative GMM and the relevance vector machine (RVM) (Tipping, 2001 ). • This study attempts to connect both fields of probabilistic models and NNs. From the equivalence of a discriminative model based on Gaussian distribution to a fully connected layer, we demonstrate that the SDGM can be used as a module of a deep NN. We also show that the SDGM can show superior performance than the fully connected layer with a softmax function via an end-to-end learning with an NN on the image recognition task. In this paper, we proposed a sparse classifier based on a GMM, which is named SDGM. In the SDGM, a GMM-based discriminative model was trained by sparse Bayesian learning. This learning algorithm improved the generalization capability by obtaining a sparse solution and automatically determined the number of components by removing redundant components. The SDGM could be embedded into NNs such as convolutional NNs and could be trained in an end-to-end manner. In the experiments, we demonstrated that the SDGM could reduce the amount of weights via sparse Bayesian learning, thereby improving its generalization capability. The comparison using benchmark datasets suggested that SDGM outperforms the conventional sparse classifiers. We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when it was used as the last layer of a deep NN. One of the limitations of this study is that sparse Bayesian learning was applied only when the SDGM was trained stand-alone. In future work, we will develop a sparse learning algorithm for a whole deep NN structure including the feature extraction part. This will improve the ability of the CNN for larger data classification.
A sparse classifier based on a discriminative Gaussian mixture model, which can also be embedded into a neural network. The paper presents a Gaussian mixture model trained via gradient descent arguments which allows for inducing sparsity and reducing the trainable model layer parameters. This paper proposes a classifier, called SDGM, based on discriminative Gaussian mixture and its sparse parameter estimation.