Using higher order knowledge to reduce training data has become a popular research topic. However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels. Based on this observation, we consider constraining output probability distribution as higher order domain knowledge. We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.   While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution. We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples. Probability is an abstract measure on how a certain event occurs independent of features of the events. Knowing how likely a certain event occurs, people leverages such prior knowledge to their decision making. For example, doctors know certain diseases are rare, even if they are told in terms of probabilities instead of "training examples". Based on this knowledge, they make less predictions on these diseases than those common ones. Do neural networks behave in a similar way? Unfortunately, the answer is no. When we train a multi-layer perceptron(MLP) for MNIST classifier BID10 ) with limited labelled examples, the output distribution can be extremely biased in favor of some of the labels. In Figure 1a , we compare the predicted number of labels with ground truth. While the training accuracy is 1.0, the model clearly overfits to those training examples and leave labels between training data points undefined in high dimensional feature space. As we plot the last hidden layer of a MLP trained with 50 labelled MNIST data as shown in Figure 1b , we find neural networks fail to learn the decision boundary correctly from a limited number of examples.Thus, it is natural to consider introducing output label probability distribution as higher order knowledge when we train neural networks. Different from traditional logical constraints BID22 ) or functional constraints BID18 , we propose a novel embedding space probabilistic constraint. Because of the sparsity of high dimensional feature space with only a few labeled examples, we perform our probabilistic constraint on neural network's embedding space, which is constructed unsupervisedly by projecting data into low dimensional space through autoencoder. Based on observation by BID21 , BID23 , embedding space preserves information of separations of different label clusters. In the embedding space, we pool softmax activation (a) Strong imbalanced output distribution of labels when training set is limited (b) Chaotic embedding space in the hidden layer of the classifier trained with 50 labelled examples Figure 1 : Limited training data cannot train neural networks to learn accurate decision boundaries outputs and optimize towards target distribution. By training with very few high quality labelled examples and marking on batches that have output distribution greatly different from target probability distribution, we use experiments to empirically prove that our model can converge to a high accuracy faster than state-of-art semi-supervised learning methods.
We introduce an embedding space approach to constrain neural network output probability distribution. This paper introduces a method to perform semi-supervised learning with deep neural networks, and the model achieves relatively high accuracy, given a small training size. This paper incorporates label distribution into model learning when a limited number of training instances is available, and proposes two techniques for handling the problem of output label distribution being wrongly biased.