Using higher order knowledge to reduce training data has become a popular research topic. However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels. Based on this observation, we consider constraining output probability distribution as higher order domain knowledge. We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.   While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution. We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.
We introduce an embedding space approach to constrain neural network output probability distribution. This paper introduces a method to perform semi-supervised learning with deep neural networks, and the model achieves relatively high accuracy, given a small training size. This paper incorporates label distribution into model learning when a limited number of training instances is available, and proposes two techniques for handling the problem of output label distribution being wrongly biased.