I've posted this question on data science community, but haven't received an answer yet. So I also posted it here in the hope that more people could see it.

I'm learning about the restricted Boltzmann machine (RBM), and I just came up with two naïve understandings of this model. But it seems these two understandings are so different, can someone tell me how can I combine these two understandings?

  1. My first understanding goes like this:

The hidden units are just structural support and we don't care about what those hidden vectors really are. We introduce those hidden units just for RBM to gain more expressive power for probability distribution :

let's say I have only one image, and I transform this image to a binary vector and feed this vector into a RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector. (my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM). And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.

  1. My second understanding goes like this:

RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

Given a RBM, each hidden units of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by this hidden units (by its weights and bias). So, if we input an image into this RBM, the RBM will project this input vector onto many hyperplanes defined by all the hidden units, and thus for an input vector, the corresponding hidden vector is very important. It is some abstract representation. And we can further feed this representation into other learning models.

So, these are my understandings, and the thing that troubles me the most is what the role that hidden vectors actually plays?

If you can answer this question by explaining how RBM is used for MNIST that would be extremely helpful for me.


  • 1
    $\begingroup$ Someone please help... $\endgroup$ – meTchaikovsky Sep 21 '17 at 11:23
  • 1
    $\begingroup$ We are here to help this world i.e each other.That's how my friendly humans perceive it.So I kindly request you to revise your question inline with the community guide-lines $\endgroup$ – quintumnia Sep 21 '17 at 16:36
  • 1
    $\begingroup$ @quintumnia sorry, my original post was such a mess... I've edited the post, and if there's still something I haven't made clear, please tell me. $\endgroup$ – meTchaikovsky Sep 22 '17 at 0:38

Generative modeling is concerned about P(X), to be able to compute it, we use representation learning to identify and disentangle the underlying explanatory factors hidden in the observed data, because if we could separate them out like this

P(X) = P(X|Z) P(Z)

Z could encode not just the digit identity but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties.

Any classifier for some task will decide which of those z elements could help him to make better decision i.e. just the part that holds the digit identity and ignore the rest, this kind of modeling is called distributed representation where each element is a simple indicator about some feature.


RBM is an undirected graphical model and no conditional dependency within a layer thus the conditional distribution over the hidden units factorize given the input image and the conditional distribution over the visible units given the hidden units also factorize.

enter image description here

Inference in RBM is tractable and we could compute p(z|x) but the tractability of the RBM does not extend to its partition function

To minimize the negative log-likelihood of the data:
The first term increases the probability of training data.

enter image description here

The second term is nothing less than an expectation over all possible configurations of the input x (under the distribution P formed by the model) so we use Monte-Carlo based algorithms to replace the expectation with an average sum.
Samples of p(x) from the model can be obtained by running a Markov chain to convergence, using Gibbs sampling as the transition operator

For RBMs, the set of visible and hidden units, since they are conditionally independent, one can perform block Gibbs sampling. In this setting, visible units are sampled simultaneously given fixed values of the hidden units. Similarly, hidden units are sampled simultaneously given the visible units.

A step in the Markov chain is thus taken as follows:

  • v: input values
  • h: hidden features, latent space, a.k.a Z
  • W: the weight parameters, please see we are using the transposed W since it's undirected graphical model.

enter image description here

where h(n+1) refers to the set of all hidden units at the n-th step of the Markov chain. What it means is that, for example, h^(n+1)_i is randomly chosen to be 1 (versus 0) with the probability of the sigmoid function and similarly for v^(n+1)_j


CD does not wait for the chain to converge. Samples are obtained after only k-steps of Gibbs sampling. In practice, k=1 has been shown to work surprisingly well.


In 2006, a breakthrough in feature learning and deep learning took place (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier.

Reducing the Dimensionality of Data with Neural Networks

enter image description here

  1. Train single layer RBM to reconstruct to input and then keep adding layer after layer until we reach some good low-level representation as depicted in the figure below, W1, W2, W3 are the matrices that have some lower dimensionality of the input space.
  2. Transpose those matrices W1, W2, W3 to be the decoder part.
  3. Fine tune the whole encoder-decoder stack to reconstruct the input. image

enter image description here

Side note: RBM is now obsolete, see VAE, GAN they are way better at modeling the manifold which the data is concentrated, yet you could sample from the manifold directly with a single inference step.


You seem to just be asking how a neural network functions. We used to use chips called Programmable Logic Arrays. Digital logic typically consists of AND/NAND, OR/NOR functions. A PLA had a grid of AND logic followed by a grid of OR logic. You could program these after they were manufactured to create specific circuits without having to manufacture specialty chips with the same logic. So, a given PLA had a fairly large number of circuit configurations and these were typically used for things like motherboard control logic.

A neural network is similar but a lot more complex. You have a visible input layer which just consists of buffers or memory locations to hold the values. And you have a visible output layer. In between these two is one or more hidden layers. If this were purely physical then these hidden nodes would be connected to specific input buffers. Since this is typically simulated the connections are only virtual.

Unlike the PLA where you knew exactly what you wanted the circuit to do and only had to program the correct logic, a neural network has unknown logic. You assume that some logical arrangement exists that will provide the correct outputs from the inputs but you don't know what it is. So, you give it some input and then check the output. You find the error in the output and then use this during back propagation to make a change in the hidden layer. You repeat this over and over until it stops converging on the correct output pattern.

Deep systems have multiple hidden layers. However, you can only train one layer at a time. So, after you have trained the first layer it becomes the input for the next layer. In other words, your inputs still go to the input layer and that goes to the first hidden layer. However, instead of the result of the first hidden layer going to the output it goes instead to the second hidden layer and the results from this go to the output. With a second hidden layer you should be able to refine the first layer and get further convergence on the correct output. This can be repeated with a third hidden layer and so on.

So, the hidden layers hold the pattern recognition logic of the neural network.

MNIST is just a database of characters. Handwriting has some variance from an ideal character such as a computer font. The goal is to get a neural network to recognize these characters in spite of these variances. If you are trying to create a system that can convert handwriting to computer text then you can use this database or the EMNIST database to train your system.


Your Answer

