title: Implementing sparse connections in neural network (Theano)
body: <p>Some use cases for neural networks requires that not all neurons are connected between two consecutive layers. For my neural network architecture, I need to have a layer, where each neuron only has connections to some prespecified neurons in the previous layer (at somewhat arbitrary places, not with a pattern such as a convolution layer). This is needed in order to model data on a specific graph. I need to implement this "Sparse" layer in Theano, but I'm not used to the Theano way of programming. </p>
  
  <p>It seems that the most efficient way of programming sparse connections in Theano would be to use <a href="http://deeplearning.net/software/theano/library/tensor/nnet/blocksparse.html">theano.tensor.nnet.blocksparse.SparseBlockGemv</a>. An alternative would be to do matrix multiplication, where many weights are set to 0 (= no connection), but that would be very inefficient compared to <code>SparseBlockGemv</code> as each neuron is only connected to 2-6 neurons in the previous layer out of ~100000 neurons. Moreover, a weight matrix of 100000x100000 would not fit on my RAM/GPU. Could someone therefore provide an example of how to implement sparse connections using the <code>SparseBlockGemv</code> method or another computationally-efficient method?</p>
  
  <p>A perfect example would be to extend the <a href="http://deeplearning.net/tutorial/mlp.html#mlp">MLP Theano Tutorial</a> with an extra layer after the hidden layer (and before softmax), where each neuron only has connections to a subset of neurons in the previous layer. However, other examples are also very welcome!</p>
  
  <p><strong>Edit:</strong> Note that the layer must be implemented in Theano as it is just a small part of a larger architecture.</p>