whose activation vector is denoted by ${\bf y}^i\in R^N$ receiving an input ${\bf x}^i \in R^M$: $${\bf y}^i = f({\bf W}^i{\bf x}^i + {\bf b}^i)$$ $${\bf y}{^i}' = f'({\bf W}^i{\bf x}^i + {\bf b}^i)$$ where ${\bf W}^i$ is the $N \times M$ weight matrix of layer $i$ and ${\bf b}^i \in R^N$ is the bias vector, and $f$ is the neuron’s activation function. In all the networks we train, we use Rectified Linear Units (ReLUs) [@Nair_Hinton10], i.e, $f(x)=max(x,0)$, with corresponding derivatives $f'(x)=H(x)$ where $H(\cdot)$ is the Heaviside step function. We pre-define for this hidden layer a fixed random classifier matrix ${\bf M}^i$ which is a $C \times N$ matrix where $C$ is the number of classification categories. The random matrix, ${\bf M}^i$, is used to convert the layer activation vector, ${\bf y}^i$, to a category score vector ${\bf s}^i \in R^C$ where ${\bf s}^i = {\bf M}^i{\bf y}^i$. Since this is a supervised learning setting, the correct input category $t$ is known during training, which allows the layer to generate a scalar loss or error signal, $E(t,{\bf s}^i)$. $E$ could be for example the cross-entropy loss or the square hinge loss. This error is then backpropagated in order to calculate 