title: Backpropagation algorithm through cross-channel local response normalization (LRN) layer
body: <p>I am working on replicating a neural network. I'm trying to get an understanding of how the standard layer types work. In particular, I'm having trouble finding a description anywhere of how cross-channel normalisation layers behave on the backward-pass.</p>
  
  <p>Since the normalization layer has no parameters, I could guess two possible options:</p>
  
  <ol>
  <li><p>The error gradients from the next (i.e. later) layer are passed backwards without doing anything to them.</p></li>
  <li><p>The error gradients are normalized in the same way the activations are normalized across channels in the forward pass.</p></li>
  </ol>
  
  <p>I can't think of a reason why you'd do one over the other based on any intuition, hence why I'd like some help on this.</p>
  
  <p>EDIT1:</p>
  
  <p>The layer is a standard layer in caffe, as described here <a href="http://caffe.berkeleyvision.org/tutorial/layers.html">http://caffe.berkeleyvision.org/tutorial/layers.html</a> (see 'Local Response Normalization (LRN)').</p>
  
  <p>The layer's implementation in the forward pass is described in section 3.3 of the alexNet paper: <a href="http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf</a></p>
  
  <p>EDIT2:</p>
  
  <p>I believe the forward and backward pass algorithms are described in both the Torch library here: <a href="https://github.com/soumith/cudnn.torch/blob/master/SpatialCrossMapLRN.lua">https://github.com/soumith/cudnn.torch/blob/master/SpatialCrossMapLRN.lua</a></p>
  
  <p>and in the Caffe library here: <a href="https://github.com/BVLC/caffe/blob/master/src/caffe/layers/lrn_layer.cpp">https://github.com/BVLC/caffe/blob/master/src/caffe/layers/lrn_layer.cpp</a></p>
  
  <p>Please could anyone who is familiar with either/both of these translate the method for the backward pass stage into plain english?</p>