title: Guided Back-propagation in TensorFlow
body: <p>I would like to implement in TensorFlow the technique of "Guided back-propagation" introduced in this <a href="http://arxiv.org/pdf/1412.6806.pdf" rel="nofollow"> Paper </a> and which is described in this <a href="https://github.com/Lasagne/Recipes/blob/master/examples/Saliency%20Maps%20and%20Guided%20Backpropagation.ipynb" rel="nofollow" > recipe </a>. </p>
  
  <p>Computationally that means that when I compute the gradient e.g., of the input wrt. the output of the NN, I will have to <strong>modify the gradients computed at every RELU unit</strong>. Concretely, the back-propagated signal on those units must be thresholded on zero, to make this technique work. In other words the partial derivative of the RELUs that are negative must be ignored.</p>
  
  <p>Given that I am interested in applying these gradient computations only on <em>test examples</em>, i.e., I don't want to update the model's parameters - how shall I do it?</p>
  
  <p>I tried (unsuccessfully) two things so far:</p>
  
  <ol>
  <li><p>Use <strong>tf.py_func</strong> to wrap my simple numpy version of a RELU, which then is eligible to redefine it's gradient operation via the <strong>g.gradient_override_map</strong> context manager.</p></li>
  <li><p>Gather the forward/backward values of BackProp and apply the thresholding on those stemming from Relus.</p></li>
  </ol>
  
  <p>I failed with both approaches because they require some knowledge of the internals of TF that currently I don't have.</p>
  
  <p>Can anyone suggest any other route, or sketch the code?</p>
  
  <p>Thanks a lot.</p>