title: Siamese network: disable shared variable gradient back propagation
body: <p>I am currently working on porting the siamese network of this paper: <a href="http://www.robots.ox.ac.uk/~luca/siamese-fc.html" rel="nofollow noreferrer">Fully-Convolutional Siamese Networks for Object Tracking</a> from matlab to tensorflow code. I am having problems of kernels becoming NaN after 80k to 150k steps and have already tried some methods of preventing nan with gradient clipping and preventing a zero loss but I am still having some problems.</p>
  
  <p>One of the things I am suspicious off is the back propagation, the siamese network I am trying to implement is unique in it's method of using one side of the networks output as a convolution "filter" for the other side. I have implemented this with some variable reshapes and a depth-wise convolution.</p>
  
  <p>Since both sides of the network share the same weights using reuse=true in the variable scope, I think there is a problem with 2 gradients back propagating into one variable as can be seen in the screenshot.<a href="https://i.stack.imgur.com/A9Bt2.png" rel="nofollow noreferrer">Gradient descent calculation.</a></p>
  
  <p>What I want to try is to only back propagate one side of the gradient for updating the variables but still have both side share the same variables. In essence make a copy of the first variable that is updated whenever the first variable changes, but set the copy variable to trainable=False. </p>