title: Why modifying the weights of a recurrent neural network in MATLAB does not cause the output to change when predicting on same data?
body: <p>I consider the following recurrent neural network (RNN):</p>
  
  <p><a href="https://i.stack.imgur.com/rNjB5.png" rel="nofollow noreferrer">RNN under consideration</a></p>
  
  <p>where <strong>x</strong> is the input (a vector of reals), <strong>h</strong> the hidden state vector and <strong>y</strong> is the output vector. I trained the network on Matlab using some data <strong>x</strong> and obtained W, V, and U.</p>
  
  <p>However, in MATLAB after changing matrix W to W', and keeping U,V the same, the output (<strong>y</strong>) of the RNN that uses W is the same as the output (<strong>y'</strong>) of the RNN that uses W' when both predict on the same data <strong>x</strong>. Those two outputs should be different just by looking at the above equation, but I don't seem to be able to do that in MATLAB (when I modify V or U, the outputs do change). How could I fix the code so that the outputs (<strong>y</strong>) and (<strong>y'</strong>) are different as they should be?</p>
  
  <p>The relevant code is shown below:</p>
  
  <pre><code>[x,t] = simplefit_dataset;  % x: input data ; t: targets
  net = newelm(x,t,5);  % Recurrent neural net with 1 hidden layer (5 nodes) and 1 output layer (1 node)
  net.layers{1}.transferFcn = 'tansig'; % 'tansig': equivalent to tanh and also is the activation function used for hidden layer 
  net.biasConnect = [0;0]; % biases set to zero for easier experimenting 
  net.derivFcn ='defaultderiv'; % defaultderiv: tells Matlab to pick whatever derivative scheme works best for this net
  view(net) % displays the network topology
  net = train(net,x,t); % trains the network
  W = net.LW{1,1}; U = net.IW{1,1}; V = net.LW{2,1}; % network matrices
  
  Y = net(x); % Y: output when predicting on data x using W
  
  net.LW{1,1} = rand(5,5); % This is the modified matrix W, W'
  Y_prime = net(x) % Y_prime: output when predicting on data x using W'
  
  max(abs(Y-Y_prime )); % The difference between the two outputs is 0 when it probably shouldn't be. 
  </code></pre>
  
  <p>Edit: minor corrections.</p>