title: Why neural network have weight space symmetry?
body: <p>"Neural nets have a weight space symmetry: we can permute all the hidden units in a given layer and obtain an equivalent solution" (From CSC321, lecture 10, Optimation)</p>
  
  <p>I don't think it make sense, is there something wrong with my understanding?</p>
  
  <p>For example, there is a simple DNN with 2 units in the only hidden layer. And there is one local optima and one global optima like this:</p>
  
  <p><a href="https://i.stack.imgur.com/VGf6p.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VGf6p.jpg" alt="enter image description here"></a></p>
  
  <p>Obviously 2 symmetric points will result in different solution, they will go into different optima(the right-bottom one is the global optima).</p>
  
  <p>Please tell me where it goes wrong?</p>