title: Ada-Delta method doesn't converge when used in Denoising AutoEncoder with MSE loss & ReLU activation?
body: <p>I just implemented AdaDelta (<a href="http://arxiv.org/abs/1212.5701" rel="nofollow">http://arxiv.org/abs/1212.5701</a>) for my own Deep Neural Network Library.
  The paper kind of says that SGD with AdaDelta is not sensitive to hyperparameters, and that it always converge to somewhere good. (at least the output reconstruction loss of AdaDelta-SGD is comparable to that of well-tuned Momentum method)</p>
  
  <p>When I used AdaDelta-SGD as learning method in in Denoising AutoEncoder, it did converge in some specific settings, but not always.
  When I used MSE as loss function, and Sigmoid as activation function, it converged very quickly, and after iterations of 100 epochs, the final reconstruction loss was better than all of plain SGD, SGD with Momentum, and AdaGrad.</p>
  
  <p>But when I used ReLU as activation function, it didn't converge but continued to be stacked(oscillating) with high(bad) reconstruction loss (just like the case when you used plain SGD with very high learning rate).
  The magnitude of reconstruction loss it stacked was about 10 to 20 times higher than the final reconstruction loss generated with Momentum method.</p>
  
  <p>I really don't understand why it happened since the paper says AdaDelta is just good.
  Please let me know the reason behind the phenomena and teach me how I could avoid it.</p>