{"push_id":2901550828,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"4f3c624711635191b121000f9d72fa066b7f6a85","before":"c0b4aa90304deeaa5ca7a70519ce621caf49438c","commits":[{"sha":"4f3c624711635191b121000f9d72fa066b7f6a85","author":{"name":"Marko Grdinić","email":"5622e3ec1a18094ea0e746d23e3f804b2589765e@gmail.com"},"message":"In review, apart from not working with standard RNNs I think the biggest clue that the scheme I had in mind would fail would be the learning rates on Mnist. The difference between optimal learning rates on 128 minibatch and online learning is literally 2x. There is no need to average the gradient nearly as much as one would expect. Hence the representation are sparse.\n\nHence it cannot be the case that the way the standard RL works is by spam.\n\nRather than worrying about reducing spam I need to turn my thinking around - what I would need is to support sparsity. Good memories would be good for that. This might go hand in hand with chaotic behavior of a NN which would encourage such a thing.\n\nLet me resume the Rudder paper. I've been doing these studies for 3 days now. If it was not for Rudder nor this idea anchoring me, I would have already been long done with the optimized plastic RNN. But I need to do this. Right now I feel like going through the proofs in the paper until I understand them. As papers go, this one is rather imposing so I am going to be studying it for a while. Maybe it will even take me a few days.","distinct":true,"url":"https://api.github.com/repos/mrakgr/The-Spiral-Language/commits/4f3c624711635191b121000f9d72fa066b7f6a85"}]}
{"actor":{"display_login":"mrakgr"}}