{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/4177","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/4177/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/4177/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/4177/events","html_url":"https://github.com/dmlc/mxnet/issues/4177","id":194741674,"number":4177,"title":"gradient clip with l2norm in mxnet","user":{"login":"feiyulv","id":18651286,"avatar_url":"https://avatars.githubusercontent.com/u/18651286?v=3","gravatar_id":"","url":"https://api.github.com/users/feiyulv","html_url":"https://github.com/feiyulv","followers_url":"https://api.github.com/users/feiyulv/followers","following_url":"https://api.github.com/users/feiyulv/following{/other_user}","gists_url":"https://api.github.com/users/feiyulv/gists{/gist_id}","starred_url":"https://api.github.com/users/feiyulv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feiyulv/subscriptions","organizations_url":"https://api.github.com/users/feiyulv/orgs","repos_url":"https://api.github.com/users/feiyulv/repos","events_url":"https://api.github.com/users/feiyulv/events{/privacy}","received_events_url":"https://api.github.com/users/feiyulv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-10T03:27:48Z","updated_at":"2016-12-10T07:47:33Z","closed_at":null,"body":"I'm trying to implement a gradient clip with l2norm using the c++ optimizer. The standard gradient clip needs to calculate the  l2norm value over all the parameters, and use a global scale for every parameter, like follows:\r\n![default](https://cloud.githubusercontent.com/assets/18651286/21070885/c3847148-beca-11e6-97c3-f33968a57144.PNG)\r\n\r\nHowever in mxnet's optimizer, I can only access one parameter. Currently, I calculate a scale for each parameter, but it seems the model can't converge as good as the theano baseline.  Is there any good idea to implement a standard gradient clip over all parameters?"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/266187526","html_url":"https://github.com/dmlc/mxnet/issues/4177#issuecomment-266187526","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/4177","id":266187526,"user":{"login":"shivarajugowda","id":6300651,"avatar_url":"https://avatars.githubusercontent.com/u/6300651?v=3","gravatar_id":"","url":"https://api.github.com/users/shivarajugowda","html_url":"https://github.com/shivarajugowda","followers_url":"https://api.github.com/users/shivarajugowda/followers","following_url":"https://api.github.com/users/shivarajugowda/following{/other_user}","gists_url":"https://api.github.com/users/shivarajugowda/gists{/gist_id}","starred_url":"https://api.github.com/users/shivarajugowda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivarajugowda/subscriptions","organizations_url":"https://api.github.com/users/shivarajugowda/orgs","repos_url":"https://api.github.com/users/shivarajugowda/repos","events_url":"https://api.github.com/users/shivarajugowda/events{/privacy}","received_events_url":"https://api.github.com/users/shivarajugowda/received_events","type":"User","site_admin":false},"created_at":"2016-12-10T07:47:33Z","updated_at":"2016-12-10T07:47:33Z","body":"Wouldn't this be expensive since the calculations happens in Python, rather than on the GPU? Or is it that the whole operation in `sum([nd.norm(grad).asnumpy()[0]**2 for grad in params_grad.values()]))` is somehow pushed down to GPU?"}}
{"actor":{"display_login":"shivarajugowda"}}