{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/2028","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/2028/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/2028/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/2028/events","html_url":"https://github.com/dmlc/mxnet/issues/2028","id":152989132,"number":2028,"title":"Incorrect SoftmaxActivation Derivative","user":{"login":"sbodenstein","id":10237782,"avatar_url":"https://avatars.githubusercontent.com/u/10237782?v=3","gravatar_id":"","url":"https://api.github.com/users/sbodenstein","html_url":"https://github.com/sbodenstein","followers_url":"https://api.github.com/users/sbodenstein/followers","following_url":"https://api.github.com/users/sbodenstein/following{/other_user}","gists_url":"https://api.github.com/users/sbodenstein/gists{/gist_id}","starred_url":"https://api.github.com/users/sbodenstein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbodenstein/subscriptions","organizations_url":"https://api.github.com/users/sbodenstein/orgs","repos_url":"https://api.github.com/users/sbodenstein/repos","events_url":"https://api.github.com/users/sbodenstein/events{/privacy}","received_events_url":"https://api.github.com/users/sbodenstein/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-04T11:27:55Z","updated_at":"2016-05-04T18:52:01Z","closed_at":null,"body":"The new CPU version of the SoftmaxActivation calculates the derivative incorrectly: \r\n```\r\nm_out_grad * m_out_data * (1.0f - m_out_data)\r\n``` \r\nThis formula is wrong (it simply gives the diagonal terms of the input gradient matrix). You can also verify that the CPU derivative now differs from the GPU version (which uses the cuDNN implementation). \r\n\r\nI have a non-MShadow CPU implementation of the gradient, but it requires temporary memory to be assigned. Should I submit a PR, or do you want to fix this in some other way?\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/216964131","html_url":"https://github.com/dmlc/mxnet/issues/2028#issuecomment-216964131","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/2028","id":216964131,"user":{"login":"piiswrong","id":568948,"avatar_url":"https://avatars.githubusercontent.com/u/568948?v=3","gravatar_id":"","url":"https://api.github.com/users/piiswrong","html_url":"https://github.com/piiswrong","followers_url":"https://api.github.com/users/piiswrong/followers","following_url":"https://api.github.com/users/piiswrong/following{/other_user}","gists_url":"https://api.github.com/users/piiswrong/gists{/gist_id}","starred_url":"https://api.github.com/users/piiswrong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piiswrong/subscriptions","organizations_url":"https://api.github.com/users/piiswrong/orgs","repos_url":"https://api.github.com/users/piiswrong/repos","events_url":"https://api.github.com/users/piiswrong/events{/privacy}","received_events_url":"https://api.github.com/users/piiswrong/received_events","type":"User","site_admin":false},"created_at":"2016-05-04T18:52:01Z","updated_at":"2016-05-04T18:52:01Z","body":"The correct formula is d/dy_i y_i - \\sum_j d/dy_j y_i y_j.\r\nSo we can do it with mshadow like this:\r\n```m_out_data*(m_out_grad- broad_cast(reduce<sum>(m_out_grad*m_out_data)))``` "}}