{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/339","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/339/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/339/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/339/events","html_url":"https://github.com/fchollet/keras/issues/339","id":93155529,"number":339,"title":"Bias initialization","user":{"login":"felipefariax","id":6319642,"avatar_url":"https://avatars.githubusercontent.com/u/6319642?v=3","gravatar_id":"","url":"https://api.github.com/users/felipefariax","html_url":"https://github.com/felipefariax","followers_url":"https://api.github.com/users/felipefariax/followers","following_url":"https://api.github.com/users/felipefariax/following{/other_user}","gists_url":"https://api.github.com/users/felipefariax/gists{/gist_id}","starred_url":"https://api.github.com/users/felipefariax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felipefariax/subscriptions","organizations_url":"https://api.github.com/users/felipefariax/orgs","repos_url":"https://api.github.com/users/felipefariax/repos","events_url":"https://api.github.com/users/felipefariax/events{/privacy}","received_events_url":"https://api.github.com/users/felipefariax/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-07-05T23:17:06Z","updated_at":"2016-05-01T04:50:37Z","closed_at":"2015-07-23T01:05:08Z","body":"Hello!\r\n\r\nI was looking my network weights, when I realize that all my bias values on a Dense layer were 0s. I saw on the Dense Layer initialization: self.b = shared_zeros((self.output_dim)). Shouldn't be self.b = shared_ones((self.output_dim))? \r\n\r\nI don't know if I didn't understand, or if it exists somewhere inside the code that changes it. I'm kinda new on Theano stuff."},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/216016066","html_url":"https://github.com/fchollet/keras/issues/339#issuecomment-216016066","issue_url":"https://api.github.com/repos/fchollet/keras/issues/339","id":216016066,"user":{"login":"geminatea","id":3892928,"avatar_url":"https://avatars.githubusercontent.com/u/3892928?v=3","gravatar_id":"","url":"https://api.github.com/users/geminatea","html_url":"https://github.com/geminatea","followers_url":"https://api.github.com/users/geminatea/followers","following_url":"https://api.github.com/users/geminatea/following{/other_user}","gists_url":"https://api.github.com/users/geminatea/gists{/gist_id}","starred_url":"https://api.github.com/users/geminatea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geminatea/subscriptions","organizations_url":"https://api.github.com/users/geminatea/orgs","repos_url":"https://api.github.com/users/geminatea/repos","events_url":"https://api.github.com/users/geminatea/events{/privacy}","received_events_url":"https://api.github.com/users/geminatea/received_events","type":"User","site_admin":false},"created_at":"2016-05-01T04:50:37Z","updated_at":"2016-05-01T04:50:37Z","body":"@fchollet, I wouldn't be surprised that initializing biases to zero is standard, but it seems to be unlikely to be optimal for many normal problems one might want to solve with FC NNs.  For instance, consider regression with relu-style activations.  If the biases are initialized to zero, the half-planes of positive activation specified by each neuron are starting in a non-generic configuration (the bounding hyperplanes all pass through the origin)!  This seems like a terrible way to start finding a solution to the regression problem since you would want the positive domains to have irregular overlaps.\r\n\r\nIt would be nice to have options for bias initialization.\r\n"}}