{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/1625","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/1625/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/1625/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/1625/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/1625","id":10318070,"number":1625,"title":"Possible math error in Binomial deviance calculations","user":{"login":"ChrisBeaumont","id":796752,"avatar_url":"https://avatars.githubusercontent.com/u/796752?v=3","gravatar_id":"","url":"https://api.github.com/users/ChrisBeaumont","html_url":"https://github.com/ChrisBeaumont","followers_url":"https://api.github.com/users/ChrisBeaumont/followers","following_url":"https://api.github.com/users/ChrisBeaumont/following{/other_user}","gists_url":"https://api.github.com/users/ChrisBeaumont/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisBeaumont/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisBeaumont/subscriptions","organizations_url":"https://api.github.com/users/ChrisBeaumont/orgs","repos_url":"https://api.github.com/users/ChrisBeaumont/repos","events_url":"https://api.github.com/users/ChrisBeaumont/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisBeaumont/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels/Bug","name":"Bug","color":"e10c02"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2013-01-25T16:57:11Z","updated_at":"2016-10-10T23:32:55Z","closed_at":null,"body":"The calculations of the binomial deviance and its gradient in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/gradient_boosting.py are bothering me. I could be missing something, but:\r\n\r\nThe deviance is calculated as `log(1 + exp(-2 * y * pred))` [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/gradient_boosting.py#L339). This matches equation 10.18 on p 346 of [Elements of Statistical Learning](http://www-stat.stanford.edu/~tibs/ElemStatLearn/). However, that derivation assumes that y is {-1, 1} valued, whereas in sklearn y is {0, 1}. Effectively, the calculation is insensitive to `pred` whenever `y=0`. The fix is to change the return line to `np.sum(np.logaddexp(0.0, -2 * (2 * y - 1) * pred)) / y.shape[0]`.\r\n\r\nThe calculation of the gradient makes sense to me if the `pred` values map to class probabilities via `P(y=1) = 1 / (1 + exp(-pred))`. However, the loss function calculation above seems to follow the convention that `P(y=1) = 1 / (1 + exp(-2 * pred))` (again, see the link above). One way to make the two equations consistent with each other is to remove the first 2 in the above equation:\r\n\r\n `np.sum(np.logaddexp(0.0, -(2 * y - 1) * pred)) / y.shape[0]`\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/252773363","html_url":"https://github.com/scikit-learn/scikit-learn/issues/1625#issuecomment-252773363","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/1625","id":252773363,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2016-10-10T23:32:55Z","updated_at":"2016-10-10T23:32:55Z","body":"@pprett can you have a look again?"}}
{"actor":{"display_login":"amueller"}}