{"action":"created","issue":{"url":"https://api.github.com/repos/scipy/scipy/issues/8692","repository_url":"https://api.github.com/repos/scipy/scipy","labels_url":"https://api.github.com/repos/scipy/scipy/issues/8692/labels{/name}","comments_url":"https://api.github.com/repos/scipy/scipy/issues/8692/comments","events_url":"https://api.github.com/repos/scipy/scipy/issues/8692/events","html_url":"https://github.com/scipy/scipy/issues/8692","id":312278358,"node_id":"MDU6SXNzdWUzMTIyNzgzNTg=","number":8692,"title":"Non-negative values of `stats.hypergeom.logcdf`","user":{"login":"fperez","id":57394,"node_id":"MDQ6VXNlcjU3Mzk0","avatar_url":"https://avatars1.githubusercontent.com/u/57394?v=4","gravatar_id":"","url":"https://api.github.com/users/fperez","html_url":"https://github.com/fperez","followers_url":"https://api.github.com/users/fperez/followers","following_url":"https://api.github.com/users/fperez/following{/other_user}","gists_url":"https://api.github.com/users/fperez/gists{/gist_id}","starred_url":"https://api.github.com/users/fperez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fperez/subscriptions","organizations_url":"https://api.github.com/users/fperez/orgs","repos_url":"https://api.github.com/users/fperez/repos","events_url":"https://api.github.com/users/fperez/events{/privacy}","received_events_url":"https://api.github.com/users/fperez/received_events","type":"User","site_admin":false},"labels":[{"id":37334597,"node_id":"MDU6TGFiZWwzNzMzNDU5Nw==","url":"https://api.github.com/repos/scipy/scipy/labels/defect","name":"defect","color":"e11d21","default":false},{"id":37340722,"node_id":"MDU6TGFiZWwzNzM0MDcyMg==","url":"https://api.github.com/repos/scipy/scipy/labels/scipy.stats","name":"scipy.stats","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-04-08T06:47:38Z","updated_at":"2018-12-17T23:43:16Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The `logcdf` method for `stats.hypergeom.logcdf` produces for certain parameters non-negative outputs, leading to probabilities > 1.\r\n\r\n### Reproducing code example:\r\n```python\r\nimport numpy as np\r\nimport scipy.stats as st\r\n\r\nx, M, n, N = 40, 1600, 50, 300\r\nlcdf = st.hypergeom.logcdf(x, M, n, N)\r\nprint(\"logcdf:\", lcdf)\r\nprint(\"probab:\", repr(np.exp(lcdf)))\r\n```\r\nresults on my system (MacOS w/scipy installed from Anaconda) in:\r\n\r\n```\r\nlogcdf: 7.578382366088447e-13\r\nprobab: 1.0000000000007578\r\n```\r\n\r\nFor reference, using this R code (with the same values of x, M, n and N as above):\r\n\r\n```R\r\nlcdf <- phyper(x, n, M-n, N, lower.tail=TRUE, log=TRUE)\r\ncat(\"logcdf:\", sprintf(\"%.12e\", lcdf), \"\\n\")\r\ncat(\"probab:\", exp(lcdf))\r\n```\r\n\r\nI get:\r\n\r\n```\r\nlogcdf: -7.565148879229e-23 \r\nprobab: 1\r\n ```\r\n\r\nNote the call to `phyper` needs to be made with slight adjustments to the parameters to match the conventions in the R definition, as warned in the [scipy.stats docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.hypergeom.html#scipy.stats.hypergeom). See the R docs for this function [here](https://www.rdocumentation.org/packages/stats/versions/3.4.3/topics/Hypergeometric).\r\n\r\n### Scipy/Numpy/Python version information:\r\n```\r\nscipy: 1.0.1\r\nnumpy: 1.14.2\r\nsys:   sys.version_info(major=3, minor=6, micro=5, releaselevel='final', serial=0)\r\n ```"},"comment":{"url":"https://api.github.com/repos/scipy/scipy/issues/comments/448041509","html_url":"https://github.com/scipy/scipy/issues/8692#issuecomment-448041509","issue_url":"https://api.github.com/repos/scipy/scipy/issues/8692","id":448041509,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0ODA0MTUwOQ==","user":{"login":"IandJMSmith","id":42864409,"node_id":"MDQ6VXNlcjQyODY0NDA5","avatar_url":"https://avatars0.githubusercontent.com/u/42864409?v=4","gravatar_id":"","url":"https://api.github.com/users/IandJMSmith","html_url":"https://github.com/IandJMSmith","followers_url":"https://api.github.com/users/IandJMSmith/followers","following_url":"https://api.github.com/users/IandJMSmith/following{/other_user}","gists_url":"https://api.github.com/users/IandJMSmith/gists{/gist_id}","starred_url":"https://api.github.com/users/IandJMSmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IandJMSmith/subscriptions","organizations_url":"https://api.github.com/users/IandJMSmith/orgs","repos_url":"https://api.github.com/users/IandJMSmith/repos","events_url":"https://api.github.com/users/IandJMSmith/events{/privacy}","received_events_url":"https://api.github.com/users/IandJMSmith/received_events","type":"User","site_admin":false},"created_at":"2018-12-17T23:43:16Z","updated_at":"2018-12-17T23:43:16Z","author_association":"NONE","body":"If you look at the Notes on Hypergeometric distribution functions in \r\n[https://stats.stackexchange.com/questions/279793/computational-shortcuts-approximations-for-beta-binomial-and-beta-negative-binom?noredirect=1&lq=1](url)\r\nthey will tell you how to calculate the pmf, cdf & sf for the Hypergeometric distribution.\r\n\r\nThe _logpmf function used in your code loses accuracy when the parameter values are large but it is accurate enough to get _logcdf correct to about 12 figures for the example reported as giving the problem.\r\n\r\nWith the recommended method for calculating the cdf in the Notes referenced above\r\nfor (i+½).(m+½) <= (n-½).(k-½)\r\nln(Pr(X<=i;n,k,m)) = ln(cf(i))+_logpmf(X=i;n,k,m)\r\nfor (i+½).(m+½) > (n-½).(k-½)\r\nln(Pr(X<=i;n,k,m)) = log1p(-Pr(X<=(n-i-1);n,m-k,m))\r\n\r\nIf you think about it, you cannot reasonably expect to use logsumexp to operate on all the _logpmf values and get the answer 0.\r\n\r\nThat said, I can provide you either with accurate answers for any examples you are interested in or with examples and answers if you would prefer. \r\n\r\nIf you are interested in the code for the Hypergeometric distribution, I can either add parameters to the pmf, cdf and sf functions to indicate that logs of the answers are required or add logpmf, logcdf and logsf functions if you would prefer. You are then welcome to modify the code yourself.\r\n\r\nI'm afraid I don't know how things are supposed to work around here and I certainly don't want to offend anyone, so please ignore this if I'm upsetting everyone.\r\n"}}
{"actor":{"display_login":"IandJMSmith"}}