{"action":"created","issue":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/4952","repository_url":"https://api.github.com/repos/statsmodels/statsmodels","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/4952/labels{/name}","comments_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/4952/comments","events_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/4952/events","html_url":"https://github.com/statsmodels/statsmodels/issues/4952","id":352611276,"node_id":"MDU6SXNzdWUzNTI2MTEyNzY=","number":4952,"title":"Zero-Inflated Poisson/Negative Binomial: not predicting for arrays of different size?","user":{"login":"Irene-GM","id":8654047,"node_id":"MDQ6VXNlcjg2NTQwNDc=","avatar_url":"https://avatars2.githubusercontent.com/u/8654047?v=4","gravatar_id":"","url":"https://api.github.com/users/Irene-GM","html_url":"https://github.com/Irene-GM","followers_url":"https://api.github.com/users/Irene-GM/followers","following_url":"https://api.github.com/users/Irene-GM/following{/other_user}","gists_url":"https://api.github.com/users/Irene-GM/gists{/gist_id}","starred_url":"https://api.github.com/users/Irene-GM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Irene-GM/subscriptions","organizations_url":"https://api.github.com/users/Irene-GM/orgs","repos_url":"https://api.github.com/users/Irene-GM/repos","events_url":"https://api.github.com/users/Irene-GM/events{/privacy}","received_events_url":"https://api.github.com/users/Irene-GM/received_events","type":"User","site_admin":false},"labels":[{"id":163856,"node_id":"MDU6TGFiZWwxNjM4NTY=","url":"https://api.github.com/repos/statsmodels/statsmodels/labels/comp-discrete","name":"comp-discrete","color":"ededed","default":false},{"id":614491160,"node_id":"MDU6TGFiZWw2MTQ0OTExNjA=","url":"https://api.github.com/repos/statsmodels/statsmodels/labels/prio_elev","name":"prio_elev","color":"d93f0b","default":false},{"id":142960,"node_id":"MDU6TGFiZWwxNDI5NjA=","url":"https://api.github.com/repos/statsmodels/statsmodels/labels/type-bug","name":"type-bug","color":"ff9999","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/statsmodels/statsmodels/milestones/11","html_url":"https://github.com/statsmodels/statsmodels/milestone/11","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/milestones/11/labels","id":2546431,"node_id":"MDk6TWlsZXN0b25lMjU0NjQzMQ==","number":11,"title":"0.10","description":"currently for issues, PR that will not be available or ready for 0.9","creator":{"login":"josef-pkt","id":440735,"node_id":"MDQ6VXNlcjQ0MDczNQ==","avatar_url":"https://avatars1.githubusercontent.com/u/440735?v=4","gravatar_id":"","url":"https://api.github.com/users/josef-pkt","html_url":"https://github.com/josef-pkt","followers_url":"https://api.github.com/users/josef-pkt/followers","following_url":"https://api.github.com/users/josef-pkt/following{/other_user}","gists_url":"https://api.github.com/users/josef-pkt/gists{/gist_id}","starred_url":"https://api.github.com/users/josef-pkt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josef-pkt/subscriptions","organizations_url":"https://api.github.com/users/josef-pkt/orgs","repos_url":"https://api.github.com/users/josef-pkt/repos","events_url":"https://api.github.com/users/josef-pkt/events{/privacy}","received_events_url":"https://api.github.com/users/josef-pkt/received_events","type":"User","site_admin":false},"open_issues":269,"closed_issues":55,"state":"open","created_at":"2017-05-27T18:40:33Z","updated_at":"2018-08-22T14:58:25Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2018-08-21T16:07:26Z","updated_at":"2018-08-22T17:49:42Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI am modelling a discrete skewed distribution with several of the available methods in `statsmodels`, namely:  `Poisson`, `Negative Binomial`, `Zero Inflated Poisson`, and `Zero Inflated Negative Binomial`. What I am doing is training with 60% of my samples, and testing the model with the remaining 40%. For this 40%, I obtain the RMSE to have a performance metric for each method.\r\n\r\nThere are no problems with the Poisson and the Negative Binomial: I use the `.fit()` method on my training samples, and the `.predict()` method on the test samples. However, I observed that for the Zero-Inflated models, **I cannot predict for the testing set**, as you can see in this snippet:\r\n\r\n ```\r\n    print(\"Model: Zero Inflated Poisson\")\r\n    zip_mod = sm.ZeroInflatedPoisson(ytrain, xtrain).fit(method=\"newton\", maxiter=50)\r\n    **zip_mean_pred = zip_mod.predict(xtest)**\r\n    zip_ppf_obs = stats.poisson.ppf(q=0.95, mu=zip_mean_pred)\r\n    zip_rmse = np.sqrt(mean_squared_error(ytest, zip_ppf_obs))\r\n    print(\"RMSE Zero-Inflated Poisson\", zip_rmse)\r\n\r\n    print(\"Model: Zero Inflated Neg. Binomial\")\r\n    zinb_mod = sm.ZeroInflatedNegativeBinomialP(ytrain, xtrain).fit(method=\"newton\", maxiter=50)\r\n    zinb_pred = zinb_mod.predict(xtest)\r\n    zinb_rmse = np.sqrt(mean_squared_error(ytest, zinb_pred))\r\n    print(\"RMSE Zero-Inflated Negative Binomial: \", zinb_rmse)\r\n```\r\n\r\nThe error that I get says: \r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/irene/.../four_models.py\", line 569, in <module>\r\n    test_poi_nb_zip_zinb_tiny_subset(meta_m, m)\r\n  File \"/home/irene/.../four_models.py\", line 325, in test_poi_nb_zip_zinb_tiny_subset\r\n    **zinb_pred = zinb_mod.predict(xtest)**\r\n  File \"/home/irene/.conda/envs/defaultpy34/lib/python2.7/site-packages/statsmodels/base/model.py\", line 854, in predict\r\n    **kwargs)\r\n  File \"/home/irene/.conda/envs/defaultpy34/lib/python2.7/site-packages/statsmodels/discrete/count_model.py\", line 444, in predict\r\n    prob_zero = (1 - prob_main) + prob_main * np.exp(llf)\r\n**ValueError: operands could not be broadcast together with shapes (480,) (320,)** \r\n```\r\nPerhaps I am doing something wrong, but I do not understand why these two methods require a test sample for the predict method as big as the one used in the fitting. Is there any way to pass to these methods a test array with a different size than the training? I tried training and testing with the same dataset and it works.\r\n\r\nThanks a lot for all the work! :-) "},"comment":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/comments/415119963","html_url":"https://github.com/statsmodels/statsmodels/issues/4952#issuecomment-415119963","issue_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/4952","id":415119963,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTExOTk2Mw==","user":{"login":"Irene-GM","id":8654047,"node_id":"MDQ6VXNlcjg2NTQwNDc=","avatar_url":"https://avatars2.githubusercontent.com/u/8654047?v=4","gravatar_id":"","url":"https://api.github.com/users/Irene-GM","html_url":"https://github.com/Irene-GM","followers_url":"https://api.github.com/users/Irene-GM/followers","following_url":"https://api.github.com/users/Irene-GM/following{/other_user}","gists_url":"https://api.github.com/users/Irene-GM/gists{/gist_id}","starred_url":"https://api.github.com/users/Irene-GM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Irene-GM/subscriptions","organizations_url":"https://api.github.com/users/Irene-GM/orgs","repos_url":"https://api.github.com/users/Irene-GM/repos","events_url":"https://api.github.com/users/Irene-GM/events{/privacy}","received_events_url":"https://api.github.com/users/Irene-GM/received_events","type":"User","site_admin":false},"created_at":"2018-08-22T17:49:42Z","updated_at":"2018-08-22T17:49:42Z","author_association":"NONE","body":"I tried your fix and it works! \r\n```\r\nprint(\"Model: Zero Inflated Poisson\")\r\nzip_mod = sm.ZeroInflatedPoisson(ytrain, xtrain).fit(method=\"nm\", maxiter=50)\r\nzip_mean_pred = zip_mod.predict(xtest, exog_infl=np.ones((len(xtest), 1)))\r\nzip_ppf_obs = stats.poisson.ppf(q=0.95, mu=zip_mean_pred)\r\nzip_rmse = np.sqrt(mean_squared_error(ytest, zip_ppf_obs))\r\n\r\nprint(\"Model: Zero Inflated Neg. Binomial\")\r\nzinb_mod = sm.ZeroInflatedNegativeBinomialP(ytrain, xtrain).fit(method=\"nm\", maxiter=50)\r\nzinb_pred = zinb_mod.predict(xtest, exog_infl=np.ones((len(xtest), 1)))\r\nzinb_rmse = np.sqrt(mean_squared_error(ytest, zinb_pred))\r\n```\r\nIt gives some warnings as you said, like this:\r\n```\r\nHessianInversionWarning: Inverting hessian failed, no bse or cov_params available\r\n  'available', HessianInversionWarning)\r\n```\r\nBut it processes my testing dataset. Tried the \"nm\" and \"newton\" optimizers.\r\nSo now i can compute the RMSE:\r\n```\r\nRMSE ZIP: 1.5642490210960658\r\nRMSE ZINB:  0.50817991749367819\r\n```\r\nThanks for the super fast reaction, much appreciated!\r\n\r\nPS - Regarding this \"overdispersion\", is there any paper/manual you recommend to learn more about it? :-)\r\n"}}
{"actor":{"display_login":"Irene-GM"}}