{"action":"created","issue":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055","repository_url":"https://api.github.com/repos/statsmodels/statsmodels","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/labels{/name}","comments_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/comments","events_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/events","html_url":"https://github.com/statsmodels/statsmodels/pull/3055","id":161327048,"number":3055,"title":"Distributed estimation","user":{"login":"lbybee","id":4998171,"avatar_url":"https://avatars.githubusercontent.com/u/4998171?v=3","gravatar_id":"","url":"https://api.github.com/users/lbybee","html_url":"https://github.com/lbybee","followers_url":"https://api.github.com/users/lbybee/followers","following_url":"https://api.github.com/users/lbybee/following{/other_user}","gists_url":"https://api.github.com/users/lbybee/gists{/gist_id}","starred_url":"https://api.github.com/users/lbybee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbybee/subscriptions","organizations_url":"https://api.github.com/users/lbybee/orgs","repos_url":"https://api.github.com/users/lbybee/repos","events_url":"https://api.github.com/users/lbybee/events{/privacy}","received_events_url":"https://api.github.com/users/lbybee/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/statsmodels/statsmodels/labels/comp-base","name":"comp-base","color":"ededed"},{"url":"https://api.github.com/repos/statsmodels/statsmodels/labels/comp-genmod","name":"comp-genmod","color":"ededed"},{"url":"https://api.github.com/repos/statsmodels/statsmodels/labels/comp-regression","name":"comp-regression","color":"ededed"},{"url":"https://api.github.com/repos/statsmodels/statsmodels/labels/type-enh","name":"type-enh","color":"99ffff"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/statsmodels/statsmodels/milestones/10","html_url":"https://github.com/statsmodels/statsmodels/milestones/0.9","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/milestones/10/labels","id":1362664,"number":10,"title":"0.9","description":"The milestone is currently for a rough roadmap","creator":{"login":"josef-pkt","id":440735,"avatar_url":"https://avatars.githubusercontent.com/u/440735?v=3","gravatar_id":"","url":"https://api.github.com/users/josef-pkt","html_url":"https://github.com/josef-pkt","followers_url":"https://api.github.com/users/josef-pkt/followers","following_url":"https://api.github.com/users/josef-pkt/following{/other_user}","gists_url":"https://api.github.com/users/josef-pkt/gists{/gist_id}","starred_url":"https://api.github.com/users/josef-pkt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josef-pkt/subscriptions","organizations_url":"https://api.github.com/users/josef-pkt/orgs","repos_url":"https://api.github.com/users/josef-pkt/repos","events_url":"https://api.github.com/users/josef-pkt/events{/privacy}","received_events_url":"https://api.github.com/users/josef-pkt/received_events","type":"User","site_admin":false},"open_issues":39,"closed_issues":2,"state":"open","created_at":"2015-10-18T23:31:11Z","updated_at":"2016-07-26T01:30:01Z","due_on":null,"closed_at":null},"comments":60,"created_at":"2016-06-21T00:47:01Z","updated_at":"2016-07-27T14:29:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/statsmodels/statsmodels/pulls/3055","html_url":"https://github.com/statsmodels/statsmodels/pull/3055","diff_url":"https://github.com/statsmodels/statsmodels/pull/3055.diff","patch_url":"https://github.com/statsmodels/statsmodels/pull/3055.patch"},"body":"Ok, I wanted to make a PR for this, there are still a couple of things that I need to fix but things are pretty close to done and I'm happy with the current approach.  \r\n\r\nThe key part of the current approach is a function `distributed_estimation`.  This function works by taking a generator for endog and exog, `endog_generator` and `exog_generator`, as well as a series of functions and key word arguments to be run on each machine and then used to recombine the results.  The generator approach allows for a variety of use cases and can handle a lot of the ideas discussed in the initial proposal.  For each data set yielded by the generators a model is initialized using `model_class` and `init_kwds` and then the function `estimation_method` is applied to the model along with the key words `fit_kwds` and `estimation_kwds`.  Finally, the results are recombined from each data set using `join_method`.  \r\n\r\nCurrently, this defaults to the distributed regularized approach discussed here:\r\n\r\nhttp://arxiv.org/pdf/1503.04337v3.pdf\r\n\r\nbut the way I've set things up means that the user should be able to apply any number of procedures here.\r\n\r\nThe current todo list:\r\n\r\n- [x] Fix hess_obs\r\n- [x] Fix joblib fit\r\n- [x] Add dask fit\r\n- [ ] Add formula handling\r\n- [ ] Add WLS/GLS for debiased regularized fit\r\n- [ ] Add duration for debiased regularized fit\r\n- [ ] Add likelihood result\r\n- [ ] Add data tests\r\n- [ ] Add pre-rolled generators\r\n\r\nLet me know if there are any comments/questions/criticisms, as I've said, it certainly isn't complete but I wanted to get this out there so I can start integrating any changes as I finish it up."},"comment":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/comments/235602840","html_url":"https://github.com/statsmodels/statsmodels/pull/3055#issuecomment-235602840","issue_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055","id":235602840,"user":{"login":"kshedden","id":2666691,"avatar_url":"https://avatars.githubusercontent.com/u/2666691?v=3","gravatar_id":"","url":"https://api.github.com/users/kshedden","html_url":"https://github.com/kshedden","followers_url":"https://api.github.com/users/kshedden/followers","following_url":"https://api.github.com/users/kshedden/following{/other_user}","gists_url":"https://api.github.com/users/kshedden/gists{/gist_id}","starred_url":"https://api.github.com/users/kshedden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kshedden/subscriptions","organizations_url":"https://api.github.com/users/kshedden/orgs","repos_url":"https://api.github.com/users/kshedden/repos","events_url":"https://api.github.com/users/kshedden/events{/privacy}","received_events_url":"https://api.github.com/users/kshedden/received_events","type":"User","site_admin":false},"created_at":"2016-07-27T14:29:38Z","updated_at":"2016-07-27T14:29:38Z","body":"You can look at either MICE or Mediation to see how to create a bare\nresults class of the appropriate type and populate it with estimates and\nstandard errors.\n\nWhen regularization is involved, I still think that the RegularizedResults\nclass should be returned, but if no regularization, you can return an\nOLSResults or GLMResults.  This will give people access to some\nmodel-specific features like predict options.  But it will also give people\nmethods and results that are not applicable (e.g. AIC and loglike), so you\nshould probably set those to NaN, but there is no way to know generically\nwhat all those results fields are.\n\nKerby\n\nOn Wed, Jul 27, 2016 at 9:11 PM, Leland Bybee <notifications@github.com>\nwrote:\n\n> One possibility would be to determine what results class is returned based\n> on the join method/as part of the join method. What I'm imagining is the\n> _join_debiased could return RegularizedResults and other join methods could\n> return a more standard likelihood result. Obviously, that makes the\n> distributed results less clear in general but it allows for more\n> flexibility with use cases/estimation methods.\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/statsmodels/statsmodels/pull/3055#issuecomment-235580105>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ACiww0lKpixcOOxPJxjUrlCiJZKeOcYkks5qZ1kegaJpZM4I6Q4M>\n> .\n>\n"}}
{"actor":{"display_login":"kshedden"}}