{"action":"created","issue":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055","repository_url":"https://api.github.com/repos/statsmodels/statsmodels","labels_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/labels{/name}","comments_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/comments","events_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055/events","html_url":"https://github.com/statsmodels/statsmodels/pull/3055","id":161327048,"number":3055,"title":"Distributed estimation","user":{"login":"lbybee","id":4998171,"avatar_url":"https://avatars.githubusercontent.com/u/4998171?v=3","gravatar_id":"","url":"https://api.github.com/users/lbybee","html_url":"https://github.com/lbybee","followers_url":"https://api.github.com/users/lbybee/followers","following_url":"https://api.github.com/users/lbybee/following{/other_user}","gists_url":"https://api.github.com/users/lbybee/gists{/gist_id}","starred_url":"https://api.github.com/users/lbybee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbybee/subscriptions","organizations_url":"https://api.github.com/users/lbybee/orgs","repos_url":"https://api.github.com/users/lbybee/repos","events_url":"https://api.github.com/users/lbybee/events{/privacy}","received_events_url":"https://api.github.com/users/lbybee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-06-21T00:47:01Z","updated_at":"2016-06-24T17:55:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/statsmodels/statsmodels/pulls/3055","html_url":"https://github.com/statsmodels/statsmodels/pull/3055","diff_url":"https://github.com/statsmodels/statsmodels/pull/3055.diff","patch_url":"https://github.com/statsmodels/statsmodels/pull/3055.patch"},"body":"Ok, I wanted to make a PR for this, there are still a couple of things that I need to fix but things are pretty close to done and I'm happy with the current approach.  \r\n\r\nThe key part of the current approach is a function `distributed_estimation`.  This function works by taking a generator for endog and exog, `endog_generator` and `exog_generator`, as well as a series of functions and key word arguments to be run on each machine and then used to recombine the results.  The generator approach allows for a variety of use cases and can handle a lot of the ideas discussed in the initial proposal.  For each data set yielded by the generators a model is initialized using `model_class` and `init_kwds` and then the function `estimation_method` is applied to the model along with the key words `fit_kwds` and `estimation_kwds`.  Finally, the results are recombined from each data set using `join_method`.  \r\n\r\nCurrently, this defaults to the distributed regularized approach discussed here:\r\n\r\nhttp://arxiv.org/pdf/1503.04337v3.pdf\r\n\r\nbut the way I've set things up means that the user should be able to apply any number of procedures here.\r\n\r\nI'm planning to have a couple of examples put together sometime this week to go along with my GSoC blog post on Friday.\r\n\r\nThere are still a number of things that I want to do with this, I'm planning to have things finalized in a couple of weeks.  A number of the smaller todos are commented in the code but I wanted to mention some of the bigger things still to do:\r\n\r\n- There is a bug with the GLM distributed regularized estimation code, should have it fixed soon.\r\n- Need to add WLS/GLS to the distributed regularized estimation.\r\n- I've considered a more general approach to the generators, though I don't have much of a use case in mind it might be nice to allow for more generators than just the data generators.  Currently a subset of variables is needed for the distributed regularized estimation for each machine, this is handled internally but it could be handled with a generator as well.\r\n- `fit_kwds` could probably be handled differently, maybe passed in within `estimation_kwds`.\r\n- Possibly replace `distributed_estimation` with a comparable class.\r\n\r\nLet me know if there are any comments/questions/criticisms, as I've said, it certainly isn't complete but I wanted to get this out there so I can start integrating any changes as I finish it up."},"comment":{"url":"https://api.github.com/repos/statsmodels/statsmodels/issues/comments/228415315","html_url":"https://github.com/statsmodels/statsmodels/pull/3055#issuecomment-228415315","issue_url":"https://api.github.com/repos/statsmodels/statsmodels/issues/3055","id":228415315,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars.githubusercontent.com/u/2354108?v=3","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T17:55:27Z","updated_at":"2016-06-24T17:55:27Z","body":"\n[![Coverage Status](https://coveralls.io/builds/6743375/badge)](https://coveralls.io/builds/6743375)\n\nCoverage increased (+0.04%) to 88.847% when pulling **9dcc379c98995dab3139470c01b91d0cfb14a830 on lbybee:distributed_estimation** into **032a9abb42efb57496f7ecfdd8eb53eea3a0389c on statsmodels:master**.\n"}}
{"actor":{"display_login":"coveralls"}}