{"action":"closed","issue":{"url":"https://api.github.com/repos/bashtage/linearmodels/issues/182","repository_url":"https://api.github.com/repos/bashtage/linearmodels","labels_url":"https://api.github.com/repos/bashtage/linearmodels/issues/182/labels{/name}","comments_url":"https://api.github.com/repos/bashtage/linearmodels/issues/182/comments","events_url":"https://api.github.com/repos/bashtage/linearmodels/issues/182/events","html_url":"https://github.com/bashtage/linearmodels/issues/182","id":385822664,"node_id":"MDU6SXNzdWUzODU4MjI2NjQ=","number":182,"title":"PanelOLS with two-way fixed effects / double clustered std. errors for a panel of millions of observations","user":{"login":"matbuechner","id":7684468,"node_id":"MDQ6VXNlcjc2ODQ0Njg=","avatar_url":"https://avatars2.githubusercontent.com/u/7684468?v=4","gravatar_id":"","url":"https://api.github.com/users/matbuechner","html_url":"https://github.com/matbuechner","followers_url":"https://api.github.com/users/matbuechner/followers","following_url":"https://api.github.com/users/matbuechner/following{/other_user}","gists_url":"https://api.github.com/users/matbuechner/gists{/gist_id}","starred_url":"https://api.github.com/users/matbuechner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matbuechner/subscriptions","organizations_url":"https://api.github.com/users/matbuechner/orgs","repos_url":"https://api.github.com/users/matbuechner/repos","events_url":"https://api.github.com/users/matbuechner/events{/privacy}","received_events_url":"https://api.github.com/users/matbuechner/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-29T16:36:09Z","updated_at":"2018-12-08T22:47:04Z","closed_at":"2018-12-08T22:47:04Z","author_association":"NONE","body":"Hi, I have come across a performance issue in regards to two-way fixed effects and double clustered standard errors. So essentially I would like to use both entity and time effects as well as double clustered errors in PanelOLS. My panel is unbalanced with about 10 million observations in total, ~5000 entities, ~2000 time steps and say about half a dozen independent variables. \r\n\r\nWhen running this setup in PanelOLS I keep having trouble with the Python process memory blowing up, ultimately resulting in the Python process being killed by my machine. I suppose this should be replicable using other data sets. For now, please see below a minimal example with a generic set of data available here (sorry for the large file, but this appears to be a large data issue):\r\n[https://www.dropbox.com/s/u9mzwdlyqp1ylu3/LargePanel.csv?dl=0](https://www.dropbox.com/s/u9mzwdlyqp1ylu3/LargePanel.csv?dl=0)\r\n\r\n```python\r\nimport pandas as pd\r\n\r\nfrom linearmodels import PanelOLS\r\n\r\ndata = pd.read_csv('LargePanel.csv')\r\ndata['Time'] = pd.to_datetime(data['Time'])\r\ndata.set_index(['EntityNo', 'Time'], inplace=True)\r\n\r\n\r\nmod = PanelOLS(data['Y1'], data[['X1', 'X2', 'X3', 'X4']],\r\n               entity_effects=True, time_effects=True)\r\nres1 = mod.fit(cov_type='clustered', cluster_entity=True, cluster_time=True)\r\n```\r\n\r\nI would be interested to learn whether there is a way to make the procedure more memory / computationally efficient."}}
{"actor":{"display_login":"bashtage"}}