{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1086","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1086/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1086/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1086/events","html_url":"https://github.com/dask/dask/issues/1086","id":146371821,"number":1086,"title":"DataFrame correlations are broken?","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-06T16:50:36Z","updated_at":"2016-04-06T16:50:40Z","closed_at":null,"body":"These seem to break with unknown divisions (although perhaps this is appropriate):\r\n\r\n```python\r\n>>> df.tip_amount.corr(df.payment_type == 2).compute()\r\n/opt/anaconda/lib/python2.7/site-packages/dask/dataframe/core.pyc in corr(self, other, method, min_periods)\r\n   1235             raise NotImplementedError(\"Only Pearson correlation has been \"\r\n   1236                                       \"implemented\")\r\n-> 1237         df = concat([self, other], axis=1)\r\n   1238         return cov_corr(df, min_periods, corr=True, scalar=True)\r\n   1239 \r\n\r\n/opt/anaconda/lib/python2.7/site-packages/dask/dataframe/multi.pyc in concat(dfs, axis, join, interleave_partitions)\r\n    552     else:\r\n    553         if axis == 1:\r\n--> 554              raise ValueError('Unable to concatenate DataFrame with unknown '\r\n    555                               'division specifying axis=1')\r\n    556         else:\r\n\r\nValueError: Unable to concatenate DataFrame with unknown division specifying axis=1\r\n```\r\n\r\nAlso I seem to be getting wrong results:\r\n\r\n```python\r\n>>> df[['tip_amount', 'payment_type']].corr().compute()\r\n               tip_amount    payment_type\r\ntip_amount       0.999169       -0.033852\r\npayment_type\t-0.033852        8.420302\r\n\r\n>>> df.head(1000)[['tip_amount', 'payment_type']].corr()\r\n               tip_amount    payment_type\r\ntip_amount       1.000000       -0.559584\r\npayment_type    -0.559584        1.000000\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/206461740","html_url":"https://github.com/dask/dask/issues/1086#issuecomment-206461740","issue_url":"https://api.github.com/repos/dask/dask/issues/1086","id":206461740,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2016-04-06T16:50:40Z","updated_at":"2016-04-06T16:50:40Z","body":"cc @jcrist "}}