{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1375","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1375/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1375/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1375/events","html_url":"https://github.com/dask/dask/issues/1375","id":165678966,"number":1375,"title":"`dd.merge` returns incorrect results","user":{"login":"ankravch","id":10063309,"avatar_url":"https://avatars.githubusercontent.com/u/10063309?v=3","gravatar_id":"","url":"https://api.github.com/users/ankravch","html_url":"https://github.com/ankravch","followers_url":"https://api.github.com/users/ankravch/followers","following_url":"https://api.github.com/users/ankravch/following{/other_user}","gists_url":"https://api.github.com/users/ankravch/gists{/gist_id}","starred_url":"https://api.github.com/users/ankravch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankravch/subscriptions","organizations_url":"https://api.github.com/users/ankravch/orgs","repos_url":"https://api.github.com/users/ankravch/repos","events_url":"https://api.github.com/users/ankravch/events{/privacy}","received_events_url":"https://api.github.com/users/ankravch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-14T23:10:35Z","updated_at":"2016-07-16T05:22:34Z","closed_at":null,"body":"Before doing smth advanced It would be priceless to make sure that dask does basic operations correctly on largish data - varying npartitions, single machine or cluster compute.\r\n\r\n```python\r\n[akravchenko@hnode1 nfs]$ python\r\nPython 3.5.1 |Anaconda custom (64-bit)| (default, Dec  7 2015, 11:16:01)\r\n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import dask.dataframe as dd\r\n>>> import pandas as pd\r\n>>> import numpy as np\r\n>>> import dask\r\n>>> from dask.distributed import Executor, progress\r\n>>> from dask.diagnostics import ProgressBar\r\n>>>\r\n>>> f0=pd.DataFrame({'v0':np.random.randint(1,10000000, (100000000,))})\r\n>>> f0_d=dd.from_pandas(f0, npartitions=100)\r\n>>> f1=f0[['v0']].drop_duplicates()\r\n>>> f1_d=dd.from_pandas(f1, npartitions=10) #npartitions=1 seems to give a correct join result\r\n```\r\n\r\nwhere\r\n\r\n```python\r\n>>> len(f0)\r\n100000000\r\n>>> len(f1)\r\n9999546\r\n```\r\n\r\njoin from dask \r\n\r\n```python\r\n>>> res=dd.merge(f0_d,f1_d, on='v0', how='inner').count()\r\n>>> dask.compute(res)\r\n(v0    130892991\r\ndtype: int64,)\r\n>>>\r\n>>> dask.compute(res)\r\n(v0    109156296\r\ndtype: int64,)\r\n>>>\r\n```\r\n\r\nand join from pandas:\r\n\r\n```python\r\n>>> res_pd=pd.merge(f0, f1, on='v0', how='inner')\r\n>>> res_pd.shape\r\n(100000000, 1)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/233109143","html_url":"https://github.com/dask/dask/issues/1375#issuecomment-233109143","issue_url":"https://api.github.com/repos/dask/dask/issues/1375","id":233109143,"user":{"login":"ankravch","id":10063309,"avatar_url":"https://avatars.githubusercontent.com/u/10063309?v=3","gravatar_id":"","url":"https://api.github.com/users/ankravch","html_url":"https://github.com/ankravch","followers_url":"https://api.github.com/users/ankravch/followers","following_url":"https://api.github.com/users/ankravch/following{/other_user}","gists_url":"https://api.github.com/users/ankravch/gists{/gist_id}","starred_url":"https://api.github.com/users/ankravch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankravch/subscriptions","organizations_url":"https://api.github.com/users/ankravch/orgs","repos_url":"https://api.github.com/users/ankravch/repos","events_url":"https://api.github.com/users/ankravch/events{/privacy}","received_events_url":"https://api.github.com/users/ankravch/received_events","type":"User","site_admin":false},"created_at":"2016-07-16T05:22:34Z","updated_at":"2016-07-16T05:22:34Z","body":"And on my Mac (OS X 10.10.5):\r\n\r\n```python\r\nAntons-MacBook-Pro:~ akravchenko$ ipython\r\nPython 3.5.1 |Anaconda 4.0.0 (x86_64)| (default, Dec  7 2015, 11:24:55) \r\nType \"copyright\", \"credits\" or \"license\" for more information.\r\n\r\nIPython 4.1.2 -- An enhanced Interactive Python.\r\n?         -> Introduction and overview of IPython's features.\r\n%quickref -> Quick reference.\r\nhelp      -> Python's own help system.\r\nobject?   -> Details about 'object', use 'object??' for extra details.\r\n\r\nIn [1]: %paste\r\n>>> import dask.dataframe as dd\r\n>>> import pandas as pd\r\n>>> import numpy as np\r\n>>> import dask\r\n\r\n## -- End pasted text --\r\n\r\nIn [2]: %paste\r\n>>> f0=pd.DataFrame({'v0':np.random.randint(1,10000000, (100000000,))})\r\n>>> f0_d=dd.from_pandas(f0, npartitions=100)\r\n>>> f1=f0[['v0']].drop_duplicates()\r\n>>> f1_d=dd.from_pandas(f1, npartitions=10)\r\n\r\n## -- End pasted text --\r\n\r\nIn [3]: len(f0)\r\nOut[3]: 100000000\r\n\r\nIn [4]: len(f1)\r\nOut[4]: 9999543\r\n\r\nIn [5]: res=dd.merge(f0_d,f1_d, on='v0', how='inner')\r\n\r\nIn [6]: res\r\nOut[6]: dd.DataFrame<hash-jo..., npartitions=100>\r\n\r\nIn [7]: t = res.compute()\r\n\r\nIn [8]: t.count()\r\nOut[8]: \r\nv0    115868592\r\ndtype: int64\r\n```"}}
{"actor":{"display_login":"ankravch"}}