{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/888","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/888/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/888/comments","events_url":"https://api.github.com/repos/dask/dask/issues/888/events","html_url":"https://github.com/dask/dask/issues/888","id":123580009,"number":888,"title":"Unstable dask._Frame.map_partitions behavior","user":{"login":"sscondie","id":328577,"avatar_url":"https://avatars.githubusercontent.com/u/328577?v=3","gravatar_id":"","url":"https://api.github.com/users/sscondie","html_url":"https://github.com/sscondie","followers_url":"https://api.github.com/users/sscondie/followers","following_url":"https://api.github.com/users/sscondie/following{/other_user}","gists_url":"https://api.github.com/users/sscondie/gists{/gist_id}","starred_url":"https://api.github.com/users/sscondie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sscondie/subscriptions","organizations_url":"https://api.github.com/users/sscondie/orgs","repos_url":"https://api.github.com/users/sscondie/repos","events_url":"https://api.github.com/users/sscondie/events{/privacy}","received_events_url":"https://api.github.com/users/sscondie/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-12-23T00:16:56Z","updated_at":"2016-03-08T23:00:11Z","closed_at":"2016-01-22T16:03:34Z","body":"Consider the following script operating on 15 csv files each about 16MB in size (each csv is an array of floats that is 10 columns and 100,000 rows).  \r\n\r\n```python\r\nimport dask.dataframe as dd\r\ndf = dd.read_csv(\"test_data_small/*.csv\")\r\nprint(df.npartitions)\r\nprint(df.mean().compute())\r\nprint(df.map_partitions(len,columns=df.columns).compute())\r\n```\r\n\r\nRunning this script on my machine (OS X (10.11.1), Python 2.7.11 [Anaconda 2.3.0 (x86_64)], pandas 0.17.1) produces (in approximately the same frequencies and in random order) one of the following results (for each result I give first a description and then stdout)\r\n\r\n\r\n**1. npartitions completes, mean completes, map_partitions seg faults** \r\n\r\n```text\r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\nSegmentation fault: 11\r\n```\r\n**2. npartitions completes, Fatal Python Error at mean**\r\n\r\n```text\r\n15\r\nFatal Python error: GC object already tracked\r\nAbort trap: 6\r\n```\r\n\r\n**3. npartitions completes, mean completes, Fatal Python Error at map_partitions**\r\n \r\n```text\r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\nFatal Python error: GC object already tracked\r\nAbort trap: 6\r\n```\r\n\r\n**4.  npartitions completes, seg fault at mean**\r\n\r\n```text\r\n15\r\nSegmentation fault: 11\r\n```\r\n\r\n**5.  Everything completes**\r\n\r\n```text\r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\n(100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001)\r\n```\r\n\r\nI reverted pandas back to 0.17.0 based on a suggestion from Matt [here] http://stackoverflow.com/questions/34128540/why-is-running-a-compute-in-dask-causing-fatal-python-error-gc-object-alrea  but that hasn't noticeably changed the results.\r\n\r\nI'm happy to debug this in dask, but there is at least some chance that this could be a pandas problem.  Has anyone else seen this kind of behavior?  If so, were you able to find an answer?"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/194007660","html_url":"https://github.com/dask/dask/issues/888#issuecomment-194007660","issue_url":"https://api.github.com/repos/dask/dask/issues/888","id":194007660,"user":{"login":"QCaudron","id":3931931,"avatar_url":"https://avatars.githubusercontent.com/u/3931931?v=3","gravatar_id":"","url":"https://api.github.com/users/QCaudron","html_url":"https://github.com/QCaudron","followers_url":"https://api.github.com/users/QCaudron/followers","following_url":"https://api.github.com/users/QCaudron/following{/other_user}","gists_url":"https://api.github.com/users/QCaudron/gists{/gist_id}","starred_url":"https://api.github.com/users/QCaudron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/QCaudron/subscriptions","organizations_url":"https://api.github.com/users/QCaudron/orgs","repos_url":"https://api.github.com/users/QCaudron/repos","events_url":"https://api.github.com/users/QCaudron/events{/privacy}","received_events_url":"https://api.github.com/users/QCaudron/received_events","type":"User","site_admin":false},"created_at":"2016-03-08T23:00:11Z","updated_at":"2016-03-08T23:00:11Z","body":"Thanks for the super fast answer - I was already updating to RC 0.18 after reading up pydata/pandas#11790. `len(df)` works fine now !"}}