{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1230","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1230/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1230/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1230/events","html_url":"https://github.com/dask/dask/issues/1230","id":158695904,"number":1230,"title":"read_csv sampling logic is not a generalized solution","user":{"login":"frol","id":304265,"avatar_url":"https://avatars.githubusercontent.com/u/304265?v=3","gravatar_id":"","url":"https://api.github.com/users/frol","html_url":"https://github.com/frol","followers_url":"https://api.github.com/users/frol/followers","following_url":"https://api.github.com/users/frol/following{/other_user}","gists_url":"https://api.github.com/users/frol/gists{/gist_id}","starred_url":"https://api.github.com/users/frol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frol/subscriptions","organizations_url":"https://api.github.com/users/frol/orgs","repos_url":"https://api.github.com/users/frol/repos","events_url":"https://api.github.com/users/frol/events{/privacy}","received_events_url":"https://api.github.com/users/frol/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-06-06T14:46:51Z","updated_at":"2016-06-06T16:31:35Z","closed_at":null,"body":"I have hit an issue with a dataset which has over 10k columns, so I obviously hit a bug with a misleading error message:\r\n\r\n```\r\n/opt/conda/lib/python3.5/site-packages/dask/async.py in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)\r\n    486                 _execute_task(task, data)  # Re-execute locally\r\n    487             else:\r\n--> 488                 raise(remote_exception(res, tb))\r\n    489         state['cache'][key] = res\r\n    490         finish_task(dsk, key, state, results, keyorder.get)\r\n\r\nValueError: ('Columns do not match', Index(['Unnamed: 0', 'Var1', 'Var2', 'Var3', 'Var4', 'Var5', 'Var6', 'Var7',\r\n       'Var8', 'Var9',...\r\n```\r\n\r\nOnce I bumped the `sample` size to `1MB` I got through the above error but hit another one described in a related issue #1166, when sample size is not enough to guess dtypes (I have very sparse character columns with a number of missings (>95%), which is not a so rare use-case).\r\n\r\nI do understand that from a performance point of view it is not a great idea to scan the whole dataset twice, but can be there implemented an option to force full scan (in fact, early stopping once `object` dtype is discovered can save some time and full scan won't take so long)?"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/224012660","html_url":"https://github.com/dask/dask/issues/1230#issuecomment-224012660","issue_url":"https://api.github.com/repos/dask/dask/issues/1230","id":224012660,"user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"created_at":"2016-06-06T16:31:35Z","updated_at":"2016-06-06T16:31:35Z","body":"Regarding the first error, that can definitely be fixed, your csv was wide enough that it wasn't grabbing all of the columns in the sample.\r\n\r\nIf the dtypes are known, you can pass in the dtypes to `read_csv`, and that will avoid your issue. In general I'd prefer to avoid recommending a full read to users, so making that cheap is a bit undesirable. Passing in a very large number to `sample` should also work, although it's a bit of a hack."}}
{"actor":{"display_login":"jcrist"}}