{"action":"closed","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1022","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1022/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1022/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1022/events","html_url":"https://github.com/dask/dask/issues/1022","id":137964863,"number":1022,"title":"read_csv refactor and test suite","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dask/dask/labels/dataframe","name":"dataframe","color":"fbca04"},{"url":"https://api.github.com/repos/dask/dask/labels/io","name":"io","color":"e99695"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-02T19:14:22Z","updated_at":"2016-05-02T23:15:30Z","closed_at":"2016-05-02T23:15:30Z","body":"A decent fraction of dask issues today are about `dask.dataframe.read_csv` failing on a new keyword argument.  Additionally, as `distributed` csv readers come online we're going to want to pull out logic from the current `read_csv` function.  It might be worth spending some quality time thinking about how best to make `read_csv` robust and how to make it more generalizable.\r\n\r\n### Current state\r\n\r\nGenerally the current `dd.read_csv` algorithm works as follows:\r\n\r\n1.  Use `pd.read_csv` and the provided keyword arguments to create a dataframe from the bit the top of the CSV file\r\n2.  Use this head dataframe to produce a *new set of keyword arguments* that will produce the same kind of dataframe when given a block of bytes from the center of the file (given clean endline boundaries).\r\n3.  Create a graph that calls `pd.read_csv` on those blocks with those keyword arguments.\r\n\r\n### Things to fix\r\n\r\nThere are some known bugs in existing issues.  \r\n\r\nThere is also the possibility of rethinking how we approach this.  Arguments that change the column names like `parse_dates` introduce a lot of headache into constructing the right kwargs.  I wonder if it might be cleaner to actually pull off the header line from the CSV file and stitch it onto each of the block reads before calling `pd.read_csv`.  This might require less error-prone logic on our part and be more robust generally.\r\n\r\nWe need to pull error-prone logic outside of the `dd.read_csv` function for reuse in other functions, like `distributed.hdfs.read_csv`.\r\n\r\nDepending on how tricky our solution is we might want a more extensive test suite.  I hear that Pandas has a decent one for this.  Perhaps we can steal something."}}