{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1108","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1108/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1108/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1108/events","html_url":"https://github.com/dask/dask/pull/1108","id":150342116,"number":1108,"title":"ENH BlockIOReader instead of textblock for read_csv","user":{"login":"wabu","id":39164,"avatar_url":"https://avatars.githubusercontent.com/u/39164?v=3","gravatar_id":"","url":"https://api.github.com/users/wabu","html_url":"https://github.com/wabu","followers_url":"https://api.github.com/users/wabu/followers","following_url":"https://api.github.com/users/wabu/following{/other_user}","gists_url":"https://api.github.com/users/wabu/gists{/gist_id}","starred_url":"https://api.github.com/users/wabu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wabu/subscriptions","organizations_url":"https://api.github.com/users/wabu/orgs","repos_url":"https://api.github.com/users/wabu/repos","events_url":"https://api.github.com/users/wabu/events{/privacy}","received_events_url":"https://api.github.com/users/wabu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dask/dask/labels/dataframe","name":"dataframe","color":"fbca04"},{"url":"https://api.github.com/repos/dask/dask/labels/io","name":"io","color":"e99695"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2016-04-22T11:16:43Z","updated_at":"2016-05-04T13:22:43Z","closed_at":"2016-04-29T00:49:48Z","pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/1108","html_url":"https://github.com/dask/dask/pull/1108","diff_url":"https://github.com/dask/dask/pull/1108.diff","patch_url":"https://github.com/dask/dask/pull/1108.patch"},"body":"I have added an IO class, the `BlockIOReader` that is used to efficiently access a text-block with `pd.read_csv.` As the `BlockIOReader` supports `readline`, it is used for `textblock` as well."},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/216862926","html_url":"https://github.com/dask/dask/pull/1108#issuecomment-216862926","issue_url":"https://api.github.com/repos/dask/dask/issues/1108","id":216862926,"user":{"login":"wabu","id":39164,"avatar_url":"https://avatars.githubusercontent.com/u/39164?v=3","gravatar_id":"","url":"https://api.github.com/users/wabu","html_url":"https://github.com/wabu","followers_url":"https://api.github.com/users/wabu/followers","following_url":"https://api.github.com/users/wabu/following{/other_user}","gists_url":"https://api.github.com/users/wabu/gists{/gist_id}","starred_url":"https://api.github.com/users/wabu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wabu/subscriptions","organizations_url":"https://api.github.com/users/wabu/orgs","repos_url":"https://api.github.com/users/wabu/repos","events_url":"https://api.github.com/users/wabu/events{/privacy}","received_events_url":"https://api.github.com/users/wabu/received_events","type":"User","site_admin":false},"created_at":"2016-05-04T13:22:43Z","updated_at":"2016-05-04T13:22:43Z","body":"Hi, as #1116 is now working on our data, here are some tests reading in an xz file with 20 blocks of about 2GB each. As for seeking in xz, using a blocksize of `2GB` is most efficient, but pressures memory. Using smaller chunk has some overhead on seeking for unaligned blocks. Here are the numbers:\r\n\r\nblocksize | 2GB | 1GB | 500MB\r\n------------ | --- | --- | ------\r\n#1116 |  MemoryError | 1min 54.3s | 3min 21.2s\r\n#1108 | 56.4s | 1min  0.6s | 1min 16.3s\r\n\r\nSo it seems having more blocks on it's own gives some overhead for #1116, as the overhead for seeking into the unaligned blocks is only 10s for #1108 for the 500MB case compared to 1GB, but around 90s for #1116. \r\n"}}