{"action":"created","issue":{"url":"https://api.github.com/repos/bokeh/datashader/issues/129","repository_url":"https://api.github.com/repos/bokeh/datashader","labels_url":"https://api.github.com/repos/bokeh/datashader/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/bokeh/datashader/issues/129/comments","events_url":"https://api.github.com/repos/bokeh/datashader/issues/129/events","html_url":"https://github.com/bokeh/datashader/issues/129","id":144275296,"number":129,"title":"Recommended file format for large files","user":{"login":"jbednar","id":1695496,"avatar_url":"https://avatars.githubusercontent.com/u/1695496?v=3","gravatar_id":"","url":"https://api.github.com/users/jbednar","html_url":"https://github.com/jbednar","followers_url":"https://api.github.com/users/jbednar/followers","following_url":"https://api.github.com/users/jbednar/following{/other_user}","gists_url":"https://api.github.com/users/jbednar/gists{/gist_id}","starred_url":"https://api.github.com/users/jbednar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbednar/subscriptions","organizations_url":"https://api.github.com/users/jbednar/orgs","repos_url":"https://api.github.com/users/jbednar/repos","events_url":"https://api.github.com/users/jbednar/events{/privacy}","received_events_url":"https://api.github.com/users/jbednar/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bokeh/datashader/labels/ready","name":"ready","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-29T14:06:59Z","updated_at":"2016-05-05T13:48:41Z","closed_at":null,"body":"Datashader is agnostic about file formats, working with anything that can be loaded into a dataframe-like object.  But because datashader focuses on having good performance for large datasets, the performance of the file format is a major factor in the usability of the library.  Thus we should use examples that serve to guide users towards good solutions for their own problems, recommending and demonstrating approaches that we find to work well.\r\n\r\nRight now, our examples use CSV and castra format.  It is of course important to show a CSV example, since nearly every dataset can be obtained in CSV for import into the library.  However, CSV is highly inefficient in both file size and reading speed, and it also truncates floating-point precision  in ways that are problematic when zooming in closely to a dataset.\r\n\r\n[Castra](https://pypi.python.org/pypi/castra) is a relatively high-performance binary format that works well for the large datasets in the examples, but it is not yet a mature project, and is not available on the main conda channel.  Should we invest in making castra be more fully supported?  If not, I think we should choose another binary format (HDF5?) to use for our examples."},"comment":{"url":"https://api.github.com/repos/bokeh/datashader/issues/comments/217158841","html_url":"https://github.com/bokeh/datashader/issues/129#issuecomment-217158841","issue_url":"https://api.github.com/repos/bokeh/datashader/issues/129","id":217158841,"user":{"login":"miktoki","id":17086561,"avatar_url":"https://avatars.githubusercontent.com/u/17086561?v=3","gravatar_id":"","url":"https://api.github.com/users/miktoki","html_url":"https://github.com/miktoki","followers_url":"https://api.github.com/users/miktoki/followers","following_url":"https://api.github.com/users/miktoki/following{/other_user}","gists_url":"https://api.github.com/users/miktoki/gists{/gist_id}","starred_url":"https://api.github.com/users/miktoki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miktoki/subscriptions","organizations_url":"https://api.github.com/users/miktoki/orgs","repos_url":"https://api.github.com/users/miktoki/repos","events_url":"https://api.github.com/users/miktoki/events{/privacy}","received_events_url":"https://api.github.com/users/miktoki/received_events","type":"User","site_admin":false},"created_at":"2016-05-05T13:48:41Z","updated_at":"2016-05-05T13:48:41Z","body":"Is [`bcolz`](https://github.com/Blosc/bcolz) an option? It seems fast, compact and somewhat pydata-friendly (numpy + import/export to pandas,HDF5), plus it intends to support GPU. It is based on `Blosc` like Castra."}}