{"action":"created","issue":{"url":"https://api.github.com/repos/dask/fastparquet/issues/344","repository_url":"https://api.github.com/repos/dask/fastparquet","labels_url":"https://api.github.com/repos/dask/fastparquet/issues/344/labels{/name}","comments_url":"https://api.github.com/repos/dask/fastparquet/issues/344/comments","events_url":"https://api.github.com/repos/dask/fastparquet/issues/344/events","html_url":"https://github.com/dask/fastparquet/issues/344","id":339165956,"node_id":"MDU6SXNzdWUzMzkxNjU5NTY=","number":344,"title":"Don't default to gzip level 9","user":{"login":"seibert","id":425352,"node_id":"MDQ6VXNlcjQyNTM1Mg==","avatar_url":"https://avatars1.githubusercontent.com/u/425352?v=4","gravatar_id":"","url":"https://api.github.com/users/seibert","html_url":"https://github.com/seibert","followers_url":"https://api.github.com/users/seibert/followers","following_url":"https://api.github.com/users/seibert/following{/other_user}","gists_url":"https://api.github.com/users/seibert/gists{/gist_id}","starred_url":"https://api.github.com/users/seibert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seibert/subscriptions","organizations_url":"https://api.github.com/users/seibert/orgs","repos_url":"https://api.github.com/users/seibert/repos","events_url":"https://api.github.com/users/seibert/events{/privacy}","received_events_url":"https://api.github.com/users/seibert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-07T18:32:46Z","updated_at":"2018-07-08T16:50:12Z","closed_at":null,"author_association":"NONE","body":"Wondering why GZIP compression in fastparquet was so slow, I noticed that the gzip compression level is defaulted to level 9:\r\n\r\nhttps://github.com/dask/fastparquet/blob/master/fastparquet/compression.py#L16-L39\r\n\r\nGzip compression is extremely speed inefficient at level 9, and defaulting to this makes gzip very impractical.  A quick example from some parquet data I'm writing (\"create\" includes some constant time spent loading the data in memory + time for fastparquet to write):\r\n\r\n* gzip 1: create time = 64.46 sec, output size = 69 MB\r\n* gzip 6: create time =  73.94 sec, output size = 59 MB\r\n* gzip 9: create time =  229.48 sec, output size = 58 MB\r\n\r\nThe extra 2.5 minutes spent going from level 6 to level 9 is basically wasted in this case.  Setting the default compression level to 6 (matching the gzip command line utility) is probably the best option."},"comment":{"url":"https://api.github.com/repos/dask/fastparquet/issues/comments/403300642","html_url":"https://github.com/dask/fastparquet/issues/344#issuecomment-403300642","issue_url":"https://api.github.com/repos/dask/fastparquet/issues/344","id":403300642,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzMwMDY0Mg==","user":{"login":"martindurant","id":6042212,"node_id":"MDQ6VXNlcjYwNDIyMTI=","avatar_url":"https://avatars1.githubusercontent.com/u/6042212?v=4","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"created_at":"2018-07-08T16:50:12Z","updated_at":"2018-07-08T16:50:12Z","author_association":"MEMBER","body":"Happy to accept a PR where the default is level=6. \r\nStill would try zstd/lz4/lzo or even brotli (whatever that is!) too, if the other end can read them."}}
{"actor":{"display_login":"martindurant"}}