{"action":"created","issue":{"url":"https://api.github.com/repos/apache/kafka/issues/2267","repository_url":"https://api.github.com/repos/apache/kafka","labels_url":"https://api.github.com/repos/apache/kafka/issues/2267/labels{/name}","comments_url":"https://api.github.com/repos/apache/kafka/issues/2267/comments","events_url":"https://api.github.com/repos/apache/kafka/issues/2267/events","html_url":"https://github.com/apache/kafka/pull/2267","id":196019927,"node_id":"MDExOlB1bGxSZXF1ZXN0OTgzMDU3MDQ=","number":2267,"title":"KAFKA-4514: Add Codec for ZStandard Compression","user":{"login":"dongjinleekr","id":2375128,"node_id":"MDQ6VXNlcjIzNzUxMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/2375128?v=4","gravatar_id":"","url":"https://api.github.com/users/dongjinleekr","html_url":"https://github.com/dongjinleekr","followers_url":"https://api.github.com/users/dongjinleekr/followers","following_url":"https://api.github.com/users/dongjinleekr/following{/other_user}","gists_url":"https://api.github.com/users/dongjinleekr/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjinleekr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjinleekr/subscriptions","organizations_url":"https://api.github.com/users/dongjinleekr/orgs","repos_url":"https://api.github.com/users/dongjinleekr/repos","events_url":"https://api.github.com/users/dongjinleekr/events{/privacy}","received_events_url":"https://api.github.com/users/dongjinleekr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"hachikuji","id":12502538,"node_id":"MDQ6VXNlcjEyNTAyNTM4","avatar_url":"https://avatars3.githubusercontent.com/u/12502538?v=4","gravatar_id":"","url":"https://api.github.com/users/hachikuji","html_url":"https://github.com/hachikuji","followers_url":"https://api.github.com/users/hachikuji/followers","following_url":"https://api.github.com/users/hachikuji/following{/other_user}","gists_url":"https://api.github.com/users/hachikuji/gists{/gist_id}","starred_url":"https://api.github.com/users/hachikuji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hachikuji/subscriptions","organizations_url":"https://api.github.com/users/hachikuji/orgs","repos_url":"https://api.github.com/users/hachikuji/repos","events_url":"https://api.github.com/users/hachikuji/events{/privacy}","received_events_url":"https://api.github.com/users/hachikuji/received_events","type":"User","site_admin":false},"assignees":[{"login":"hachikuji","id":12502538,"node_id":"MDQ6VXNlcjEyNTAyNTM4","avatar_url":"https://avatars3.githubusercontent.com/u/12502538?v=4","gravatar_id":"","url":"https://api.github.com/users/hachikuji","html_url":"https://github.com/hachikuji","followers_url":"https://api.github.com/users/hachikuji/followers","following_url":"https://api.github.com/users/hachikuji/following{/other_user}","gists_url":"https://api.github.com/users/hachikuji/gists{/gist_id}","starred_url":"https://api.github.com/users/hachikuji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hachikuji/subscriptions","organizations_url":"https://api.github.com/users/hachikuji/orgs","repos_url":"https://api.github.com/users/hachikuji/repos","events_url":"https://api.github.com/users/hachikuji/events{/privacy}","received_events_url":"https://api.github.com/users/hachikuji/received_events","type":"User","site_admin":false}],"milestone":null,"comments":122,"created_at":"2016-12-16T09:52:13Z","updated_at":"2018-11-15T15:50:50Z","closed_at":"2018-10-10T00:13:34Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/kafka/pulls/2267","html_url":"https://github.com/apache/kafka/pull/2267","diff_url":"https://github.com/apache/kafka/pull/2267.diff","patch_url":"https://github.com/apache/kafka/pull/2267.patch"},"body":"Hello. This PR resolves [KAFKA-4514: Add Codec for ZStandard Compression](https://issues.apache.org/jira/browse/KAFKA-4514). Please have a look when you are free. Since I am a total newbie of Apache Kafka, feel free to point out the deficiencies.\r\n\r\nAdd to the feature itself, I have a question: **Should we support an option for ZStandard compression level?**\r\n\r\nAccording to [ZStandard official documentation](https://github.com/facebook/zstd/blob/dev/lib/zstd.h), it supports compression level of 1 ~ 22. Because of that, [Hadoop added a new configuration option named \"io.compression.codec.zstd.level\", whose default value is 3](https://issues.apache.org/jira/browse/HADOOP-13578). In this PR, I configured the compression level to 1 as a temporary one but wondering following problems:\r\n\r\n- Should we provide a configurable option?\r\n- Would it better to change the default value, from 1 to another one?\r\n\r\nI am looking forward to your advice. Thanks."},"comment":{"url":"https://api.github.com/repos/apache/kafka/issues/comments/439087933","html_url":"https://github.com/apache/kafka/pull/2267#issuecomment-439087933","issue_url":"https://api.github.com/repos/apache/kafka/issues/2267","id":439087933,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTA4NzkzMw==","user":{"login":"dongjinleekr","id":2375128,"node_id":"MDQ6VXNlcjIzNzUxMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/2375128?v=4","gravatar_id":"","url":"https://api.github.com/users/dongjinleekr","html_url":"https://github.com/dongjinleekr","followers_url":"https://api.github.com/users/dongjinleekr/followers","following_url":"https://api.github.com/users/dongjinleekr/following{/other_user}","gists_url":"https://api.github.com/users/dongjinleekr/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjinleekr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjinleekr/subscriptions","organizations_url":"https://api.github.com/users/dongjinleekr/orgs","repos_url":"https://api.github.com/users/dongjinleekr/repos","events_url":"https://api.github.com/users/dongjinleekr/events{/privacy}","received_events_url":"https://api.github.com/users/dongjinleekr/received_events","type":"User","site_admin":false},"created_at":"2018-11-15T15:50:50Z","updated_at":"2018-11-15T15:50:50Z","author_association":"CONTRIBUTOR","body":"@davewat @scottcarey @ijuma Sorry for the late reply. In fact, I already investigated this issue (support compression level for zstd) but I concluded that it would be much better to put this issue into separated one and focus on implementing the ZSTD feature only.\r\n\r\nHere is why: All compression codecs (i.e., Gzip, Snappy, LZ4 and ZSTD) support some parameters to change the degree of compression; However, only LZ4 and ZSTD supports the concept of 'level' - in the case of GZIP and Snappy, they require the block size parameter, not 'level'.\r\n\r\nTo make the compression level feature available, we must modify the API signatures of `MemoryRecordsBuilder` to support compression level, and add some validation logic[^1]. It requires additional modifications to read `ProducerConfig` value and pass it into `MemoryRecordsBuilder.` Of course, this work requires a bunch of modifications and some policies on various codecs. It is why I decided to put off this issue and use the default level for ZSTD, that is, 3.\r\n\r\nHow about your opinion? Do you really need it? Does it sound reasonable?\r\n\r\n**If the answer is Yes, please file it to Jira with 'needs-kip' tag; Then, I will take the issue - I will make the proposal.**\r\n\r\n[^1]: Check whether given `CompressionCodec` supports the concept of 'level,' and whether given compression level is valid for the `CompressionCodec`. (e.g., ZSTD supports 22 levels but LZ4 supports 4 levels only.)"}}
{"actor":{"display_login":"dongjinleekr"}}