{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22358","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22358/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22358/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22358/events","html_url":"https://github.com/apache/spark/pull/22358","id":357953686,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEzODQxNTA4","number":22358,"title":"[SPARK-25366][SQL]Zstd and brotil CompressionCodec are not supported for parquet files","user":{"login":"10110346","id":24688163,"node_id":"MDQ6VXNlcjI0Njg4MTYz","avatar_url":"https://avatars2.githubusercontent.com/u/24688163?v=4","gravatar_id":"","url":"https://api.github.com/users/10110346","html_url":"https://github.com/10110346","followers_url":"https://api.github.com/users/10110346/followers","following_url":"https://api.github.com/users/10110346/following{/other_user}","gists_url":"https://api.github.com/users/10110346/gists{/gist_id}","starred_url":"https://api.github.com/users/10110346/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/10110346/subscriptions","organizations_url":"https://api.github.com/users/10110346/orgs","repos_url":"https://api.github.com/users/10110346/repos","events_url":"https://api.github.com/users/10110346/events{/privacy}","received_events_url":"https://api.github.com/users/10110346/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-09-07T07:36:42Z","updated_at":"2018-09-08T04:44:12Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22358","html_url":"https://github.com/apache/spark/pull/22358","diff_url":"https://github.com/apache/spark/pull/22358.diff","patch_url":"https://github.com/apache/spark/pull/22358.patch"},"body":"## What changes were proposed in this pull request?\r\nHadoop2.6  and  hadoop2.7 do not contain zstd and brotil compressioncodec ,hadoop 3.1 also contains only zstd  compressioncodec .\r\n So I think we should remove zstd and brotil  for the time being.\r\n\r\n**set  `spark.sql.parquet.compression.codec=brotli`:**\r\nCaused by: org.apache.parquet.hadoop.BadConfigurationException: Class org.apache.hadoop.io.compress.BrotliCodec was not found\r\n        at org.apache.parquet.hadoop.CodecFactory.getCodec(CodecFactory.java:235)\r\n        at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.<init>(CodecFactory.java:142)\r\n        at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:206)\r\n        at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:189)\r\n        at org.apache.parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:153)\r\n        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:411)\r\n        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)\r\n        at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)\r\n        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:161)        \r\n        \r\n**set  `spark.sql.parquet.compression.codec=zstd`:**       \r\nCaused by: org.apache.parquet.hadoop.BadConfigurationException: Class org.apache.hadoop.io.compress.ZStandardCodec was not found\r\n        at org.apache.parquet.hadoop.CodecFactory.getCodec(CodecFactory.java:235)\r\n        at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.<init>(CodecFactory.java:142)\r\n        at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:206)\r\n        at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:189)\r\n        at org.apache.parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:153)\r\n        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:411)\r\n        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)\r\n        at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)\r\n        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:161)\r\n\r\n## How was this patch tested?\r\nExist unit test\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/419612604","html_url":"https://github.com/apache/spark/pull/22358#issuecomment-419612604","issue_url":"https://api.github.com/repos/apache/spark/issues/22358","id":419612604,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTYxMjYwNA==","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2018-09-08T04:44:12Z","updated_at":"2018-09-08T04:44:12Z","author_association":"MEMBER","body":"If the codecs are found, then we support it. One thing we should do might be to document to explicitly provide the codec but I am not sure how many users are confused about it."}}
{"actor":{"display_login":"HyukjinKwon"}}