{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21556","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21556/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21556/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21556/events","html_url":"https://github.com/apache/spark/pull/21556","id":331974071,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk0NTcwMzYw","number":21556,"title":"[SPARK-24549][SQL] Support Decimal type push down to the parquet data sources","user":{"login":"wangyum","id":5399861,"node_id":"MDQ6VXNlcjUzOTk4NjE=","avatar_url":"https://avatars0.githubusercontent.com/u/5399861?v=4","gravatar_id":"","url":"https://api.github.com/users/wangyum","html_url":"https://github.com/wangyum","followers_url":"https://api.github.com/users/wangyum/followers","following_url":"https://api.github.com/users/wangyum/following{/other_user}","gists_url":"https://api.github.com/users/wangyum/gists{/gist_id}","starred_url":"https://api.github.com/users/wangyum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wangyum/subscriptions","organizations_url":"https://api.github.com/users/wangyum/orgs","repos_url":"https://api.github.com/users/wangyum/repos","events_url":"https://api.github.com/users/wangyum/events{/privacy}","received_events_url":"https://api.github.com/users/wangyum/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-06-13T12:32:35Z","updated_at":"2018-07-11T07:06:04Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21556","html_url":"https://github.com/apache/spark/pull/21556","diff_url":"https://github.com/apache/spark/pull/21556.diff","patch_url":"https://github.com/apache/spark/pull/21556.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSupport Decimal type push down to the parquet data sources.\r\nThe Decimal comparator used is: [`BINARY_AS_SIGNED_INTEGER_COMPARATOR`](https://github.com/apache/parquet-mr/blob/c6764c4a0848abf1d581e22df8b33e28ee9f2ced/parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveComparator.java#L224-L292).\r\n\r\n## How was this patch tested?\r\n\r\nunit tests and manual tests.\r\n\r\n**manual tests**:\r\n```scala\r\nspark.range(10000000).selectExpr(\"id\", \"cast(id as decimal(9)) as d1\", \"cast(id as decimal(9, 2)) as d2\", \"cast(id as decimal(18)) as d3\", \"cast(id as decimal(18, 4)) as d4\", \"cast(id as decimal(38)) as d5\", \"cast(id as decimal(38, 18)) as d6\").coalesce(1).write.option(\"parquet.block.size\", 1048576).parquet(\"/tmp/spark/parquet/decimal\")\r\nval df = spark.read.parquet(\"/tmp/spark/parquet/decimal/\")\r\nspark.sql(\"set spark.sql.parquet.filterPushdown.decimal=true\")\r\n// Only read about 1 MB data\r\ndf.filter(\"d2 = 10000\").show\r\n// Only read about 1 MB data\r\ndf.filter(\"d4 = 10000\").show\r\nspark.sql(\"set spark.sql.parquet.filterPushdown.decimal=false\")\r\n// Read 174.3 MB data\r\ndf.filter(\"d2 = 10000\").show\r\n// Read 174.3 MB data\r\ndf.filter(\"d4 = 10000\").show\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/404067189","html_url":"https://github.com/apache/spark/pull/21556#issuecomment-404067189","issue_url":"https://api.github.com/repos/apache/spark/issues/21556","id":404067189,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDA2NzE4OQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-07-11T07:06:04Z","updated_at":"2018-07-11T07:06:04Z","author_association":"NONE","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}