{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21547","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21547/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21547/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21547/events","html_url":"https://github.com/apache/spark/pull/21547","id":331811914,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk0NDUwNjU3","number":21547,"title":"[SPARK-24538][SQL] ByteArrayDecimalType support push down to the data sources","user":{"login":"wangyum","id":5399861,"node_id":"MDQ6VXNlcjUzOTk4NjE=","avatar_url":"https://avatars0.githubusercontent.com/u/5399861?v=4","gravatar_id":"","url":"https://api.github.com/users/wangyum","html_url":"https://github.com/wangyum","followers_url":"https://api.github.com/users/wangyum/followers","following_url":"https://api.github.com/users/wangyum/following{/other_user}","gists_url":"https://api.github.com/users/wangyum/gists{/gist_id}","starred_url":"https://api.github.com/users/wangyum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wangyum/subscriptions","organizations_url":"https://api.github.com/users/wangyum/orgs","repos_url":"https://api.github.com/users/wangyum/repos","events_url":"https://api.github.com/users/wangyum/events{/privacy}","received_events_url":"https://api.github.com/users/wangyum/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-06-13T01:46:43Z","updated_at":"2018-06-14T02:42:39Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21547","html_url":"https://github.com/apache/spark/pull/21547","diff_url":"https://github.com/apache/spark/pull/21547.diff","patch_url":"https://github.com/apache/spark/pull/21547.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n[ByteArrayDecimalType](https://github.com/apache/spark/blob/e28eb431146bcdcaf02a6f6c406ca30920592a6a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L230) support push down to the data sources.\r\n\r\n## How was this patch tested?\r\nunit tests and manual tests.\r\n\r\n**manual tests**:\r\n```scala\r\nspark.range(10000000).selectExpr(\"id\", \"cast(id as decimal(9)) as d1\", \"cast(id as decimal(9, 2)) as d2\", \"cast(id as decimal(18)) as d3\", \"cast(id as decimal(18, 4)) as d4\", \"cast(id as decimal(38)) as d5\", \"cast(id as decimal(38, 18)) as d6\").coalesce(1).write.option(\"parquet.block.size\", 1048576).parquet(\"/tmp/spark/parquet/decimal\")\r\nval df = spark.read.parquet(\"/tmp/spark/parquet/decimal/\")\r\n// Only read about 1 MB data\r\ndf.filter(\"d6 = 10000\").show\r\nspark.sql(\"set spark.sql.parquet.filterPushdown=false\")\r\n// Read 174.3 MB data\r\ndf.filter(\"d6 = 10000\").show\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/397151283","html_url":"https://github.com/apache/spark/pull/21547#issuecomment-397151283","issue_url":"https://api.github.com/repos/apache/spark/issues/21547","id":397151283,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzE1MTI4Mw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-06-14T02:42:39Z","updated_at":"2018-06-14T02:42:39Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}