{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15898","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15898/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15898/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15898/events","html_url":"https://github.com/apache/spark/pull/15898","id":189582723,"number":15898,"title":"[SPARK-18457][SQL] ORC and other columnar formats using HiveShim read all columns when doing a simple count\t","user":{"login":"aray","id":981982,"avatar_url":"https://avatars.githubusercontent.com/u/981982?v=3","gravatar_id":"","url":"https://api.github.com/users/aray","html_url":"https://github.com/aray","followers_url":"https://api.github.com/users/aray/followers","following_url":"https://api.github.com/users/aray/following{/other_user}","gists_url":"https://api.github.com/users/aray/gists{/gist_id}","starred_url":"https://api.github.com/users/aray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aray/subscriptions","organizations_url":"https://api.github.com/users/aray/orgs","repos_url":"https://api.github.com/users/aray/repos","events_url":"https://api.github.com/users/aray/events{/privacy}","received_events_url":"https://api.github.com/users/aray/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-11-16T04:16:32Z","updated_at":"2016-11-18T15:43:03Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15898","html_url":"https://github.com/apache/spark/pull/15898","diff_url":"https://github.com/apache/spark/pull/15898.diff","patch_url":"https://github.com/apache/spark/pull/15898.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen reading zero columns (e.g., count(*)) from ORC or any other format that uses HiveShim, actually set the read column list to empty for Hive to use.\r\n\r\n## How was this patch tested?\r\n\r\nQuery correctness is handled by existing unit tests. I'm happy to add more if anyone can point out some case that is not covered.\r\n\r\nReduction in data read can be verified in the UI when built with a recent version of Hadoop say:\r\n```\r\nbuild/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -Phive -DskipTests clean package\r\n```\r\nHowever the default Hadoop 2.2 that is used for unit tests does not report actual bytes read and instead just full file sizes (see FileScanRDD.scala line 80). Therefore I don't think there is a good way to add a unit test for this.\r\n\r\nI tested with the following setup using above build options\r\n```\r\ncase class OrcData(intField: Long, stringField: String)\r\nspark.range(1,1000000).map(i => OrcData(i, s\"part-$i\")).toDF().write.format(\"orc\").save(\"orc_test\")\r\n\r\nsql(\r\n      s\"\"\"CREATE EXTERNAL TABLE orc_test(\r\n         |  intField LONG,\r\n         |  stringField STRING\r\n         |)\r\n         |STORED AS ORC\r\n         |LOCATION '${System.getProperty(\"user.dir\") + \"/orc_test\"}'\r\n       \"\"\".stripMargin)\r\n```\r\n\r\n## Results\r\n\r\nquery | Spark 2.0.2 | this PR\r\n---|---|---\r\n`sql(\"select count(*) from orc_test\").collect`|4.4 MB|199.4 KB\r\n`sql(\"select intField from orc_test\").collect`|743.4 KB|743.4 KB\r\n`sql(\"select * from orc_test\").collect`|4.4 MB|4.4 MB\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/261563074","html_url":"https://github.com/apache/spark/pull/15898#issuecomment-261563074","issue_url":"https://api.github.com/repos/apache/spark/issues/15898","id":261563074,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-11-18T15:43:03Z","updated_at":"2016-11-18T15:43:03Z","body":"**[Test build #68853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68853/consoleFull)** for PR 15898 at commit [`022f32a`](https://github.com/apache/spark/commit/022f32ab92c6fefc0f14393b2bc6c1b89d7b3ca2).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}