{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10735","labels_url":"https://api.github.com/repos/apache/spark/issues/10735/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10735/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10735/events","html_url":"https://github.com/apache/spark/pull/10735","id":126353382,"number":10735,"title":"[SPARK-12796] [SQL] Whole stage codegen","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2016-01-13T06:41:00Z","updated_at":"2016-01-15T08:18:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10735","html_url":"https://github.com/apache/spark/pull/10735","diff_url":"https://github.com/apache/spark/pull/10735.diff","patch_url":"https://github.com/apache/spark/pull/10735.patch"},"body":"This is the initial work for whole stage codegen, it support Projection/Filter/Range, we will continue work on this to support more physical operators. \r\n\r\nA micro benchmark show that a query with range, filter and projection could be 3X faster then before. \r\n\r\nIt's turned on by default. For a tree that have at least two chained plans, a WholeStageCodegen will be inserted into it, for example, the following plan\r\n```\r\nLimit 10\r\n+- Project [(id#5L + 1) AS (id + 1)#6L]\r\n   +- Filter ((id#5L & 1) = 1)\r\n      +- Range 0, 1, 4, 10, [id#5L]\r\n```\r\nwill be translated into\r\n```\r\nLimit 10\r\n+- WholeStageCodegen\r\n      +- Project [(id#1L + 1) AS (id + 1)#2L]\r\n         +- Filter ((id#1L & 1) = 1)\r\n            +- Range 0, 1, 4, 10, [id#1L]\r\n``` \r\n\r\nHere is the call graph to generate Java source for A and B (A  support codegen, but B does not):\r\n\r\n```\r\n  *   WholeStageCodegen       Plan A               FakeInput        Plan B\r\n  * =========================================================================\r\n  *\r\n  * -> execute()\r\n  *     |\r\n  *  doExecute() -------->   produce()\r\n  *                             |\r\n  *                          doProduce()  -------> produce()\r\n  *                                                   |\r\n  *                                                doProduce() ---> execute()\r\n  *                                                   |\r\n  *                                                consume()\r\n  *                          doConsume()  ------------|\r\n  *                             |\r\n  *  doConsume()  <-----    consume()\r\n``` \r\n\r\nA SparkPlan that support codegen need to implement doProduce() and doConsume():\r\n\r\n```\r\ndef doProduce(ctx: CodegenContext): (RDD[InternalRow], String)\r\ndef doConsume(ctx: CodegenContext, child: SparkPlan, input: Seq[ExprCode]): String\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/171899058","html_url":"https://github.com/apache/spark/pull/10735#issuecomment-171899058","issue_url":"https://api.github.com/repos/apache/spark/issues/10735","id":171899058,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-01-15T08:18:26Z","updated_at":"2016-01-15T08:18:26Z","body":"**[Test build #49451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49451/consoleFull)** for PR 10735 at commit [`c1dd60a`](https://github.com/apache/spark/commit/c1dd60a21e1a016b5479cf89cdef035ac53167f0)."}}