{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10735","labels_url":"https://api.github.com/repos/apache/spark/issues/10735/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10735/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10735/events","html_url":"https://github.com/apache/spark/pull/10735","id":126353382,"number":10735,"title":"[SPARK-12796] [SQL] Whole stage codegen","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-13T06:41:00Z","updated_at":"2016-01-13T23:23:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10735","html_url":"https://github.com/apache/spark/pull/10735","diff_url":"https://github.com/apache/spark/pull/10735.diff","patch_url":"https://github.com/apache/spark/pull/10735.patch"},"body":"This is the initial work for whole stage codegen, it support Projection/Filter/Range, we will continue work on this to support more physical operators. \r\n\r\nA micro benchmark show that a query with range, filter and projection could be 3X faster then before. \r\n\r\nIt's turned on by default, automatically fallback to old behavior if the compile stage failed to compile (could because the generated Java source code is too large, over 64k Loc).\r\n\r\n  Here is the call graph of three SparkPlan, A and B support codegen, but C does not.\r\n```\r\n    *\r\n    *   SparkPlan A            SparkPlan B          SparkPlan C\r\n    * ===============================================================\r\n    * -> doExecute()\r\n    *     |\r\n    *  doCodeGen()\r\n    *     |\r\n    *  produce()\r\n    *     |\r\n    *  doProduce() -------->   produce()\r\n    *                             |\r\n    *                          doProduce()  -------> produce()\r\n    *                                                   |\r\n    *                                                doProduce() (fetch row from upstream)\r\n    *                                                   |\r\n    *                                                consume()\r\n    *                          doConsume()  ------------|\r\n    *                             |\r\n    *  doConsume()  <-----    consume()\r\n    *     |\r\n    *   consume() (omit the rows)\r\n    *\r\n    * SparkPlan A and B should override doProduce() and doConsume().\r\n``` "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/171469846","html_url":"https://github.com/apache/spark/pull/10735#issuecomment-171469846","issue_url":"https://api.github.com/repos/apache/spark/issues/10735","id":171469846,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-01-13T23:23:10Z","updated_at":"2016-01-13T23:23:10Z","body":"Merged build finished. Test FAILed."}}