{"push_id":937897942,"size":4,"distinct_size":4,"ref":"refs/heads/gen_agg2","head":"6911a05e951d6452995936a50b5949e40595e7a0","before":"640a4b57a687e16ad02c22c4fe787a540ff890a8","commits":[{"sha":"2f7d0b68a29de9755fc9fafd9a52c048981ad880","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12856] [SQL] speed up hashCode of unsafe array\n\nWe iterate the bytes to calculate hashCode before, but now we have `Murmur3_x86_32.hashUnsafeBytes` that don't require the bytes to be word algned, we should use that instead.\n\nA simple benchmark shows it's about 3 X faster, benchmark code: https://gist.github.com/cloud-fan/fa77713ccebf0823b2ab#file-arrayhashbenchmark-scala\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10784 from cloud-fan/array-hashcode.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/2f7d0b68a29de9755fc9fafd9a52c048981ad880"},{"sha":"86972fa52152d2149b88ba75be048a6986006285","author":{"email":"1880ad50d4044a0399fce6619a898bea749151dd@alumni.carnegiemellon.edu","name":"Jeff Lam"},"message":"[SPARK-12722][DOCS] Fixed typo in Pipeline example\n\nhttp://spark.apache.org/docs/latest/ml-guide.html#example-pipeline\n```\nval sameModel = Pipeline.load(\"/tmp/spark-logistic-regression-model\")\n```\nshould be\n```\nval sameModel = PipelineModel.load(\"/tmp/spark-logistic-regression-model\")\n```\ncc: jkbradley\n\nAuthor: Jeff Lam <sha0lin@alumni.carnegiemellon.edu>\n\nCloses #10769 from Agent007/SPARK-12722.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/86972fa52152d2149b88ba75be048a6986006285"},{"sha":"3c0d2365d57fc49ac9bf0d7cc9bd2ef633fb5fb6","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12796] [SQL] Whole stage codegen\n\nThis is the initial work for whole stage codegen, it support Projection/Filter/Range, we will continue work on this to support more physical operators.\n\nA micro benchmark show that a query with range, filter and projection could be 3X faster then before.\n\nIt's turned on by default. For a tree that have at least two chained plans, a WholeStageCodegen will be inserted into it, for example, the following plan\n```\nLimit 10\n+- Project [(id#5L + 1) AS (id + 1)#6L]\n   +- Filter ((id#5L & 1) = 1)\n      +- Range 0, 1, 4, 10, [id#5L]\n```\nwill be translated into\n```\nLimit 10\n+- WholeStageCodegen\n      +- Project [(id#1L + 1) AS (id + 1)#2L]\n         +- Filter ((id#1L & 1) = 1)\n            +- Range 0, 1, 4, 10, [id#1L]\n```\n\nHere is the call graph to generate Java source for A and B (A  support codegen, but B does not):\n\n```\n  *   WholeStageCodegen       Plan A               FakeInput        Plan B\n  * =========================================================================\n  *\n  * -> execute()\n  *     |\n  *  doExecute() -------->   produce()\n  *                             |\n  *                          doProduce()  -------> produce()\n  *                                                   |\n  *                                                doProduce() ---> execute()\n  *                                                   |\n  *                                                consume()\n  *                          doConsume()  ------------|\n  *                             |\n  *  doConsume()  <-----    consume()\n```\n\nA SparkPlan that support codegen need to implement doProduce() and doConsume():\n\n```\ndef doProduce(ctx: CodegenContext): (RDD[InternalRow], String)\ndef doConsume(ctx: CodegenContext, child: SparkPlan, input: Seq[ExprCode]): String\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10735 from davies/whole2.","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/3c0d2365d57fc49ac9bf0d7cc9bd2ef633fb5fb6"},{"sha":"6911a05e951d6452995936a50b5949e40595e7a0","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"Merge branch 'master' of github.com:apache/spark into gen_agg2\n\nConflicts:\n\tsql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegen.scala\n\tsql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala","distinct":true,"url":"https://api.github.com/repos/davies/spark/commits/6911a05e951d6452995936a50b5949e40595e7a0"}]}