{"push_id":1134121004,"size":3,"distinct_size":3,"ref":"refs/heads/branch-2.0","head":"f3570bcea697704f6f10fa62109300ce3cf6b28b","before":"36045106d43b3952c55bae4439dbc86892399b3c","commits":[{"sha":"2c1b6b58d161dadfab1208e05d4ef549cc2e735c","author":{"name":"Yadong Qi","email":"a4dd261dfdc77f6e3f21308e2b46fcb2783647d6@gmail.com"},"message":"[SPARK-15549][SQL] Disable bucketing when the output doesn't contain all bucketing columns\n\n## What changes were proposed in this pull request?\nI create a bucketed table bucketed_table with bucket column i,\n```scala\ncase class Data(i: Int, j: Int, k: Int)\nsc.makeRDD(Array((1, 2, 3))).map(x => Data(x._1, x._2, x._3)).toDF.write.bucketBy(2, \"i\").saveAsTable(\"bucketed_table\")\n```\n\nand I run the following SQLs:\n```sql\nSELECT j FROM bucketed_table;\nError in query: bucket column i not found in existing columns (j);\n\nSELECT j, MAX(k) FROM bucketed_table GROUP BY j;\nError in query: bucket column i not found in existing columns (j, k);\n```\n\nI think we should add a check that, we only enable bucketing when it satisfies all conditions below:\n1. the conf is enabled\n2. the relation is bucketed\n3. the output contains all bucketing columns\n\n## How was this patch tested?\nUpdated test cases to reflect the changes.\n\nAuthor: Yadong Qi <qiyadong2010@gmail.com>\n\nCloses #13321 from watermen/SPARK-15549.\n\n(cherry picked from commit b4c32c4952f7af2733258aa4e27f21e8832c8a3a)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/2c1b6b58d161dadfab1208e05d4ef549cc2e735c"},{"sha":"a2f68ded22d5d8727311fb039714400cbf48156a","author":{"name":"felixcheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15637][SPARKR] fix R tests on R 3.2.2\n\n## What changes were proposed in this pull request?\n\nChange version check in R tests\n\n## How was this patch tested?\n\nR tests\nshivaram\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #13369 from felixcheung/rversioncheck.\n\n(cherry picked from commit 74c1b79f3f82751d166bccba877501a8cabc9b7c)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/a2f68ded22d5d8727311fb039714400cbf48156a"},{"sha":"f3570bcea697704f6f10fa62109300ce3cf6b28b","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15636][SQL] Make aggregate expressions more concise in explain\n\n## What changes were proposed in this pull request?\nThis patch reduces the verbosity of aggregate expressions in explain (but does not actually remove any information). As an example, for the following command:\n```\nspark.range(10).selectExpr(\"sum(id) + 1\", \"count(distinct id)\").explain(true)\n```\n\nOutput before this patch:\n```\n== Physical Plan ==\n*TungstenAggregate(key=[], functions=[(sum(id#0L),mode=Final,isDistinct=false),(count(id#0L),mode=Final,isDistinct=true)], output=[(sum(id) + 1)#3L,count(DISTINCT id)#16L])\n+- Exchange SinglePartition, None\n   +- *TungstenAggregate(key=[], functions=[(sum(id#0L),mode=PartialMerge,isDistinct=false),(count(id#0L),mode=Partial,isDistinct=true)], output=[sum#18L,count#21L])\n      +- *TungstenAggregate(key=[id#0L], functions=[(sum(id#0L),mode=PartialMerge,isDistinct=false)], output=[id#0L,sum#18L])\n         +- Exchange hashpartitioning(id#0L, 5), None\n            +- *TungstenAggregate(key=[id#0L], functions=[(sum(id#0L),mode=Partial,isDistinct=false)], output=[id#0L,sum#18L])\n               +- *Range (0, 10, splits=2)\n```\n\nOutput after this patch:\n```\n== Physical Plan ==\n*TungstenAggregate(key=[], functions=[sum(id#0L),count(distinct id#0L)], output=[(sum(id) + 1)#3L,count(DISTINCT id)#16L])\n+- Exchange SinglePartition, None\n   +- *TungstenAggregate(key=[], functions=[merge_sum(id#0L),partial_count(distinct id#0L)], output=[sum#18L,count#21L])\n      +- *TungstenAggregate(key=[id#0L], functions=[merge_sum(id#0L)], output=[id#0L,sum#18L])\n         +- Exchange hashpartitioning(id#0L, 5), None\n            +- *TungstenAggregate(key=[id#0L], functions=[partial_sum(id#0L)], output=[id#0L,sum#18L])\n               +- *Range (0, 10, splits=2)\n```\n\nNote the change from `(sum(id#0L),mode=PartialMerge,isDistinct=false)` to `merge_sum(id#0L)`.\n\nIn general aggregate explain is still very verbose, but further work will be done as follow-up pull requests.\n\n## How was this patch tested?\nTested manually.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13367 from rxin/SPARK-15636.\n\n(cherry picked from commit 472f16181d199684996a156b0e429bc525d65a57)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/f3570bcea697704f6f10fa62109300ce3cf6b28b"}]}
{"actor":{"display_login":"jfota"}}