{"push_id":1236448110,"size":3,"distinct_size":3,"ref":"refs/heads/catalog-table","head":"d2660124a5e09828f567e70b49badb64c3c2dfa4","before":"9030c5b908db7ecd13dab5cdbf9a6fc5e9dcba62","commits":[{"sha":"faaefab26ffea3a5edfeaff42db222c8cd3ff5f1","author":{"name":"Hiroshi Inoue","email":"3e15e65404b975101e094443b1e5ff2d3461d1d4@jp.ibm.com"},"message":"[SPARK-15726][SQL] Make DatasetBenchmark fairer among Dataset, DataFrame and RDD\n\n## What changes were proposed in this pull request?\n\nDatasetBenchmark compares the performances of RDD, DataFrame and Dataset while running the same operations. However, there are two problems that make the comparisons unfair.\n\n1) In backToBackMap test case, only DataFrame implementation executes less work compared to RDD or Dataset implementations. This test case processes Long+String pairs, but the output from the DataFrame implementation does not include String part while RDD or Dataset generates Long+String pairs as output. This difference significantly changes the performance characteristics due to the String manipulation and creation overheads.\n\n2) In back-to-back map and back-to-back filter test cases, `map` or `filter` operation is executed only once regardless of `numChains` parameter for RDD. Hence the execution times for RDD have been largely underestimated.\n\nOf course, these issues do not affect Spark users, but it may confuse Spark developers.\n\n## How was this patch tested?\nBy executing the DatasetBenchmark\n\nAuthor: Hiroshi Inoue <inouehrs@jp.ibm.com>\n\nCloses #13459 from inouehrs/fix_benchmark_fairness.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/faaefab26ffea3a5edfeaff42db222c8cd3ff5f1"},{"sha":"5effc016c893ce917d535cc1b5026d8e4c846721","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16879][SQL] unify logical plans for CREATE TABLE and CTAS\n\n## What changes were proposed in this pull request?\n\nwe have various logical plans for CREATE TABLE and CTAS: `CreateTableUsing`, `CreateTableUsingAsSelect`, `CreateHiveTableAsSelectLogicalPlan`. This PR unifies them to reduce the complexity and centralize the error handling.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14482 from cloud-fan/table.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/5effc016c893ce917d535cc1b5026d8e4c846721"},{"sha":"d2660124a5e09828f567e70b49badb64c3c2dfa4","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"Merge remote-tracking branch 'origin/master' into catalog-table","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/d2660124a5e09828f567e70b49badb64c3c2dfa4"}]}
{"actor":{"display_login":"cloud-fan"}}