{"push_id":1062559498,"size":1000,"distinct_size":1,"ref":"refs/heads/closure","head":"b78b2cef91a07784e2c92bb266585af4e2df9896","before":"39b103529a848ba48d970feea5859d96c3692c81","commits":[{"sha":"b9dfdcc63bb12bc24de96060e756889c2ceda519","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-13031] [SQL] cleanup codegen and improve test coverage\"\n\nThis reverts commit cc18a7199240bf3b03410c1ba6704fe7ce6ae38e.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b9dfdcc63bb12bc24de96060e756889c2ceda519"},{"sha":"66449b8dcdbc3dca126c34b42c4d0419c7648696","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12968][SQL] Implement command to set current database\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12968\n\nImplement command to set current database.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10916 from viirya/ddl-use-database.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/66449b8dcdbc3dca126c34b42c4d0419c7648696"},{"sha":"721ced28b522cc00b45ca7fa32a99e80ad3de2f7","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13067] [SQL] workaround for a weird scala reflection problem\n\nA simple workaround to avoid getting parameter types when convert a\nlogical plan to json.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10970 from cloud-fan/reflection.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/721ced28b522cc00b45ca7fa32a99e80ad3de2f7"},{"sha":"8d3cc3de7d116190911e7943ef3233fe3b7db1bf","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-13050][BUILD] Scalatest tags fail build with the addition of the sketch module\n\nA dependency on the spark test tags was left out of the sketch module pom file causing builds to fail when test tags were used. This dependency is found in the pom file for every other module in spark.\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #10954 from ajbozarth/spark13050.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/8d3cc3de7d116190911e7943ef3233fe3b7db1bf"},{"sha":"55561e7693dd2a5bf3c7f8026c725421801fd0ec","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13031][SQL] cleanup codegen and improve test coverage\n\n1. enable whole stage codegen during tests even there is only one operator supports that.\n2. split doProduce() into two APIs: upstream() and doProduce()\n3. generate prefix for fresh names of each operator\n4. pass UnsafeRow to parent directly (avoid getters and create UnsafeRow again)\n5. fix bugs and tests.\n\nThis PR re-open #10944 and fix the bug.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10977 from davies/gen_refactor.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/55561e7693dd2a5bf3c7f8026c725421801fd0ec"},{"sha":"e51b6eaa9e9c007e194d858195291b2b9fb27322","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-13032][ML][PYSPARK] PySpark support model export/import and take LinearRegression as example\n\n* Implement ```MLWriter/MLWritable/MLReader/MLReadable``` for PySpark.\n* Making ```LinearRegression``` to support ```save/load``` as example. After this merged, the work for other transformers/estimators will be easy, then we can list and distribute the tasks to the community.\n\ncc mengxr jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #10469 from yanboliang/spark-11939.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e51b6eaa9e9c007e194d858195291b2b9fb27322"},{"sha":"e4c1162b6b3dbc8fc95cfe75c6e0bc2915575fb2","author":{"email":"02e10e341a2fea4868ba58fdfcfe400dcd4d1dd0@yahoo-inc.com","name":"zhuol"},"message":"[SPARK-10873] Support column sort and search for History Server.\n\n[SPARK-10873] Support column sort and search for History Server using jQuery DataTable and REST API. Before this commit, the history server was generated hard-coded html and can not support search, also, the sorting was disabled if there is any application that has more than one attempt. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve user experience.\n\n1. Create the historypage-template.html for displaying application information in datables.\n2. historypage.js uses jQuery to access the data from /api/v1/applications REST API, and use DataTable to display each application's information. For application that has more than one attempt, the RowsGroup is used to merge such entries while at the same time supporting sort and search.\n3. \"duration\" and \"lastUpdated\" rest API are added to application's \"attempts\".\n4. External javascirpt and css files for datatables, RowsGroup and jquery plugins are added with licenses clarified.\n\nSnapshots for how it looks like now:\n\nHistory page view:\n![historypage](https://cloud.githubusercontent.com/assets/11683054/12184383/89bad774-b55a-11e5-84e4-b0276172976f.png)\n\nSearch:\n![search](https://cloud.githubusercontent.com/assets/11683054/12184385/8d3b94b0-b55a-11e5-869a-cc0ef0a4242a.png)\n\nSort by started time:\n![sort-by-started-time](https://cloud.githubusercontent.com/assets/11683054/12184387/8f757c3c-b55a-11e5-98c8-577936366566.png)\n\nAuthor: zhuol <zhuol@yahoo-inc.com>\n\nCloses #10648 from zhuoliu/10873.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e4c1162b6b3dbc8fc95cfe75c6e0bc2915575fb2"},{"sha":"c5f745ede01831b59c57effa7de88c648b82c13d","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13072] [SQL] simplify and improve murmur3 hash expression codegen\n\nsimplify(remove several unnecessary local variables) the generated code of hash expression, and avoid null check if possible.\n\ngenerated code comparison for `hash(int, double, string, array<string>)`:\n**before:**\n```\n  public UnsafeRow apply(InternalRow i) {\n    /* hash(input[0, int],input[1, double],input[2, string],input[3, array<int>],42) */\n    int value1 = 42;\n    /* input[0, int] */\n    int value3 = i.getInt(0);\n    if (!false) {\n      value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(value3, value1);\n    }\n    /* input[1, double] */\n    double value5 = i.getDouble(1);\n    if (!false) {\n      value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(Double.doubleToLongBits(value5), value1);\n    }\n    /* input[2, string] */\n    boolean isNull6 = i.isNullAt(2);\n    UTF8String value7 = isNull6 ? null : (i.getUTF8String(2));\n    if (!isNull6) {\n      value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value7.getBaseObject(), value7.getBaseOffset(), value7.numBytes(), value1);\n    }\n    /* input[3, array<int>] */\n    boolean isNull8 = i.isNullAt(3);\n    ArrayData value9 = isNull8 ? null : (i.getArray(3));\n    if (!isNull8) {\n      int result10 = value1;\n      for (int index11 = 0; index11 < value9.numElements(); index11++) {\n        if (!value9.isNullAt(index11)) {\n          final int element12 = value9.getInt(index11);\n          result10 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element12, result10);\n        }\n      }\n      value1 = result10;\n    }\n  }\n```\n**after:**\n```\n  public UnsafeRow apply(InternalRow i) {\n    /* hash(input[0, int],input[1, double],input[2, string],input[3, array<int>],42) */\n    int value1 = 42;\n    /* input[0, int] */\n    int value3 = i.getInt(0);\n    value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(value3, value1);\n    /* input[1, double] */\n    double value5 = i.getDouble(1);\n    value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(Double.doubleToLongBits(value5), value1);\n    /* input[2, string] */\n    boolean isNull6 = i.isNullAt(2);\n    UTF8String value7 = isNull6 ? null : (i.getUTF8String(2));\n\n    if (!isNull6) {\n      value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value7.getBaseObject(), value7.getBaseOffset(), value7.numBytes(), value1);\n    }\n\n    /* input[3, array<int>] */\n    boolean isNull8 = i.isNullAt(3);\n    ArrayData value9 = isNull8 ? null : (i.getArray(3));\n    if (!isNull8) {\n      for (int index10 = 0; index10 < value9.numElements(); index10++) {\n        final int element11 = value9.getInt(index10);\n        value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element11, value1);\n      }\n    }\n\n    rowWriter14.write(0, value1);\n    return result12;\n  }\n```\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10974 from cloud-fan/codegen.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/c5f745ede01831b59c57effa7de88c648b82c13d"},{"sha":"5f686cc8b74ea9e36f56c31f14df90d134fd9343","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12656] [SQL] Implement Intersect with Left-semi Join\n\nOur current Intersect physical operator simply delegates to RDD.intersect. We should remove the Intersect physical operator and simply transform a logical intersect into a semi-join with distinct. This way, we can take advantage of all the benefits of join implementations (e.g. managed memory, code generation, broadcast joins).\n\nAfter a search, I found one of the mainstream RDBMS did the same. In their query explain, Intersect is replaced by Left-semi Join. Left-semi Join could help outer-join elimination in Optimizer, as shown in the PR: https://github.com/apache/spark/pull/10566\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\nAuthor: xiaoli <lixiao1983@gmail.com>\nAuthor: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>\n\nCloses #10630 from gatorsmile/IntersectBySemiJoin.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/5f686cc8b74ea9e36f56c31f14df90d134fd9343"},{"sha":"2b027e9a386fe4009f61ad03b169335af5a9a5c6","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12818] Polishes spark-sketch module\n\nFixes various minor code and Javadoc styling issues.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10985 from liancheng/sketch-polishing.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2b027e9a386fe4009f61ad03b169335af5a9a5c6"},{"sha":"e38b0baa38c6894335f187eaa4c8ea5c02d4563b","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13055] SQLHistoryListener throws ClassCastException\n\nThis is an existing issue uncovered recently by #10835. The reason for the exception was because the `SQLHistoryListener` gets all sorts of accumulators, not just the ones that represent SQL metrics. For example, the listener gets the `internal.metrics.shuffleRead.remoteBlocksFetched`, which is an Int, then it proceeds to cast the Int to a Long, which fails.\n\nThe fix is to mark accumulators representing SQL metrics using some internal metadata. Then we can identify which ones are SQL metrics and only process those in the `SQLHistoryListener`.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10971 from andrewor14/fix-sql-history.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e38b0baa38c6894335f187eaa4c8ea5c02d4563b"},{"sha":"2cbc412821641cf9446c0621ffa1976bd7fc4fa1","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13076][SQL] Rename ClientInterface -> HiveClient\n\nAnd ClientWrapper -> HiveClientImpl.\n\nI have some followup pull requests to introduce a new internal catalog, and I think this new naming reflects better the functionality of the two classes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10981 from rxin/SPARK-13076.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2cbc412821641cf9446c0621ffa1976bd7fc4fa1"},{"sha":"e6ceac49a311faf3413acda57a6612fe806adf90","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13096][TEST] Fix flaky verifyPeakExecutionMemorySet\n\nPreviously we would assert things before all events are guaranteed to have been processed. To fix this, just block until all events are actually processed, i.e. until the listener queue is empty.\n\nhttps://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/79/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling/\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10990 from andrewor14/accum-suite-less-flaky.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e6ceac49a311faf3413acda57a6612fe806adf90"},{"sha":"70e69fc4dd619654f5d24b8b84f6a94f7705c59b","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13088] Fix DAG viz in latest version of chrome\n\nApparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit/7d6c0002e4c74b82a02c5917876576f71e215590, which is taken from the fix in the main repo: https://github.com/cpettitt/dagre-d3/commit/1ef067f1c6ad2e0980f6f0ca471bce998784b7b2\n\nUpstream issue: https://github.com/cpettitt/dagre-d3/issues/202\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10986 from andrewor14/fix-dag-viz.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/70e69fc4dd619654f5d24b8b84f6a94f7705c59b"},{"sha":"12252d1da90fa7d2dffa3a7c249ecc8821dee130","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13071] Coalescing HadoopRDD overwrites existing input metrics\n\nThis issue is causing tests to fail consistently in master with Hadoop 2.6 / 2.7. This is because for Hadoop 2.5+ we overwrite existing values of `InputMetrics#bytesRead` in each call to `HadoopRDD#compute`. In the case of coalesce, e.g.\n```\nsc.textFile(..., 4).coalesce(2).count()\n```\nwe will call `compute` multiple times in the same task, overwriting `bytesRead` values from previous calls to `compute`.\n\nFor a regression test, see `InputOutputMetricsSuite.input metrics for old hadoop with coalesce`. I did not add a new regression test because it's impossible without significant refactoring; there's a lot of existing duplicate code in this corner of Spark.\n\nThis was caused by #10835.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10973 from andrewor14/fix-input-metrics-coalesce.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/12252d1da90fa7d2dffa3a7c249ecc8821dee130"},{"sha":"e6a02c66d53f59ba2d5c1548494ae80a385f9f5c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12914] [SQL] generate aggregation with grouping keys\n\nThis PR add support for grouping keys for generated TungstenAggregate.\n\nSpilling and performance improvements for BytesToBytesMap will be done by followup PR.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10855 from davies/gen_keys.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e6a02c66d53f59ba2d5c1548494ae80a385f9f5c"},{"sha":"dab246f7e4664d36073ec49d9df8a11c5e998cdb","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13098] [SQL] remove GenericInternalRowWithSchema\n\nThis class is only used for serialization of Python DataFrame. However, we don't require internal row there, so `GenericRowWithSchema` can also do the job.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10992 from cloud-fan/python.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/dab246f7e4664d36073ec49d9df8a11c5e998cdb"},{"sha":"289373b28cd2546165187de2e6a9185a1257b1e7","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-6363][BUILD] Make Scala 2.11 the default Scala version\n\nThis patch changes Spark's build to make Scala 2.11 the default Scala version. To be clear, this does not mean that Spark will stop supporting Scala 2.10: users will still be able to compile Spark for Scala 2.10 by following the instructions on the \"Building Spark\" page; however, it does mean that Scala 2.11 will be the default Scala version used by our CI builds (including pull request builds).\n\nThe Scala 2.11 compiler is faster than 2.10, so I think we'll be able to look forward to a slight speedup in our CI builds (it looks like it's about 2X faster for the Maven compile-only builds, for instance).\n\nAfter this patch is merged, I'll update Jenkins to add new compile-only jobs to ensure that Scala 2.10 compilation doesn't break.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10608 from JoshRosen/SPARK-6363.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/289373b28cd2546165187de2e6a9185a1257b1e7"},{"sha":"de283719980ae78b740e507e4d70c7ebbf6c5f74","author":{"email":"e28ad19e4e56395ff72e0397107366a06912a175@haizhi.com","name":"wangyang"},"message":"[SPARK-13100][SQL] improving the performance of stringToDate method in DateTimeUtils.scala\n\n In jdk1.7 TimeZone.getTimeZone() is synchronized, so use an instance variable to hold an GMT TimeZone object instead of instantiate it every time.\n\nAuthor: wangyang <wangyang@haizhi.com>\n\nCloses #10994 from wangyang1992/datetimeUtil.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/de283719980ae78b740e507e4d70c7ebbf6c5f74"},{"sha":"a1303de0a0e9d0c80327977abf52a79e2aa95e1f","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-13070][SQL] Better error message when Parquet schema merging fails\n\nMake sure we throw better error messages when Parquet schema merging fails.\n\nAuthor: Cheng Lian <lian@databricks.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10979 from viirya/schema-merging-failure-message.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a1303de0a0e9d0c80327977abf52a79e2aa95e1f"}]}