{"push_id":1425608836,"size":1000,"distinct_size":1,"ref":"refs/heads/minor","head":"913054381d0b4f11b5dba494439c0339fc456a00","before":"af6a0bfa581838def69ac44aaa0d5f40b20cc40e","commits":[{"sha":"8a6b7037bb058d00cc767895c3292509576ea2f9","author":{"name":"Andrew Ash","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewash.com"},"message":"Correct example value for spark.ssl.YYY.XXX settings\n\nDocs adjustment to:\n- link to other relevant section of docs\n- correct statement about the only value when actually other values are supported\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #14581 from ash211/patch-10.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/8a6b7037bb058d00cc767895c3292509576ea2f9"},{"sha":"a45fefd17ec4a499b988a2f9931ce397918d3bef","author":{"name":"huangzhaowei","email":"15ea3a7898b05bc595ae00860e248ea58e35620c@gmail.com"},"message":"[SPARK-16941] Use concurrentHashMap instead of scala Map in SparkSQLOperationManager.\n\n## What changes were proposed in this pull request?\nThriftServer will have some thread-safe problem in **SparkSQLOperationManager**.\nAdd a SynchronizedMap trait for the maps in it to avoid this problem.\n\nDetails in [SPARK-16941](https://issues.apache.org/jira/browse/SPARK-16941)\n\n## How was this patch tested?\nNA\n\nAuthor: huangzhaowei <carlmartinmax@gmail.com>\n\nCloses #14534 from SaintBacchus/SPARK-16941.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a45fefd17ec4a499b988a2f9931ce397918d3bef"},{"sha":"7186e8c3180b7f38250cf2f2de791472bf5325a5","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16886][EXAMPLES][DOC] Fix some examples to be consistent and indentation in documentation\n\n## What changes were proposed in this pull request?\n\nOriginally this PR was based on #14491 but I realised that fixing examples are more sensible rather than comments.\n\nThis PR fixes three things below:\n\n - Fix two wrong examples in `structured-streaming-programming-guide.md`. Loading via `read.load(..)` without `as` will be `Dataset<Row>` not `Dataset<String>` in Java.\n\n- Fix indentation across `structured-streaming-programming-guide.md`. Python has 4 spaces and Scala and Java have double spaces. These are inconsistent across the examples.\n\n- Fix `StructuredNetworkWordCountWindowed` and  `StructuredNetworkWordCount` in Java and Scala to initially load `DataFrame` and `Dataset<Row>` to be consistent with the comments and some examples in `structured-streaming-programming-guide.md` and to match Scala and Java to Python one (Python one loads it as `DataFrame` initially).\n\n## How was this patch tested?\n\nN/A\n\nCloses https://github.com/apache/spark/pull/14491\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\nAuthor: Ganesh Chand <ganeshchand@Ganeshs-MacBook-Pro-2.local>\n\nCloses #14564 from HyukjinKwon/SPARK-16886.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7186e8c3180b7f38250cf2f2de791472bf5325a5"},{"sha":"4d496802f592dca96dada73b24afc93c668a7f26","author":{"name":"Michael Gummelt","email":"997b1eeb9667009a1fe95939cfe9ef99c4f83395@mesosphere.io"},"message":"[SPARK-16952] don't lookup spark home directory when executor uri is set\n\n## What changes were proposed in this pull request?\n\nremove requirement to set spark.mesos.executor.home when spark.executor.uri is used\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #14552 from mgummelt/fix-spark-home.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4d496802f592dca96dada73b24afc93c668a7f26"},{"sha":"0f72e4f04b227b9cd5d7ae5958e09b1def49420a","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16958] [SQL] Reuse subqueries within the same query\n\n## What changes were proposed in this pull request?\n\nThere could be multiple subqueries that generate same results, we could re-use the result instead of running it multiple times.\n\nThis PR also cleanup up how we run subqueries.\n\nFor SQL query\n```sql\nselect id,(select avg(id) from t) from t where id > (select avg(id) from t)\n```\nThe explain is\n```\n== Physical Plan ==\n*Project [id#15L, Subquery subquery29 AS scalarsubquery()#35]\n:  +- Subquery subquery29\n:     +- *HashAggregate(keys=[], functions=[avg(id#15L)])\n:        +- Exchange SinglePartition\n:           +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)])\n:              +- *Range (0, 1000, splits=4)\n+- *Filter (cast(id#15L as double) > Subquery subquery29)\n   :  +- Subquery subquery29\n   :     +- *HashAggregate(keys=[], functions=[avg(id#15L)])\n   :        +- Exchange SinglePartition\n   :           +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)])\n   :              +- *Range (0, 1000, splits=4)\n   +- *Range (0, 1000, splits=4)\n```\nThe visualized plan:\n\n![reuse-subquery](https://cloud.githubusercontent.com/assets/40902/17573229/e578d93c-5f0d-11e6-8a3c-0150d81d3aed.png)\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14548 from davies/subq.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/0f72e4f04b227b9cd5d7ae5958e09b1def49420a"},{"sha":"acaf2a81ad5238fd1bc81e7be2c328f40c07e755","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17021][SQL] simplify the constructor parameters of QuantileSummaries\n\n## What changes were proposed in this pull request?\n\n1. `sampled` doesn't need to be `ArrayBuffer`, we never update it, but assign new value\n2. `count` doesn't need to be `var`, we never mutate it.\n3. `headSampled` doesn't need to be in constructor, we never pass a non-empty `headSampled` to constructor\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14603 from cloud-fan/simply.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/acaf2a81ad5238fd1bc81e7be2c328f40c07e755"},{"sha":"cf9367826c38e5f34ae69b409f5d09c55ed1d319","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17018][SQL] literals.sql for testing literal parsing\n\n## What changes were proposed in this pull request?\nThis patch adds literals.sql for testing literal parsing end-to-end in SQL.\n\n## How was this patch tested?\nThe patch itself is only about adding test cases.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14598 from petermaxlee/SPARK-17018-2.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cf9367826c38e5f34ae69b409f5d09c55ed1d319"},{"sha":"1c9a386c6b6812a3931f3fb0004249894a01f657","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-13602][CORE] Add shutdown hook to DriverRunner to prevent driver process leak\n\n## What changes were proposed in this pull request?\n\nAdded shutdown hook to DriverRunner to kill the driver process in case the Worker JVM exits suddenly and the `WorkerWatcher` was unable to properly catch this.  Did some cleanup to consolidate driver state management and setting of finalized vars within the running thread.\n\n## How was this patch tested?\n\nAdded unit tests to verify that final state and exception variables are set accordingly for successfull, failed, and errors in the driver process.  Retrofitted existing test to verify killing of mocked process ends with the correct state and stops properly\n\nManually tested (with deploy-mode=cluster) that the shutdown hook is called by forcibly exiting the `Worker` and various points in the code with the `WorkerWatcher` both disabled and enabled.  Also, manually killed the driver through the ui and verified that the `DriverRunner` interrupted, killed the process and exited properly.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #11746 from BryanCutler/DriverRunner-shutdown-hook-SPARK-13602.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/1c9a386c6b6812a3931f3fb0004249894a01f657"},{"sha":"4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae","author":{"name":"huangzhaowei","email":"15ea3a7898b05bc595ae00860e248ea58e35620c@gmail.com"},"message":"[SPARK-16868][WEB UI] Fix executor be both dead and alive on executor ui.\n\n## What changes were proposed in this pull request?\nIn a heavy pressure of the spark application, since the executor will register it to driver block manager twice(because of heart beats), the executor will show as picture show:\n![image](https://cloud.githubusercontent.com/assets/7404824/17467245/c1359094-5d4e-11e6-843a-f6d6347e1bf6.png)\n\n## How was this patch tested?\nNA\n\nDetails in: [SPARK-16868](https://issues.apache.org/jira/browse/SPARK-16868)\n\nAuthor: huangzhaowei <carlmartinmax@gmail.com>\n\nCloses #14530 from SaintBacchus/SPARK-16868.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae"},{"sha":"ea0bf91b4a2ca3ef472906e50e31fd6268b6f53e","author":{"name":"WangTaoTheTonic","email":"1ad23389494ada22a8de0c1983b95d18ec2d0658@huawei.com"},"message":"[SPARK-17022][YARN] Handle potential deadlock in driver handling messages\n\n## What changes were proposed in this pull request?\n\nWe directly send RequestExecutors to AM instead of transfer it to yarnShedulerBackend first, to avoid potential deadlock.\n\n## How was this patch tested?\n\nmanual tests\n\nAuthor: WangTaoTheTonic <wangtao111@huawei.com>\n\nCloses #14605 from WangTaoTheTonic/lock.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ea0bf91b4a2ca3ef472906e50e31fd6268b6f53e"},{"sha":"7a9e25c38380e6c62080d62ad38a4830e44fe753","author":{"name":"Jeff Zhang","email":"8540ccd5710ce8e6da3f6c1c8189a4ef2fa98c60@apache.org"},"message":"[SPARK-13081][PYSPARK][SPARK_SUBMIT] Allow set pythonExec of driver and executor through conf…\n\nBefore this PR, user have to export environment variable to specify the python of driver & executor which is not so convenient for users. This PR is trying to allow user to specify python through configuration \"--pyspark-driver-python\" & \"--pyspark-executor-python\"\n\nManually test in local & yarn mode for pyspark-shell and pyspark batch mode.\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #13146 from zjffdu/SPARK-13081.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7a9e25c38380e6c62080d62ad38a4830e44fe753"},{"sha":"ac84fb64dd85257da06f93a48fed9bb188140423","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16434][SQL] Avoid per-record type dispatch in JSON when reading\n\n## What changes were proposed in this pull request?\n\nCurrently, `JacksonParser.parse` is doing type-based dispatch for each row to convert the tokens to appropriate values for Spark.\nIt might not have to be done like this because the schema is already kept.\n\nSo, appropriate converters can be created first according to the schema once, and then apply them to each row.\n\nThis PR corrects `JacksonParser` so that it creates all converters for the schema once and then applies them to each row rather than type dispatching for every row.\n\nBenchmark was proceeded with the codes below:\n\n#### Parser tests\n\n**Before**\n\n```scala\ntest(\"Benchmark for JSON converter\") {\n  val N = 500 << 8\n  val row =\n    \"\"\"{\"struct\":{\"field1\": true, \"field2\": 92233720368547758070},\n    \"structWithArrayFields\":{\"field1\":[4, 5, 6], \"field2\":[\"str1\", \"str2\"]},\n    \"arrayOfString\":[\"str1\", \"str2\"],\n    \"arrayOfInteger\":[1, 2147483647, -2147483648],\n    \"arrayOfLong\":[21474836470, 9223372036854775807, -9223372036854775808],\n    \"arrayOfBigInteger\":[922337203685477580700, -922337203685477580800],\n    \"arrayOfDouble\":[1.2, 1.7976931348623157E308, 4.9E-324, 2.2250738585072014E-308],\n    \"arrayOfBoolean\":[true, false, true],\n    \"arrayOfNull\":[null, null, null, null],\n    \"arrayOfStruct\":[{\"field1\": true, \"field2\": \"str1\"}, {\"field1\": false}, {\"field3\": null}],\n    \"arrayOfArray1\":[[1, 2, 3], [\"str1\", \"str2\"]],\n    \"arrayOfArray2\":[[1, 2, 3], [1.1, 2.1, 3.1]]\n   }\"\"\"\n  val data = List.fill(N)(row)\n  val dummyOption = new JSONOptions(Map.empty[String, String])\n  val schema =\n    InferSchema.infer(spark.sparkContext.parallelize(Seq(row)), \"\", dummyOption)\n  val factory = new JsonFactory()\n\n  val benchmark = new Benchmark(\"JSON converter\", N)\n  benchmark.addCase(\"convert JSON file\", 10) { _ =>\n    data.foreach { input =>\n      val parser = factory.createParser(input)\n      parser.nextToken()\n      JacksonParser.convertRootField(factory, parser, schema)\n    }\n  }\n  benchmark.run()\n}\n```\n\n```\nJSON converter:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nconvert JSON file                             1697 / 1807          0.1       13256.9       1.0X\n```\n\n**After**\n\n```scala\ntest(\"Benchmark for JSON converter\") {\n  val N = 500 << 8\n  val row =\n    \"\"\"{\"struct\":{\"field1\": true, \"field2\": 92233720368547758070},\n    \"structWithArrayFields\":{\"field1\":[4, 5, 6], \"field2\":[\"str1\", \"str2\"]},\n    \"arrayOfString\":[\"str1\", \"str2\"],\n    \"arrayOfInteger\":[1, 2147483647, -2147483648],\n    \"arrayOfLong\":[21474836470, 9223372036854775807, -9223372036854775808],\n    \"arrayOfBigInteger\":[922337203685477580700, -922337203685477580800],\n    \"arrayOfDouble\":[1.2, 1.7976931348623157E308, 4.9E-324, 2.2250738585072014E-308],\n    \"arrayOfBoolean\":[true, false, true],\n    \"arrayOfNull\":[null, null, null, null],\n    \"arrayOfStruct\":[{\"field1\": true, \"field2\": \"str1\"}, {\"field1\": false}, {\"field3\": null}],\n    \"arrayOfArray1\":[[1, 2, 3], [\"str1\", \"str2\"]],\n    \"arrayOfArray2\":[[1, 2, 3], [1.1, 2.1, 3.1]]\n   }\"\"\"\n  val data = List.fill(N)(row)\n  val dummyOption = new JSONOptions(Map.empty[String, String], new SQLConf())\n  val schema =\n    InferSchema.infer(spark.sparkContext.parallelize(Seq(row)), dummyOption)\n\n  val benchmark = new Benchmark(\"JSON converter\", N)\n  benchmark.addCase(\"convert JSON file\", 10) { _ =>\n    val parser = new JacksonParser(schema, dummyOption)\n    data.foreach { input =>\n      parser.parse(input)\n    }\n  }\n  benchmark.run()\n}\n```\n\n```\nJSON converter:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nconvert JSON file                             1401 / 1461          0.1       10947.4       1.0X\n```\n\nIt seems parsing time is improved by roughly ~20%\n\n#### End-to-End test\n\n```scala\ntest(\"Benchmark for JSON reader\") {\n  val N = 500 << 8\n  val row =\n    \"\"\"{\"struct\":{\"field1\": true, \"field2\": 92233720368547758070},\n    \"structWithArrayFields\":{\"field1\":[4, 5, 6], \"field2\":[\"str1\", \"str2\"]},\n    \"arrayOfString\":[\"str1\", \"str2\"],\n    \"arrayOfInteger\":[1, 2147483647, -2147483648],\n    \"arrayOfLong\":[21474836470, 9223372036854775807, -9223372036854775808],\n    \"arrayOfBigInteger\":[922337203685477580700, -922337203685477580800],\n    \"arrayOfDouble\":[1.2, 1.7976931348623157E308, 4.9E-324, 2.2250738585072014E-308],\n    \"arrayOfBoolean\":[true, false, true],\n    \"arrayOfNull\":[null, null, null, null],\n    \"arrayOfStruct\":[{\"field1\": true, \"field2\": \"str1\"}, {\"field1\": false}, {\"field3\": null}],\n    \"arrayOfArray1\":[[1, 2, 3], [\"str1\", \"str2\"]],\n    \"arrayOfArray2\":[[1, 2, 3], [1.1, 2.1, 3.1]]\n   }\"\"\"\n  val df = spark.sqlContext.read.json(spark.sparkContext.parallelize(List.fill(N)(row)))\n  withTempPath { path =>\n    df.write.format(\"json\").save(path.getCanonicalPath)\n\n    val benchmark = new Benchmark(\"JSON reader\", N)\n    benchmark.addCase(\"reading JSON file\", 10) { _ =>\n      spark.read.format(\"json\").load(path.getCanonicalPath).collect()\n    }\n    benchmark.run()\n  }\n}\n```\n\n**Before**\n\n```\nJSON reader:                             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nreading JSON file                             6485 / 6924          0.0       50665.0       1.0X\n```\n\n**After**\n\n```\nJSON reader:                             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nreading JSON file                             6350 / 6529          0.0       49609.3       1.0X\n```\n\n## How was this patch tested?\n\nExisting test cases should cover this.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14102 from HyukjinKwon/SPARK-16434.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ac84fb64dd85257da06f93a48fed9bb188140423"},{"sha":"ccc6dc0f4b62837c73fca0e3c8b9c14be798b062","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[MINOR][ML] Rename TreeEnsembleModels to TreeEnsembleModel for PySpark\n\n## What changes were proposed in this pull request?\nFix the typo of ```TreeEnsembleModels``` for PySpark, it should ```TreeEnsembleModel``` which will be consistent with Scala. What's more, it represents a tree ensemble model, so  ```TreeEnsembleModel``` should be more reasonable. This should not be used public, so it will not involve  breaking change.\n\n## How was this patch tested?\nNo new tests, should pass existing ones.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14454 from yanboliang/TreeEnsembleModel.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ccc6dc0f4b62837c73fca0e3c8b9c14be798b062"},{"sha":"abff92bfdc7d4c9d2308794f0350561fe0ceb4dd","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16975][SQL] Column-partition path starting '_' should be handled correctly\n\n## What changes were proposed in this pull request?\n\nCurrently, Spark ignores path names starting with underscore `_` and `.`. This causes read-failures for the column-partitioned file data sources whose partition column names starts from '_', e.g. `_col`.\n\n**Before**\n```scala\nscala> spark.range(10).withColumn(\"_locality_code\", $\"id\").write.partitionBy(\"_locality_code\").save(\"/tmp/parquet\")\nscala> spark.read.parquet(\"/tmp/parquet\")\norg.apache.spark.sql.AnalysisException: Unable to infer schema for ParquetFormat at /tmp/parquet20. It must be specified manually;\n```\n\n**After**\n```scala\nscala> spark.range(10).withColumn(\"_locality_code\", $\"id\").write.partitionBy(\"_locality_code\").save(\"/tmp/parquet\")\nscala> spark.read.parquet(\"/tmp/parquet\")\nres2: org.apache.spark.sql.DataFrame = [id: bigint, _locality_code: int]\n```\n\n## How was this patch tested?\n\nPass the Jenkins with a new test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14585 from dongjoon-hyun/SPARK-16975-PARQUET.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/abff92bfdc7d4c9d2308794f0350561fe0ceb4dd"},{"sha":"00e103a6edd1a1f001a94d41dd1f7acc40a1e30f","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17013][SQL] Parse negative numeric literals\n\n## What changes were proposed in this pull request?\nThis patch updates the SQL parser to parse negative numeric literals as numeric literals, instead of unary minus of positive literals.\n\nThis allows the parser to parse the minimal value for each data type, e.g. \"-32768S\".\n\n## How was this patch tested?\nUpdated test cases.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14608 from petermaxlee/SPARK-17013.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/00e103a6edd1a1f001a94d41dd1f7acc40a1e30f"},{"sha":"993923c8f5ca719daf905285738b7fdcaf944d8c","author":{"name":"hongshen","email":"0ddf15eae228419f727fc3ecdc8f1b3f1a502622@126.com"},"message":"[SPARK-16985] Change dataFormat from yyyyMMddHHmm to yyyyMMddHHmmss\n\n## What changes were proposed in this pull request?\n\nIn our cluster, sometimes the sql output maybe overrided. When I submit some sql, all insert into the same table, and the sql will cost less one minute, here is the detail,\n1 sql1, 11:03 insert into table.\n2 sql2, 11:04:11 insert into table.\n3 sql3, 11:04:48 insert into table.\n4 sql4, 11:05 insert into table.\n5 sql5, 11:06 insert into table.\nThe sql3's output file will override the sql2's output file. here is the log:\n```\n16/05/04 11:04:11 INFO hive.SparkHiveHadoopWriter: XXfinalPath=hdfs://tl-sng-gdt-nn-tdw.tencent-distribute.com:54310/tmp/assorz/tdw-tdwadmin/20160504/04559505496526517_-1_1204544348/10000/_tmp.p_20160428/attempt_201605041104_0001_m_000000_1\n\n16/05/04 11:04:48 INFO hive.SparkHiveHadoopWriter: XXfinalPath=hdfs://tl-sng-gdt-nn-tdw.tencent-distribute.com:54310/tmp/assorz/tdw-tdwadmin/20160504/04559505496526517_-1_212180468/10000/_tmp.p_20160428/attempt_201605041104_0001_m_000000_1\n\n```\n\nThe reason is the output file use SimpleDateFormat(\"yyyyMMddHHmm\"), if two sql insert into the same table in the same minute, the output will be overrite. I think we should change dateFormat to \"yyyyMMddHHmmss\", in our cluster, we can't finished a sql in one second.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: hongshen <shenh062326@126.com>\n\nCloses #14574 from shenh062326/SPARK-16985.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/993923c8f5ca719daf905285738b7fdcaf944d8c"},{"sha":"f4482225c405b9cfe078deac74e4c28e2dcc97c3","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][DOC] Fix style in examples across documentation\n\n## What changes were proposed in this pull request?\n\nThis PR fixes the documentation as below:\n\n  -  Python has 4 spaces and Java and Scala has 2 spaces (See https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide).\n\n  - Avoid excessive parentheses and curly braces for anonymous functions. (See https://github.com/databricks/scala-style-guide#anonymous)\n\n## How was this patch tested?\n\nN/A\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14593 from HyukjinKwon/minor-documentation.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/f4482225c405b9cfe078deac74e4c28e2dcc97c3"},{"sha":"79e2caa1328843457841d71642b60be919ebb1e0","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16598][SQL][TEST] Added a test case for verifying the table identifier parsing\n\n#### What changes were proposed in this pull request?\nSo far, the test cases of `TableIdentifierParserSuite` do not cover the quoted cases. We should add one for avoiding regression.\n\n#### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14244 from gatorsmile/quotedIdentifiers.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/79e2caa1328843457841d71642b60be919ebb1e0"},{"sha":"bbae20ade14e50541e4403ca7b45bf6c11695d15","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17033][ML][MLLIB] GaussianMixture should use treeAggregate to improve performance\n\n## What changes were proposed in this pull request?\n```GaussianMixture``` should use ```treeAggregate``` rather than ```aggregate``` to improve performance and scalability. In my test of dataset with 200 features and 1M instance, I found there is 20% increased performance.\nBTW, we should destroy broadcast variable ```compute``` at the end of each iteration.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14621 from yanboliang/spark-17033.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/bbae20ade14e50541e4403ca7b45bf6c11695d15"},{"sha":"2a105134e9a3efd46b761fab5e563ddebb26575d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16771][SQL] WITH clause should not fall into infinite loop.\n\n## What changes were proposed in this pull request?\n\nThis PR changes the CTE resolving rule to use only **forward-declared** tables in order to prevent infinite loops. More specifically, new logic is like the following.\n\n* Resolve CTEs in `WITH` clauses first before replacing the main SQL body.\n* When resolving CTEs, only forward-declared CTEs or base tables are referenced.\n  - Self-referencing is not allowed any more.\n  - Cross-referencing is not allowed any more.\n\n**Reported Error Scenarios**\n```scala\nscala> sql(\"WITH t AS (SELECT 1 FROM t) SELECT * FROM t\")\njava.lang.StackOverflowError\n...\nscala> sql(\"WITH t1 AS (SELECT * FROM t2), t2 AS (SELECT 2 FROM t1) SELECT * FROM t1, t2\")\njava.lang.StackOverflowError\n...\n```\nNote that `t`, `t1`, and `t2` are not declared in database. Spark falls into infinite loops before resolving table names.\n\n## How was this patch tested?\n\nPass the Jenkins tests with new two testcases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14397 from dongjoon-hyun/SPARK-16771-TREENODE.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2a105134e9a3efd46b761fab5e563ddebb26575d"}]}
{"actor":{"display_login":"cloud-fan"}}