{"push_id":1249954731,"size":15,"distinct_size":15,"ref":"refs/heads/master","head":"ddf0d1e3fe18bcd01e1447feea1b76ce86087b3b","before":"00e103a6edd1a1f001a94d41dd1f7acc40a1e30f","commits":[{"sha":"993923c8f5ca719daf905285738b7fdcaf944d8c","author":{"name":"hongshen","email":"0ddf15eae228419f727fc3ecdc8f1b3f1a502622@126.com"},"message":"[SPARK-16985] Change dataFormat from yyyyMMddHHmm to yyyyMMddHHmmss\n\n## What changes were proposed in this pull request?\n\nIn our cluster, sometimes the sql output maybe overrided. When I submit some sql, all insert into the same table, and the sql will cost less one minute, here is the detail,\n1 sql1, 11:03 insert into table.\n2 sql2, 11:04:11 insert into table.\n3 sql3, 11:04:48 insert into table.\n4 sql4, 11:05 insert into table.\n5 sql5, 11:06 insert into table.\nThe sql3's output file will override the sql2's output file. here is the log:\n```\n16/05/04 11:04:11 INFO hive.SparkHiveHadoopWriter: XXfinalPath=hdfs://tl-sng-gdt-nn-tdw.tencent-distribute.com:54310/tmp/assorz/tdw-tdwadmin/20160504/04559505496526517_-1_1204544348/10000/_tmp.p_20160428/attempt_201605041104_0001_m_000000_1\n\n16/05/04 11:04:48 INFO hive.SparkHiveHadoopWriter: XXfinalPath=hdfs://tl-sng-gdt-nn-tdw.tencent-distribute.com:54310/tmp/assorz/tdw-tdwadmin/20160504/04559505496526517_-1_212180468/10000/_tmp.p_20160428/attempt_201605041104_0001_m_000000_1\n\n```\n\nThe reason is the output file use SimpleDateFormat(\"yyyyMMddHHmm\"), if two sql insert into the same table in the same minute, the output will be overrite. I think we should change dateFormat to \"yyyyMMddHHmmss\", in our cluster, we can't finished a sql in one second.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: hongshen <shenh062326@126.com>\n\nCloses #14574 from shenh062326/SPARK-16985.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/993923c8f5ca719daf905285738b7fdcaf944d8c"},{"sha":"f4482225c405b9cfe078deac74e4c28e2dcc97c3","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][DOC] Fix style in examples across documentation\n\n## What changes were proposed in this pull request?\n\nThis PR fixes the documentation as below:\n\n  -  Python has 4 spaces and Java and Scala has 2 spaces (See https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide).\n\n  - Avoid excessive parentheses and curly braces for anonymous functions. (See https://github.com/databricks/scala-style-guide#anonymous)\n\n## How was this patch tested?\n\nN/A\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14593 from HyukjinKwon/minor-documentation.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/f4482225c405b9cfe078deac74e4c28e2dcc97c3"},{"sha":"79e2caa1328843457841d71642b60be919ebb1e0","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16598][SQL][TEST] Added a test case for verifying the table identifier parsing\n\n#### What changes were proposed in this pull request?\nSo far, the test cases of `TableIdentifierParserSuite` do not cover the quoted cases. We should add one for avoiding regression.\n\n#### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14244 from gatorsmile/quotedIdentifiers.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/79e2caa1328843457841d71642b60be919ebb1e0"},{"sha":"bbae20ade14e50541e4403ca7b45bf6c11695d15","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17033][ML][MLLIB] GaussianMixture should use treeAggregate to improve performance\n\n## What changes were proposed in this pull request?\n```GaussianMixture``` should use ```treeAggregate``` rather than ```aggregate``` to improve performance and scalability. In my test of dataset with 200 features and 1M instance, I found there is 20% increased performance.\nBTW, we should destroy broadcast variable ```compute``` at the end of each iteration.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14621 from yanboliang/spark-17033.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/bbae20ade14e50541e4403ca7b45bf6c11695d15"},{"sha":"2a105134e9a3efd46b761fab5e563ddebb26575d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16771][SQL] WITH clause should not fall into infinite loop.\n\n## What changes were proposed in this pull request?\n\nThis PR changes the CTE resolving rule to use only **forward-declared** tables in order to prevent infinite loops. More specifically, new logic is like the following.\n\n* Resolve CTEs in `WITH` clauses first before replacing the main SQL body.\n* When resolving CTEs, only forward-declared CTEs or base tables are referenced.\n  - Self-referencing is not allowed any more.\n  - Cross-referencing is not allowed any more.\n\n**Reported Error Scenarios**\n```scala\nscala> sql(\"WITH t AS (SELECT 1 FROM t) SELECT * FROM t\")\njava.lang.StackOverflowError\n...\nscala> sql(\"WITH t1 AS (SELECT * FROM t2), t2 AS (SELECT 2 FROM t1) SELECT * FROM t1, t2\")\njava.lang.StackOverflowError\n...\n```\nNote that `t`, `t1`, and `t2` are not declared in database. Spark falls into infinite loops before resolving table names.\n\n## How was this patch tested?\n\nPass the Jenkins tests with new two testcases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14397 from dongjoon-hyun/SPARK-16771-TREENODE.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/2a105134e9a3efd46b761fab5e563ddebb26575d"},{"sha":"91f2735a180f0af1f15303fd0a32633dfd1c1fe0","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[DOC] add config option spark.ui.enabled into document\n\n## What changes were proposed in this pull request?\n\nThe configuration doc lost the config option `spark.ui.enabled` (default value is `true`)\nI think this option is important because many cases we would like to turn it off.\nso I add it.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14604 from WeichenXu123/add_doc_param_spark_ui_enabled.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/91f2735a180f0af1f15303fd0a32633dfd1c1fe0"},{"sha":"e46cb78b3b9fd04a50b5ae50f360db612d656a48","author":{"name":"Jagadeesan","email":"77eaebe535b25af84722e54783804ea15c5d230a@us.ibm.com"},"message":"[SPARK-12370][DOCUMENTATION] Documentation should link to examples …\n\n## What changes were proposed in this pull request?\n\nWhen documentation is built is should reference examples from the same build. There are times when the docs have links that point to files in the GitHub head which may not be valid on the current release. Changed that in URLs to make them point to the right tag in git using ```SPARK_VERSION_SHORT```\n\n…from its own release version] [Streaming programming guide]\n\nAuthor: Jagadeesan <as2@us.ibm.com>\n\nCloses #14596 from jagadeesanas2/SPARK-12370.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/e46cb78b3b9fd04a50b5ae50f360db612d656a48"},{"sha":"7f7133bdccecaccd6dfb52f13c18c1e320d65f86","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[MINOR][CORE] fix warnings on depreciated methods in MesosClusterSchedulerSuite and DiskBlockObjectWriterSuite\n\n## What changes were proposed in this pull request?\n\nFixed warnings below after scanning through warnings during build:\n\n```\n[warn] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSuite.scala:34: imported `Utils' is permanently hidden by definition of object Utils in package mesos\n[warn] import org.apache.spark.scheduler.cluster.mesos.Utils\n[warn]                                                 ^\n```\n\nand\n```\n[warn] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala:113: method shuffleBytesWritten in class ShuffleWriteMetrics is deprecated: use bytesWritten instead\n[warn]     assert(writeMetrics.shuffleBytesWritten === file.length())\n[warn]                         ^\n[warn] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala:119: method shuffleBytesWritten in class ShuffleWriteMetrics is deprecated: use bytesWritten instead\n[warn]     assert(writeMetrics.shuffleBytesWritten === file.length())\n[warn]                         ^\n[warn] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala:131: method shuffleBytesWritten in class ShuffleWriteMetrics is deprecated: use bytesWritten instead\n[warn]     assert(writeMetrics.shuffleBytesWritten === file.length())\n[warn]                         ^\n[warn] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala:135: method shuffleBytesWritten in class ShuffleWriteMetrics is deprecated: use bytesWritten instead\n[warn]     assert(writeMetrics.shuffleBytesWritten === file.length())\n[warn]                         ^\n```\n\n## How was this patch tested?\n\nTested manually on local laptop.\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #14609 from keypointt/suiteWarnings.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/7f7133bdccecaccd6dfb52f13c18c1e320d65f86"},{"sha":"8c8acdec9365136cba13060ce36c22b28e29b59b","author":{"name":"GraceH","email":"e70f0e6a931ab7293132ea4f5c74d74fb0d68827@qq.com"},"message":"[SPARK-16968] Add additional options in jdbc when creating a new table\n\n## What changes were proposed in this pull request?\n\nIn the PR, we just allow the user to add additional options when create a new table in JDBC writer.\nThe options can be table_options or partition_options.\nE.g., \"CREATE TABLE t (name string) ENGINE=InnoDB DEFAULT CHARSET=utf8\"\n\nHere is the usage example:\n```\ndf.write.option(\"createTableOptions\", \"ENGINE=InnoDB DEFAULT CHARSET=utf8\").jdbc(...)\n```\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\nwill apply test result soon.\n\nAuthor: GraceH <93113783@qq.com>\n\nCloses #14559 from GraceH/jdbc_options.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/8c8acdec9365136cba13060ce36c22b28e29b59b"},{"sha":"67f025d90e6ba8c039ff45e26d34f20d24b92e6a","author":{"name":"Luciano Resende","email":"a964278a26a7baf3aa6d46d63840770571cd46e0@apache.org"},"message":"[SPARK-17023][BUILD] Upgrade to Kafka 0.10.0.1 release\n\n## What changes were proposed in this pull request?\nUpdate Kafka streaming connector to use Kafka 0.10.0.1 release\n\n## How was this patch tested?\nTested via Spark unit and integration tests\n\nAuthor: Luciano Resende <lresende@apache.org>\n\nCloses #14606 from lresende/kafka-upgrade.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/67f025d90e6ba8c039ff45e26d34f20d24b92e6a"},{"sha":"cdaa562c9a09e2e83e6df4e84d911ce1428a7a7c","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16966][SQL][CORE] App Name is a randomUUID even when \"spark.app.name\" exists\n\n## What changes were proposed in this pull request?\n\nDon't override app name specified in `SparkConf` with a random app name. Only set it if the conf has no app name even after options have been applied.\n\nSee also https://github.com/apache/spark/pull/14602\nThis is similar to Sherry302 's original proposal in https://github.com/apache/spark/pull/14556\n\n## How was this patch tested?\n\nJenkins test, with new case reproducing the bug\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14630 from srowen/SPARK-16966.2.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/cdaa562c9a09e2e83e6df4e84d911ce1428a7a7c"},{"sha":"0ebf7c1bff736cf54ec47957d71394d5b75b47a7","author":{"name":"zero323","email":"173f3ab6399f1b748c2cb687d75e045296a62aa6@users.noreply.github.com"},"message":"[SPARK-17027][ML] Avoid integer overflow in PolynomialExpansion.getPolySize\n\n## What changes were proposed in this pull request?\n\nReplaces custom choose function with o.a.commons.math3.CombinatoricsUtils.binomialCoefficient\n\n## How was this patch tested?\n\nSpark unit tests\n\nAuthor: zero323 <zero323@users.noreply.github.com>\n\nCloses #14614 from zero323/SPARK-17027.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/0ebf7c1bff736cf54ec47957d71394d5b75b47a7"},{"sha":"2a3d286f3421f6836b71afcbda3084222752e6b1","author":{"name":"Zhenglai Zhang","email":"0522130a845b4b2e33ab68c8ccdc01da08ba68ad@hotmail.com"},"message":"[WIP][MINOR][TYPO] Fix several trivival typos\n\n## What changes were proposed in this pull request?\n\n* Fixed one typo `\"overriden\"` as `\"overridden\"`, also make sure no other same typo.\n* Fixed one typo `\"lowcase\"` as `\"lowercase\"`, also make sure no other same typo.\n\n## How was this patch tested?\n\nSince the change is very tiny, so I just make sure compilation is successful.\nI am new to the spark community,  please feel free to let me do other necessary steps.\n\nThanks in advance!\n\n----\nUpdated: Found another typo `lowcase` later and fixed then in the same patch\n\nAuthor: Zhenglai Zhang <zhenglaizhang@hotmail.com>\n\nCloses #14622 from zhenglaizhang/fixtypo.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/2a3d286f3421f6836b71afcbda3084222752e6b1"},{"sha":"1a028bdefa6312bf0eec46b89a1947da7e9d84af","author":{"name":"Stavros Kontopoulos","email":"42bafac589feb2200b51ee576d084d6a9af943fa@lightbend.com"},"message":"[SPARK-11714][MESOS] Make Spark on Mesos honor port restrictions on coarse grain mode\n\n- Make mesos coarse grained scheduler accept port offers and pre-assign ports\n\nPrevious attempt was for fine grained: https://github.com/apache/spark/pull/10808\n\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@typesafe.com>\n\nCloses #11157 from skonto/honour_ports_coarse.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/1a028bdefa6312bf0eec46b89a1947da7e9d84af"},{"sha":"ddf0d1e3fe18bcd01e1447feea1b76ce86087b3b","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[TRIVIAL][ML] Fix LogisticRegression typo in error message.\n\n## What changes were proposed in this pull request?\nFix ```LogisticRegression``` typo in error message.\n\n## How was this patch tested?\nDocs change, no new tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14633 from yanboliang/lr-typo.","distinct":true,"url":"https://api.github.com/repos/SaintBacchus/spark/commits/ddf0d1e3fe18bcd01e1447feea1b76ce86087b3b"}]}
{"actor":{"display_login":"SaintBacchus"}}