{"push_id":1433693514,"size":149,"distinct_size":35,"ref":"refs/heads/master","head":"086b0c8f6788b205bc630d5ccf078f77b9751af3","before":"4ac9759f807d217b6f67badc6d5f6b7138eb92d2","commits":[{"sha":"95eb06bd7d0f7110ef62c8d1cb6337c72b10d99f","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.\n\n## What changes were proposed in this pull request?\n```spark.mlp``` should support ```RFormula``` like other ML algorithm wrappers.\nBTW, I did some cleanup and improvement for ```spark.mlp```.\n\n## How was this patch tested?\nUnit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15883 from yanboliang/spark-18438.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/95eb06bd7d0f7110ef62c8d1cb6337c72b10d99f"},{"sha":"74f5c2176d8449e41f520febd38109edaf3f4172","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18433][SQL] Improve DataSource option keys to be more case-insensitive\n\n## What changes were proposed in this pull request?\n\nThis PR aims to improve DataSource option keys to be more case-insensitive\n\nDataSource partially use CaseInsensitiveMap in code-path. For example, the following fails to find url.\n\n```scala\nval df = spark.createDataFrame(sparkContext.parallelize(arr2x2), schema2)\ndf.write.format(\"jdbc\")\n    .option(\"UrL\", url1)\n    .option(\"dbtable\", \"TEST.SAVETEST\")\n    .options(properties.asScala)\n    .save()\n```\n\nThis PR makes DataSource options to use CaseInsensitiveMap internally and also makes DataSource to use CaseInsensitiveMap generally except `InMemoryFileIndex` and `InsertIntoHadoopFsRelationCommand`. We can not pass them CaseInsensitiveMap because they creates new case-sensitive HadoopConfs by calling newHadoopConfWithOptions(options) inside.\n\n## How was this patch tested?\n\nPass the Jenkins test with newly added test cases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15884 from dongjoon-hyun/SPARK-18433.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/74f5c2176d8449e41f520febd38109edaf3f4172"},{"sha":"3e01f128284993f39463c0ccd902b774f57cce76","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[DOC][MINOR] Kafka doc: breakup into lines\n\n## Before\n\n![before](https://cloud.githubusercontent.com/assets/15843379/20340231/99b039fe-ac1b-11e6-9ba9-b44582427459.png)\n\n## After\n\n![after](https://cloud.githubusercontent.com/assets/15843379/20340236/9d5796e2-ac1b-11e6-92bb-6da40ba1a383.png)\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #15903 from lw-lin/kafka-doc-lines.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/3e01f128284993f39463c0ccd902b774f57cce76"},{"sha":"43a26899e5dd2364297eaf8985bd68367e4735a7","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-18400][STREAMING] NPE when resharding Kinesis Stream\n\n## What changes were proposed in this pull request?\n\nAvoid NPE in KinesisRecordProcessor when shutdown happens without successful init\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15882 from srowen/SPARK-18400.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/43a26899e5dd2364297eaf8985bd68367e4735a7"},{"sha":"e6145772eda8d6d3727605e80a7c2f182c801003","author":{"name":"uncleGen","email":"6269a8ea04865dd92a1986593446a029690f118a@gmail.com"},"message":"[SPARK-18410][STREAMING] Add structured kafka example\n\n## What changes were proposed in this pull request?\n\nThis PR provides structured kafka wordcount examples\n\n## How was this patch tested?\n\nAuthor: uncleGen <hustyugm@gmail.com>\n\nCloses #15849 from uncleGen/SPARK-18410.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/e6145772eda8d6d3727605e80a7c2f182c801003"},{"sha":"241e04bc03efb1379622c0c84299e617512973ac","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[MINOR][DOC] Fix typos in the 'configuration', 'monitoring' and 'sql-programming-guide' documentation\n\n## What changes were proposed in this pull request?\n\nFix typos in the 'configuration', 'monitoring' and 'sql-programming-guide' documentation.\n\n## How was this patch tested?\nManually.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15886 from weiqingy/fixTypo.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/241e04bc03efb1379622c0c84299e617512973ac"},{"sha":"c68f1a38af67957ee28889667193da8f64bb4342","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-18434][ML] Add missing ParamValidations for ML algos\n\n## What changes were proposed in this pull request?\nAdd missing ParamValidations for ML algos\n## How was this patch tested?\nexisting tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #15881 from zhengruifeng/arg_checking.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/c68f1a38af67957ee28889667193da8f64bb4342"},{"sha":"a75e3fe923372c56bc1b2f4baeaaf5868ad28341","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-18446][ML][DOCS] Add links to API docs for ML algos\n\n## What changes were proposed in this pull request?\nAdd links to API docs for ML algos\n## How was this patch tested?\nManual checking for the API links\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #15890 from zhengruifeng/algo_link.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/a75e3fe923372c56bc1b2f4baeaaf5868ad28341"},{"sha":"7569cf6cb85bda7d0e76d3e75e286d4796e77e08","author":{"name":"Xianyang Liu","email":"aa7698c29891c5a517fa7b8cd0b48e8488977230@icloud.com"},"message":"[SPARK-18420][BUILD] Fix the errors caused by lint check in Java\n\n## What changes were proposed in this pull request?\n\nSmall fix, fix the errors caused by lint check in Java\n\n- Clear unused objects and `UnusedImports`.\n- Add comments around the method `finalize` of `NioBufferedFileInputStream`to turn off checkstyle.\n- Cut the line which is longer than 100 characters into two lines.\n\n## How was this patch tested?\nTravis CI.\n```\n$ build/mvn -T 4 -q -DskipTests -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive -Phive-thriftserver install\n$ dev/lint-java\n```\nBefore:\n```\nCheckstyle checks failed at following occurrences:\n[ERROR] src/main/java/org/apache/spark/network/util/TransportConf.java:[21,8] (imports) UnusedImports: Unused import - org.apache.commons.crypto.cipher.CryptoCipherFactory.\n[ERROR] src/test/java/org/apache/spark/network/sasl/SparkSaslSuite.java:[516,5] (modifier) RedundantModifier: Redundant 'public' modifier.\n[ERROR] src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java:[133] (coding) NoFinalizer: Avoid using finalizer method.\n[ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeMapData.java:[71] (sizes) LineLength: Line is longer than 100 characters (found 113).\n[ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java:[112] (sizes) LineLength: Line is longer than 100 characters (found 110).\n[ERROR] src/test/java/org/apache/spark/sql/catalyst/expressions/HiveHasherSuite.java:[31,17] (modifier) ModifierOrder: 'static' modifier out of order with the JLS suggestions.\n[ERROR]src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java:[64] (sizes) LineLength: Line is longer than 100 characters (found 103).\n[ERROR] src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java:[22,8] (imports) UnusedImports: Unused import - org.apache.spark.ml.linalg.Vectors.\n[ERROR] src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java:[51] (regexp) RegexpSingleline: No trailing whitespace allowed.\n```\n\nAfter:\n```\n$ build/mvn -T 4 -q -DskipTests -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive -Phive-thriftserver install\n$ dev/lint-java\nUsing `mvn` from path: /home/travis/build/ConeyLiu/spark/build/apache-maven-3.3.9/bin/mvn\nCheckstyle checks passed.\n```\n\nAuthor: Xianyang Liu <xyliu0530@icloud.com>\n\nCloses #15865 from ConeyLiu/master.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/7569cf6cb85bda7d0e76d3e75e286d4796e77e08"},{"sha":"608ecc512b759514c75a1b475582f237ed569f10","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-18415][SQL] Weird Plan Output when CTE used in RunnableCommand\n\n### What changes were proposed in this pull request?\nCurrently, when CTE is used in RunnableCommand, the Analyzer does not replace the logical node `With`. The child plan of RunnableCommand is not resolved. Thus, the output of the `With` plan node looks very confusing.\nFor example,\n```\nsql(\n  \"\"\"\n    |CREATE VIEW cte_view AS\n    |WITH w AS (SELECT 1 AS n), cte1 (select 2), cte2 as (select 3)\n    |SELECT n FROM w\n  \"\"\".stripMargin).explain()\n```\nThe output is like\n```\nExecutedCommand\n   +- CreateViewCommand `cte_view`, WITH w AS (SELECT 1 AS n), cte1 (select 2), cte2 as (select 3)\nSELECT n FROM w, false, false, PersistedView\n         +- 'With [(w,SubqueryAlias w\n+- Project [1 AS n#16]\n   +- OneRowRelation$\n), (cte1,'SubqueryAlias cte1\n+- 'Project [unresolvedalias(2, None)]\n   +- OneRowRelation$\n), (cte2,'SubqueryAlias cte2\n+- 'Project [unresolvedalias(3, None)]\n   +- OneRowRelation$\n)]\n            +- 'Project ['n]\n               +- 'UnresolvedRelation `w`\n```\nAfter the fix, the output is as shown below.\n```\nExecutedCommand\n   +- CreateViewCommand `cte_view`, WITH w AS (SELECT 1 AS n), cte1 (select 2), cte2 as (select 3)\nSELECT n FROM w, false, false, PersistedView\n         +- CTE [w, cte1, cte2]\n            :  :- SubqueryAlias w\n            :  :  +- Project [1 AS n#16]\n            :  :     +- OneRowRelation$\n            :  :- 'SubqueryAlias cte1\n            :  :  +- 'Project [unresolvedalias(2, None)]\n            :  :     +- OneRowRelation$\n            :  +- 'SubqueryAlias cte2\n            :     +- 'Project [unresolvedalias(3, None)]\n            :        +- OneRowRelation$\n            +- 'Project ['n]\n               +- 'UnresolvedRelation `w`\n```\n\nBTW, this PR also fixes the output of the view type.\n\n### How was this patch tested?\nManual\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #15854 from gatorsmile/cteName.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/608ecc512b759514c75a1b475582f237ed569f10"},{"sha":"0048ce7ce64b02cbb6a1c4a2963a0b1b9541047e","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-18459][SPARK-18460][STRUCTUREDSTREAMING] Rename triggerId to batchId and add triggerDetails to json in StreamingQueryStatus\n\n## What changes were proposed in this pull request?\n\nSPARK-18459: triggerId seems like a number that should be increasing with each trigger, whether or not there is data in it. However, actually, triggerId increases only where there is a batch of data in a trigger. So its better to rename it to batchId.\n\nSPARK-18460: triggerDetails was missing from json representation. Fixed it.\n\n## How was this patch tested?\nUpdated existing unit tests.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #15895 from tdas/SPARK-18459.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/0048ce7ce64b02cbb6a1c4a2963a0b1b9541047e"},{"sha":"bb6cdfd9a6a6b6c91aada7c3174436146045ed1e","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries\n\n## What changes were proposed in this pull request?\n<img width=\"941\" alt=\"screen shot 2016-11-15 at 6 27 32 pm\" src=\"https://cloud.githubusercontent.com/assets/663212/20332521/4190b858-ab61-11e6-93a6-4bdc05105ed9.png\">\n<img width=\"940\" alt=\"screen shot 2016-11-15 at 6 27 45 pm\" src=\"https://cloud.githubusercontent.com/assets/663212/20332525/44a0d01e-ab61-11e6-8668-47f925490d4f.png\">\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #15897 from tdas/SPARK-18461.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/bb6cdfd9a6a6b6c91aada7c3174436146045ed1e"},{"sha":"a36a76ac43c36a3b897a748bd9f138b629dbc684","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed\n\n## What changes were proposed in this pull request?\n\nThis PR aims to provide a pip installable PySpark package. This does a bunch of work to copy the jars over and package them with the Python code (to prevent challenges from trying to use different versions of the Python code with different versions of the JAR). It does not currently publish to PyPI but that is the natural follow up (SPARK-18129).\n\nDone:\n- pip installable on conda [manual tested]\n- setup.py installed on a non-pip managed system (RHEL) with YARN [manual tested]\n- Automated testing of this (virtualenv)\n- packaging and signing with release-build*\n\nPossible follow up work:\n- release-build update to publish to PyPI (SPARK-18128)\n- figure out who owns the pyspark package name on prod PyPI (is it someone with in the project or should we ask PyPI or should we choose a different name to publish with like ApachePySpark?)\n- Windows support and or testing ( SPARK-18136 )\n- investigate details of wheel caching and see if we can avoid cleaning the wheel cache during our test\n- consider how we want to number our dev/snapshot versions\n\nExplicitly out of scope:\n- Using pip installed PySpark to start a standalone cluster\n- Using pip installed PySpark for non-Python Spark programs\n\n*I've done some work to test release-build locally but as a non-committer I've just done local testing.\n## How was this patch tested?\n\nAutomated testing with virtualenv, manual testing with conda, a system wide install, and YARN integration.\n\nrelease-build changes tested locally as a non-committer (no testing of upload artifacts to Apache staging websites)\n\nAuthor: Holden Karau <holden@us.ibm.com>\nAuthor: Juliet Hougland <juliet@cloudera.com>\nAuthor: Juliet Hougland <not@myemail.com>\n\nCloses #15659 from holdenk/SPARK-1267-pip-install-pyspark.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/a36a76ac43c36a3b897a748bd9f138b629dbc684"},{"sha":"2ca8ae9aa1b29bf1f46d0b656d9885e438e67f53","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-18186] Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support\n\n## What changes were proposed in this pull request?\n\nWhile being evaluated in Spark SQL, Hive UDAFs don't support partial aggregation. This PR migrates `HiveUDAFFunction`s to `TypedImperativeAggregate`, which already provides partial aggregation support for aggregate functions that may use arbitrary Java objects as aggregation states.\n\nThe following snippet shows the effect of this PR:\n\n```scala\nimport org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax\nsql(s\"CREATE FUNCTION hive_max AS '${classOf[GenericUDAFMax].getName}'\")\n\nspark.range(100).createOrReplaceTempView(\"t\")\n\n// A query using both Spark SQL native `max` and Hive `max`\nsql(s\"SELECT max(id), hive_max(id) FROM t\").explain()\n```\n\nBefore this PR:\n\n```\n== Physical Plan ==\nSortAggregate(key=[], functions=[max(id#1L), default.hive_max(default.hive_max, HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax7475f57e), id#1L, false, 0, 0)])\n+- Exchange SinglePartition\n   +- *Range (0, 100, step=1, splits=Some(1))\n```\n\nAfter this PR:\n\n```\n== Physical Plan ==\nSortAggregate(key=[], functions=[max(id#1L), default.hive_max(default.hive_max, HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax5e18a6a7), id#1L, false, 0, 0)])\n+- Exchange SinglePartition\n   +- SortAggregate(key=[], functions=[partial_max(id#1L), partial_default.hive_max(default.hive_max, HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax5e18a6a7), id#1L, false, 0, 0)])\n      +- *Range (0, 100, step=1, splits=Some(1))\n```\n\nThe tricky part of the PR is mostly about updating and passing around aggregation states of `HiveUDAFFunction`s since the aggregation state of a Hive UDAF may appear in three different forms. Let's take a look at the testing `MockUDAF` added in this PR as an example. This UDAF computes the count of non-null values together with the count of nulls of a given column. Its aggregation state may appear as the following forms at different time:\n\n1. A `MockUDAFBuffer`, which is a concrete subclass of `GenericUDAFEvaluator.AggregationBuffer`\n\n   The form used by Hive UDAF API. This form is required by the following scenarios:\n\n   - Calling `GenericUDAFEvaluator.iterate()` to update an existing aggregation state with new input values.\n   - Calling `GenericUDAFEvaluator.terminate()` to get the final aggregated value from an existing aggregation state.\n   - Calling `GenericUDAFEvaluator.merge()` to merge other aggregation states into an existing aggregation state.\n\n     The existing aggregation state to be updated must be in this form.\n\n   Conversions:\n\n   - To form 2:\n\n     `GenericUDAFEvaluator.terminatePartial()`\n\n   - To form 3:\n\n     Convert to form 2 first, and then to 3.\n\n2. An `Object[]` array containing two `java.lang.Long` values.\n\n   The form used to interact with Hive's `ObjectInspector`s. This form is required by the following scenarios:\n\n   - Calling `GenericUDAFEvaluator.terminatePartial()` to convert an existing aggregation state in form 1 to form 2.\n   - Calling `GenericUDAFEvaluator.merge()` to merge other aggregation states into an existing aggregation state.\n\n     The input aggregation state must be in this form.\n\n   Conversions:\n\n   - To form 1:\n\n     No direct method. Have to create an empty `AggregationBuffer` and merge it into the empty buffer.\n\n   - To form 3:\n\n     `unwrapperFor()`/`unwrap()` method of `HiveInspectors`\n\n3. The byte array that holds data of an `UnsafeRow` with two `LongType` fields.\n\n   The form used by Spark SQL to shuffle partial aggregation results. This form is required because `TypedImperativeAggregate` always asks its subclasses to serialize their aggregation states into a byte array.\n\n   Conversions:\n\n   - To form 1:\n\n     Convert to form 2 first, and then to 1.\n\n   - To form 2:\n\n     `wrapperFor()`/`wrap()` method of `HiveInspectors`\n\nHere're some micro-benchmark results produced by the most recent master and this PR branch.\n\nMaster:\n\n```\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.10.5\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n\nhive udaf vs spark af:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nw/o groupBy                                    339 /  372          3.1         323.2       1.0X\nw/ groupBy                                     503 /  529          2.1         479.7       0.7X\n```\n\nThis PR:\n\n```\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.10.5\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n\nhive udaf vs spark af:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nw/o groupBy                                    116 /  126          9.0         110.8       1.0X\nw/ groupBy                                     151 /  159          6.9         144.0       0.8X\n```\n\nBenchmark code snippet:\n\n```scala\n  test(\"Hive UDAF benchmark\") {\n    val N = 1 << 20\n\n    sparkSession.sql(s\"CREATE TEMPORARY FUNCTION hive_max AS '${classOf[GenericUDAFMax].getName}'\")\n\n    val benchmark = new Benchmark(\n      name = \"hive udaf vs spark af\",\n      valuesPerIteration = N,\n      minNumIters = 5,\n      warmupTime = 5.seconds,\n      minTime = 5.seconds,\n      outputPerIteration = true\n    )\n\n    benchmark.addCase(\"w/o groupBy\") { _ =>\n      sparkSession.range(N).agg(\"id\" -> \"hive_max\").collect()\n    }\n\n    benchmark.addCase(\"w/ groupBy\") { _ =>\n      sparkSession.range(N).groupBy($\"id\" % 10).agg(\"id\" -> \"hive_max\").collect()\n    }\n\n    benchmark.run()\n\n    sparkSession.sql(s\"DROP TEMPORARY FUNCTION IF EXISTS hive_max\")\n  }\n```\n\n## How was this patch tested?\n\nNew test suite `HiveUDAFSuite` is added.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #15703 from liancheng/partial-agg-hive-udaf.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/2ca8ae9aa1b29bf1f46d0b656d9885e438e67f53"},{"sha":"55589987be89ff78dadf44498352fbbd811a206e","author":{"name":"Artur Sukhenko","email":"423148ee03a5d8e0710210b712bacea5e4c6b8fa@gmail.com"},"message":"[YARN][DOC] Increasing NodeManager's heap size with External Shuffle Service\n\n## What changes were proposed in this pull request?\n\nSuggest users to increase `NodeManager's` heap size if `External Shuffle Service` is enabled as\n`NM` can spend a lot of time doing GC resulting in  shuffle operations being a bottleneck due to `Shuffle Read blocked time` bumped up.\nAlso because of GC  `NodeManager` can use an enormous amount of CPU and cluster performance will suffer.\nI have seen NodeManager using 5-13G RAM and up to 2700% CPU with `spark_shuffle` service on.\n\n## How was this patch tested?\n\n#### Added step 5:\n![shuffle_service](https://cloud.githubusercontent.com/assets/15244468/20355499/2fec0fde-ac2a-11e6-8f8b-1c80daf71be1.png)\n\nAuthor: Artur Sukhenko <artur.sukhenko@gmail.com>\n\nCloses #15906 from Devian-ua/nmHeapSize.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/55589987be89ff78dadf44498352fbbd811a206e"},{"sha":"170eeb345f951de89a39fe565697b3e913011768","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-18442][SQL] Fix nullability of WrapOption.\n\n## What changes were proposed in this pull request?\n\nThe nullability of `WrapOption` should be `false`.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #15887 from ueshin/issues/SPARK-18442.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/170eeb345f951de89a39fe565697b3e913011768"},{"sha":"07b3f045cd6f79b92bc86b3b1b51d3d5e6bd37ce","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18464][SQL] support old table which doesn't store schema in metastore\n\n## What changes were proposed in this pull request?\n\nBefore Spark 2.1, users can create an external data source table without schema, and we will infer the table schema at runtime. In Spark 2.1, we decided to infer the schema when the table was created, so that we don't need to infer it again and again at runtime.\n\nThis is a good improvement, but we should still respect and support old tables which doesn't store table schema in metastore.\n\n## How was this patch tested?\n\nregression test.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15900 from cloud-fan/hive-catalog.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/07b3f045cd6f79b92bc86b3b1b51d3d5e6bd37ce"},{"sha":"a3cac7bd86a6fe8e9b42da1bf580aaeb59378304","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[YARN][DOC] Remove non-Yarn specific configurations from running-on-yarn.md\n\n## What changes were proposed in this pull request?\n\nRemove `spark.driver.memory`, `spark.executor.memory`,  `spark.driver.cores`, and `spark.executor.cores` from `running-on-yarn.md` as they are not Yarn-specific, and they are also defined in`configuration.md`.\n\n## How was this patch tested?\nBuild passed & Manually check.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15869 from weiqingy/yarnDoc.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/a3cac7bd86a6fe8e9b42da1bf580aaeb59378304"},{"sha":"49b6f456aca350e9e2c170782aa5cc75e7822680","author":{"name":"anabranch","email":"7006623d14062d99b561e140f496506078132657@gmail.com"},"message":"[SPARK-18365][DOCS] Improve Sample Method Documentation\n\n## What changes were proposed in this pull request?\n\nI found the documentation for the sample method to be confusing, this adds more clarification across all languages.\n\n- [x] Scala\n- [x] Python\n- [x] R\n- [x] RDD Scala\n- [ ] RDD Python with SEED\n- [X] RDD Java\n- [x] RDD Java with SEED\n- [x] RDD Python\n\n## How was this patch tested?\n\nNA\n\nPlease review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.\n\nAuthor: anabranch <wac.chambers@gmail.com>\nAuthor: Bill Chambers <bill@databricks.com>\n\nCloses #15815 from anabranch/SPARK-18365.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/49b6f456aca350e9e2c170782aa5cc75e7822680"},{"sha":"de77c67750dc868d75d6af173c3820b75a9fe4b7","author":{"name":"VinceShieh","email":"2445c9be3c24c11954fcdbf8bdec5f86733bb70d@intel.com"},"message":"[SPARK-17462][MLLIB]use VersionUtils to parse Spark version strings\n\n## What changes were proposed in this pull request?\n\nSeveral places in MLlib use custom regexes or other approaches to parse Spark versions.\nThose should be fixed to use the VersionUtils. This PR replaces custom regexes with\nVersionUtils to get Spark version numbers.\n## How was this patch tested?\n\nExisting tests.\n\nSigned-off-by: VinceShieh vincent.xieintel.com\n\nAuthor: VinceShieh <vincent.xie@intel.com>\n\nCloses #15055 from VinceShieh/SPARK-17462.","distinct":false,"url":"https://api.github.com/repos/wangmiao1981/spark/commits/de77c67750dc868d75d6af173c3820b75a9fe4b7"}]}
{"actor":{"display_login":"wangmiao1981"}}