{"push_id":1398559558,"size":34,"distinct_size":1,"ref":"refs/heads/SAAS-467_branch-2.0","head":"104e574acfaab3fd844704dd1fbabfe60625b3cb","before":"1d9476a4b0ca1613e12884ddebaa7be0aca0f795","commits":[{"sha":"9ed8976c6efc853cbe6dde3c5f44d55d063c7a20","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18164][SQL] ForeachSink should fail the Spark job if `process` throws exception\n\n## What changes were proposed in this pull request?\n\nFixed the issue that ForeachSink didn't rethrow the exception.\n\n## How was this patch tested?\n\nThe fixed unit test.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15674 from zsxwing/foreach-sink-error.\n\n(cherry picked from commit 59cccbda489f25add3e10997e950de7e88704aa7)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/9ed8976c6efc853cbe6dde3c5f44d55d063c7a20"},{"sha":"9f924747d7f37729f89a30279ca774b12e342ba2","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-16312][FOLLOW-UP][STREAMING][KAFKA][DOC] Add java code snippet for Kafka 0.10 integration doc\n\n## What changes were proposed in this pull request?\n\nadded java code snippet for Kafka 0.10 integration doc\n\n## How was this patch tested?\n\nSKIP_API=1 jekyll build\n\n## Screenshot\n\n![kafka-doc](https://cloud.githubusercontent.com/assets/15843379/19826272/bf0d8a4c-9db8-11e6-9e40-1396723df4bc.png)\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #15679 from lw-lin/kafka-010-examples.\n\n(cherry picked from commit 505b927cb7ff037adb797b9c3b9ecac3f885b7c8)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/9f924747d7f37729f89a30279ca774b12e342ba2"},{"sha":"300d596a5177ae372194f73f717174b7ff7acd36","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18143][SQL] Ignore Structured Streaming event logs to avoid breaking history server (branch 2.0)\n\n## What changes were proposed in this pull request?\n\nBackport #15663 to branch-2.0 and fixed conflicts in `ReplayListenerBus`.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15695 from zsxwing/fix-event-log-2.0.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/300d596a5177ae372194f73f717174b7ff7acd36"},{"sha":"e06f43e33ac1650fc401cd90f4123e4dbd23e066","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18030][TESTS] Fix flaky FileStreamSourceSuite by not deleting the files\n\n## What changes were proposed in this pull request?\n\nThe test `when schema inference is turned on, should read partition data` should not delete files because the source maybe is listing files. This PR just removes the delete actions since they are not necessary.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15699 from zsxwing/SPARK-18030.\n\n(cherry picked from commit de3f87fa712c305fdd463fc36acffc5418c95c4d)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/e06f43e33ac1650fc401cd90f4123e4dbd23e066"},{"sha":"4d2672a408b6061b0e20b622960f7c6f14271275","author":{"name":"Wang Lei","email":"2ae6f49ce33307a941f16d81e58cb429f0aaf78d@kongming-inc.com"},"message":"[SPARK-18114][MESOS] Fix mesos cluster scheduler generage command option error\n\nEnclose --conf option value with \"\" to support multi value configs like spark.driver.extraJavaOptions, without \"\", driver will fail to start.\n\nJenkins Tests.\n\nTest in our production environment, also unit tests, It is a very small change.\n\nAuthor: Wang Lei <lei.wang@kongming-inc.com>\n\nCloses #15643 from LeightonWong/messos-cluster.\n\n(cherry picked from commit 9b377aa49f14af31f54164378d60e0fdea2142e5)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/4d2672a408b6061b0e20b622960f7c6f14271275"},{"sha":"58655f51f65d852ec65a65b54f26b3c8eac8cc60","author":{"name":"Ergin Seyfe","email":"551bf3c71d9da660a52ecd3224fc4c86679b46d0@fb.com"},"message":"[SPARK-18189][SQL] Fix serialization issue in KeyValueGroupedDataset\n\n## What changes were proposed in this pull request?\nLikewise [DataSet.scala](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L156) KeyValueGroupedDataset should mark the queryExecution as transient.\n\nAs mentioned in the Jira ticket, without transient we saw serialization issues like\n\n```\nCaused by: java.io.NotSerializableException: org.apache.spark.sql.execution.QueryExecution\nSerialization stack:\n        - object not serializable (class: org.apache.spark.sql.execution.QueryExecution, value: ==\n```\n\n## How was this patch tested?\n\nRun the query which is specified in the Jira ticket before and after:\n```\nval a = spark.createDataFrame(sc.parallelize(Seq((1,2),(3,4)))).as[(Int,Int)]\nval grouped = a.groupByKey(\n{x:(Int,Int)=>x._1}\n)\nval mappedGroups = grouped.mapGroups((k,x)=>\n{(k,1)}\n)\nval yyy = sc.broadcast(1)\nval last = mappedGroups.rdd.map(xx=>\n{ val simpley = yyy.value 1 }\n)\n```\n\nAuthor: Ergin Seyfe <eseyfe@fb.com>\n\nCloses #15706 from seyfe/keyvaluegrouped_serialization.\n\n(cherry picked from commit 8a538c97b556f80f67c80519af0ce879557050d5)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/58655f51f65d852ec65a65b54f26b3c8eac8cc60"},{"sha":"4176da8be57bb0b36b9f2c580a547713c2048d17","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-18148][SQL] Misleading Error Message for Aggregation Without Window/GroupBy\n\n## What changes were proposed in this pull request?\n\nAggregation Without Window/GroupBy expressions will fail in `checkAnalysis`, the error message is a bit misleading, we should generate a more specific error message for this case.\n\nFor example,\n\n```\nspark.read.load(\"/some-data\")\n  .withColumn(\"date_dt\", to_date($\"date\"))\n  .withColumn(\"year\", year($\"date_dt\"))\n  .withColumn(\"week\", weekofyear($\"date_dt\"))\n  .withColumn(\"user_count\", count($\"userId\"))\n  .withColumn(\"daily_max_in_week\", max($\"user_count\").over(weeklyWindow))\n)\n```\n\ncreates the following output:\n\n```\norg.apache.spark.sql.AnalysisException: expression '`randomColumn`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;\n```\n\nIn the error message above, `randomColumn` doesn't appear in the query(acturally it's added by function `withColumn`), so the message is not enough for the user to address the problem.\n## How was this patch tested?\n\nManually test\n\nBefore:\n\n```\nscala> spark.sql(\"select col, count(col) from tbl\")\norg.apache.spark.sql.AnalysisException: expression 'tbl.`col`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;\n```\n\nAfter:\n\n```\nscala> spark.sql(\"select col, count(col) from tbl\")\norg.apache.spark.sql.AnalysisException: grouping expressions sequence is empty, and 'tbl.`col`' is not an aggregate function. Wrap '(count(col#231L) AS count(col)#239L)' in windowing function(s) or wrap 'tbl.`col`' in first() (or first_value) if you don't care which value you get.;;\n```\n\nAlso add new test sqls in `group-by.sql`.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #15672 from jiangxb1987/groupBy-empty.\n\n(cherry picked from commit d0272b436512b71f04313e109d3d21a6e9deefca)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/4176da8be57bb0b36b9f2c580a547713c2048d17"},{"sha":"a01b950602c4bb56c5a7d6213cdf6b7515ff36ec","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-18114][HOTFIX] Fix line-too-long style error from backport of SPARK-18114\n\n## What changes were proposed in this pull request?\n\nFix style error introduced in cherry-pick of https://github.com/apache/spark/pull/15643 to branch-2.0.\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15719 from srowen/SPARK-18114.2.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/a01b950602c4bb56c5a7d6213cdf6b7515ff36ec"},{"sha":"a6abe1ee22141931614bf27a4f371c46d8379e33","author":{"name":"Patrick Wendell","email":"dfc81a5a586cf6efa894b43abd4f9d9faf041679@gmail.com"},"message":"Preparing Spark release v2.0.2-rc2","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/a6abe1ee22141931614bf27a4f371c46d8379e33"},{"sha":"d401a74d40c7d29ea4f8ab6dfafb24baa2b87ece","author":{"name":"Patrick Wendell","email":"dfc81a5a586cf6efa894b43abd4f9d9faf041679@gmail.com"},"message":"Preparing development version 2.0.3-SNAPSHOT","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/d401a74d40c7d29ea4f8ab6dfafb24baa2b87ece"},{"sha":"81f0804252bbfdc280e013e5f7b016759e66406e","author":{"name":"CodingCat","email":"66d548c0de2a741ca15ed6f9c9c0da75022b9805@gmail.com"},"message":"[SPARK-18144][SQL] logging StreamingQueryListener$QueryStartedEvent\n\n## What changes were proposed in this pull request?\n\nThe PR fixes the bug that the QueryStartedEvent is not logged\n\nthe postToAll() in the original code is actually calling StreamingQueryListenerBus.postToAll() which has no listener at all....we shall post by sparkListenerBus.postToAll(s) and this.postToAll() to trigger local listeners as well as the listeners registered in LiveListenerBus\n\nzsxwing\n## How was this patch tested?\n\nThe following snapshot shows that QueryStartedEvent has been logged correctly\n\n![image](https://cloud.githubusercontent.com/assets/678008/19821553/007a7d28-9d2d-11e6-9f13-49851559cdaa.png)\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #15675 from CodingCat/SPARK-18144.\n\n(cherry picked from commit 85c5424d466f4a5765c825e0e2ab30da97611285)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/81f0804252bbfdc280e013e5f7b016759e66406e"},{"sha":"09178b6eefd33011c3e90164356a5d6c3ae737bd","author":{"name":"Jagadeesan","email":"77eaebe535b25af84722e54783804ea15c5d230a@us.ibm.com"},"message":"[SPARK-18133][BRANCH-2.0][EXAMPLES][ML] Python ML Pipeline Exampl…\n\n## What changes were proposed in this pull request?\n\n[Fix] [branch-2.0] In Python 3, there is only one integer type (i.e., int), which mostly behaves like the long type in Python 2. Since Python 3 won't accept \"L\", so removed \"L\" in all examples.\n\n## How was this patch tested?\n\nUnit tests.\n\n…e has syntax errors]\n\nAuthor: Jagadeesan <as2@us.ibm.com>\n\nCloses #15729 from jagadeesanas2/SPARK-18133_branch2.0.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/09178b6eefd33011c3e90164356a5d6c3ae737bd"},{"sha":"eb790c5b110c1227d0d6c8593254de196f65da35","author":{"name":"Artur Sukhenko","email":"423148ee03a5d8e0710210b712bacea5e4c6b8fa@gmail.com"},"message":"[SPARK-16796][WEB UI] Mask spark.authenticate.secret on Spark environ…\n\n## What changes were proposed in this pull request?\n\nMask `spark.authenticate.secret` on Spark environment page (Web UI).\nThis is addition to https://github.com/apache/spark/pull/14409\n\n## How was this patch tested?\n`./dev/run-tests`\n[info] ScalaTest\n[info] Run completed in 1 hour, 8 minutes, 38 seconds.\n[info] Total number of tests run: 2166\n[info] Suites: completed 65, aborted 0\n[info] Tests: succeeded 2166, failed 0, canceled 0, ignored 590, pending 0\n[info] All tests passed.\n\nAuthor: Artur Sukhenko <artur.sukhenko@gmail.com>\n\nCloses #14484 from Devian-ua/SPARK-16796.\n\n(cherry picked from commit 14dba45208d8a5511be2cf8ddf22e688ef141e88)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/eb790c5b110c1227d0d6c8593254de196f65da35"},{"sha":"1696bcfadabb91693bf1ab556a321949d1e4fe45","author":{"name":"Jeff Zhang","email":"8540ccd5710ce8e6da3f6c1c8189a4ef2fa98c60@apache.org"},"message":"[SPARK-18160][CORE][YARN] spark.files & spark.jars should not be passed to driver in yarn mode\n\n## What changes were proposed in this pull request?\n\nspark.files is still passed to driver in yarn mode, so SparkContext will still handle it which cause the error in the jira desc.\n\n## How was this patch tested?\n\nTested manually in a 5 node cluster. As this issue only happens in multiple node cluster, so I didn't write test for it.\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #15669 from zjffdu/SPARK-18160.\n\n(cherry picked from commit 3c24299b71e23e159edbb972347b13430f92a465)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/1696bcfadabb91693bf1ab556a321949d1e4fe45"},{"sha":"3253ae7f722a996cf0af21608e1a27d5d2a12004","author":{"name":"wangzhenhua","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-18111][SQL] Wrong approximate quantile answer when multiple records have the minimum value(for branch 2.0)\n\n## What changes were proposed in this pull request?\nWhen multiple records have the minimum value, the answer of `StatFunctions.multipleApproxQuantiles` is wrong.\n\n## How was this patch tested?\nadd a test case\n\nAuthor: wangzhenhua <wangzhenhua@huawei.com>\n\nCloses #15732 from wzhfy/percentile2.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/3253ae7f722a996cf0af21608e1a27d5d2a12004"},{"sha":"dae1581d9461346511098dc83938939a0f930048","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18200][GRAPHX] Support zero as an initial capacity in OpenHashSet\n\n## What changes were proposed in this pull request?\n\n[SPARK-18200](https://issues.apache.org/jira/browse/SPARK-18200) reports Apache Spark 2.x raises `java.lang.IllegalArgumentException: requirement failed: Invalid initial capacity` while running `triangleCount`. The root cause is that `VertexSet`, a type alias of `OpenHashSet`, does not allow zero as a initial size. This PR loosens the restriction to allow zero.\n\n## How was this patch tested?\n\nPass the Jenkins test with a new test case in `OpenHashSetSuite`.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15741 from dongjoon-hyun/SPARK-18200.\n\n(cherry picked from commit d24e736471f34ef8f2c12766393379c4213fe96e)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/dae1581d9461346511098dc83938939a0f930048"},{"sha":"c864e8a8020f4890f1839766851e7f4917da5c70","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18200][GRAPHX][FOLLOW-UP] Support zero as an initial capacity in OpenHashSet\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up PR of #15741 in order to keep `nextPowerOf2` consistent.\n\n**Before**\n```\nnextPowerOf2(0) => 2\nnextPowerOf2(1) => 1\nnextPowerOf2(2) => 2\nnextPowerOf2(3) => 4\nnextPowerOf2(4) => 4\nnextPowerOf2(5) => 8\n```\n\n**After**\n```\nnextPowerOf2(0) => 1\nnextPowerOf2(1) => 1\nnextPowerOf2(2) => 2\nnextPowerOf2(3) => 4\nnextPowerOf2(4) => 4\nnextPowerOf2(5) => 8\n```\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15754 from dongjoon-hyun/SPARK-18200-2.\n\n(cherry picked from commit 27602c33751cebf6cd173c0de103454608cf6625)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/c864e8a8020f4890f1839766851e7f4917da5c70"},{"sha":"399597b04a83bbe3cc748c21446de0d808d08155","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17337][SPARK-16804][SQL][BRANCH-2.0] Backport subquery related PRs\n\n## What changes were proposed in this pull request?\nThis PR backports two subquery related PRs to branch-2.0:\n\n- https://github.com/apache/spark/pull/14411\n- https://github.com/apache/spark/pull/15761\n\n## How was this patch tested?\nAdded a tests  to `SubquerySuite`.\n\nAuthor: Nattavut Sutyanyong <nsy.can@gmail.com>\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #15772 from hvanhovell/SPARK-17337-2.0.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/399597b04a83bbe3cc748c21446de0d808d08155"},{"sha":"8b99e204a9a056fd071f9bd75f3e0a29f90bccc0","author":{"name":"Ergin Seyfe","email":"551bf3c71d9da660a52ecd3224fc4c86679b46d0@fb.com"},"message":"[SPARK-18189][SQL][FOLLOWUP] Move test from ReplSuite to prevent java.lang.ClassCircularityError\n\n## What changes were proposed in this pull request?\nMove the test which is causing java.lang.ClassCircularityError from ReplSuite to DatasetSuite.\n\n## How was this patch tested?\n> build/mvn -DskipTests -Phadoop-2.3 -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl -Pmesos clean package\n> build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.repl.ReplSuite test\n\nAuthor: Ergin Seyfe <eseyfe@fb.com>\n\nCloses #15774 from seyfe/fix_replsuite_test_error_branch2.0.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/8b99e204a9a056fd071f9bd75f3e0a29f90bccc0"},{"sha":"d023c6c456445d7b9e377d1b2fb8a85754ee3648","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-17981][SPARK-17957][SQL][BACKPORT-2.0] Fix Incorrect Nullability Setting to False in FilterExec\n\n### What changes were proposed in this pull request?\n\n**This PR is to backport the fix https://github.com/apache/spark/pull/15523 to 2.0.**\n\nWhen `FilterExec` contains `isNotNull`, which could be inferred and pushed down or users specified, we convert the nullability of the involved columns if the top-layer expression is null-intolerant. However, this is not correct, if the top-layer expression is not a leaf expression, it could still tolerate the null when it has null-tolerant child expressions.\n\nFor example, `cast(coalesce(a#5, a#15) as double)`. Although `cast` is a null-intolerant expression, but obviously`coalesce` is null-tolerant. Thus, it could eat null.\n\nWhen the nullability is wrong, we could generate incorrect results in different cases. For example,\n\n``` Scala\n    val df1 = Seq((1, 2), (2, 3)).toDF(\"a\", \"b\")\n    val df2 = Seq((2, 5), (3, 4)).toDF(\"a\", \"c\")\n    val joinedDf = df1.join(df2, Seq(\"a\"), \"outer\").na.fill(0)\n    val df3 = Seq((3, 1)).toDF(\"a\", \"d\")\n    joinedDf.join(df3, \"a\").show\n```\n\nThe optimized plan is like\n\n```\nProject [a#29, b#30, c#31, d#42]\n+- Join Inner, (a#29 = a#41)\n   :- Project [cast(coalesce(cast(coalesce(a#5, a#15) as double), 0.0) as int) AS a#29, cast(coalesce(cast(b#6 as double), 0.0) as int) AS b#30, cast(coalesce(cast(c#16 as double), 0.0) as int) AS c#31]\n   :  +- Filter isnotnull(cast(coalesce(cast(coalesce(a#5, a#15) as double), 0.0) as int))\n   :     +- Join FullOuter, (a#5 = a#15)\n   :        :- LocalRelation [a#5, b#6]\n   :        +- LocalRelation [a#15, c#16]\n   +- LocalRelation [a#41, d#42]\n```\n\nWithout the fix, it returns an empty result. With the fix, it can return a correct answer:\n\n```\n+---+---+---+---+\n|  a|  b|  c|  d|\n+---+---+---+---+\n|  3|  0|  4|  1|\n+---+---+---+---+\n```\n### How was this patch tested?\n\nAdded test cases to verify the nullability changes in FilterExec. Also added a test case for verifying the reported incorrect result.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #15781 from gatorsmile/nullabilityFix.","distinct":false,"url":"https://api.github.com/repos/vijoshi/spark/commits/d023c6c456445d7b9e377d1b2fb8a85754ee3648"}]}
{"actor":{"display_login":"vijoshi"}}