{"push_id":3075663278,"size":10,"distinct_size":3,"ref":"refs/heads/master","head":"8e8d1177e623d5f995fb9ba1d9574675e1e70d56","before":"ab00533490953164cb2360bf2b9adc2c9fa962db","commits":[{"sha":"aeda76e2b74ef07b2814770d68cf145cdbb0197c","author":{"name":"Huon Wilson","email":"43a4f5cb87a5ab0d8cffacce88b235f4a61a732d@data61.csiro.au"},"message":"[GRAPHX] Remove unused variables left over by previous refactoring.\n\n## What changes were proposed in this pull request?\n\nSome variables were previously used for indexing the routing table's backing\narray, but that indexing now happens elsewhere, and so the variables aren't needed.\n\n## How was this patch tested?\n\nUnit tests.\n\n(This contribution is my original work and I license the work to Spark under its open source license.)\n\nCloses #23112 from huonw/remove-unused-variables.\n\nAuthored-by: Huon Wilson <Huon.Wilson@data61.csiro.au>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/aeda76e2b74ef07b2814770d68cf145cdbb0197c"},{"sha":"dd8c179c28c5df20210b70a69d93d866ccaca4cc","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-25867][ML] Remove KMeans computeCost\n\n## What changes were proposed in this pull request?\n\nThe PR removes the deprecated method `computeCost` of `KMeans`.\n\n## How was this patch tested?\n\nNA\n\nCloses #22875 from mgaido91/SPARK-25867.\n\nAuthored-by: Marco Gaido <marcogaido91@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/dd8c179c28c5df20210b70a69d93d866ccaca4cc"},{"sha":"d81d95a7e8a621e42c9c61305c32df72b6e868be","author":{"name":"oraviv","email":"1429817b238f1fb0092db18cfea4aed87eba1df5@paypal.com"},"message":"[SPARK-19368][MLLIB] BlockMatrix.toIndexedRowMatrix() optimization for sparse matrices\n\n## What changes were proposed in this pull request?\n\nOptimization [SPARK-12869] was made for dense matrices but caused great performance issue for sparse matrices because manipulating them is very inefficient. When manipulating sparse matrices in Breeze we better use VectorBuilder.\n\n## How was this patch tested?\n\nchecked it against a use case that we have that after moving to Spark 2 took 6.5 hours instead of 20 mins. After the change it is back to 20 mins again.\n\nCloses #16732 from uzadude/SparseVector_optimization.\n\nAuthored-by: oraviv <oraviv@paypal.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/d81d95a7e8a621e42c9c61305c32df72b6e868be"},{"sha":"1d766f0e222c24e8e8cad68e664e83f4f71f7541","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-26144][BUILD] `build/mvn` should detect `scala.version` based on `scala.binary.version`\n\n## What changes were proposed in this pull request?\n\nCurrently, `build/mvn` downloads and uses **Scala 2.12.7** in `Scala-2.11` Jenkins job. The root cause is `build/mvn` got the first match from `pom.xml` blindly.\n\n- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/6/consoleFull\n```\nexec: curl -s -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz\nexec: curl -s -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz\nexec: curl -s -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz\n```\n\n## How was this patch tested?\n\nManual.\n```\n$ build/mvn clean\nexec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz\n...\n$ git clean -fdx\n$ dev/change-scala-version.sh 2.11\n$ build/mvn clean\nexec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz\n```\n\nCloses #23118 from dongjoon-hyun/SPARK-26144.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/1d766f0e222c24e8e8cad68e664e83f4f71f7541"},{"sha":"76aae7f1fd512f150ffcdb618107b12e1e97fe43","author":{"name":"jerryshao","email":"7ffee68d8ba2a2ff2bad042b9992aeb7dbd2d082@tencent.com"},"message":"[SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect\n\n## What changes were proposed in this pull request?\n\nThis PR is a follow-up PR of #21600 to fix the unnecessary UI redirect.\n\n## How was this patch tested?\n\nLocal verification\n\nCloses #23116 from jerryshao/SPARK-24553.\n\nAuthored-by: jerryshao <jerryshao@tencent.com>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/76aae7f1fd512f150ffcdb618107b12e1e97fe43"},{"sha":"0ec7b99ea2b638453ed38bb092905bee4f907fe5","author":{"name":"Alon Doron","email":"7fc40a908ad54d89c9632e7fbe0936b5b343c843@palantir.com"},"message":"[SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float\n\nGROUP BY treats -0.0 and 0.0 as different values which is unlike hive's behavior.\nIn addition current behavior with codegen is unpredictable (see example in JIRA ticket).\n\n## What changes were proposed in this pull request?\n\nIn Platform.putDouble/Float() checking if the value is -0.0, and if so replacing with 0.0.\nThis is used by UnsafeRow so it won't have -0.0 values.\n\n## How was this patch tested?\n\nAdded tests\n\nCloses #23043 from adoron/adoron-spark-26021-replace-minus-zero-with-zero.\n\nAuthored-by: Alon Doron <adoron@palantir.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/0ec7b99ea2b638453ed38bb092905bee4f907fe5"},{"sha":"1d3dd58d21400b5652b75af7e7e53aad85a31528","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zookeeper 3.4.7 test dependency to Kafka modules\n\n## What changes were proposed in this pull request?\n\nThis is a followup of #23099 . After upgrading to Kafka 2.1.0, maven test fails due to Zookeeper test dependency while sbt test succeeds.\n\n- [sbt test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/5203/)\n- [maven test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5653/)\n\nThe root cause is that the embedded Kafka server is using [Zookeepr 3.4.7 API](https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html\n) while Apache Spark provides Zookeeper 3.4.6. This PR adds a test dependency.\n\n```\nKafkaMicroBatchV2SourceSuite:\n*** RUN ABORTED ***\n...\norg.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123)\n  ...\n  Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback\n  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\n  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)\n  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n  at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)\n  at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)\n  at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)\n  at kafka.server.KafkaServer.startup(KafkaServer.scala:202)\n  at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120)\n  at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116)\n  ...\n```\n\n## How was this patch tested?\n\nPass the maven Jenkins test.\n\nCloses #23119 from dongjoon-hyun/SPARK-25954-2.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/1d3dd58d21400b5652b75af7e7e53aad85a31528"},{"sha":"92fc0a8f9619a8e7f8382d6a5c288aeceb03a472","author":{"name":"Shixiong Zhu","email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com"},"message":"[SPARK-26069][TESTS][FOLLOWUP] Add another possible error message\n\n## What changes were proposed in this pull request?\n\n`org.apache.spark.network.RpcIntegrationSuite.sendRpcWithStreamFailures` is still flaky and here is error message:\n\n```\nsbt.ForkMain$ForkError: java.lang.AssertionError: Got a non-empty set [Failed to send RPC RPC 8249697863992194475 to /172.17.0.2:41177: java.io.IOException: Broken pipe]\n\tat org.junit.Assert.fail(Assert.java:88)\n\tat org.junit.Assert.assertTrue(Assert.java:41)\n\tat org.apache.spark.network.RpcIntegrationSuite.assertErrorAndClosed(RpcIntegrationSuite.java:389)\n\tat org.apache.spark.network.RpcIntegrationSuite.sendRpcWithStreamFailures(RpcIntegrationSuite.java:347)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)\n\tat org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)\n\tat org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)\n\tat org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)\n\tat org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)\n\tat org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)\n\tat org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)\n\tat org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)\n\tat org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)\n\tat org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)\n\tat org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)\n\tat org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)\n\tat org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)\n\tat org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)\n\tat org.junit.runners.ParentRunner.run(ParentRunner.java:363)\n\tat org.junit.runners.Suite.runChild(Suite.java:128)\n\tat org.junit.runners.Suite.runChild(Suite.java:27)\n\tat org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)\n\tat org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)\n\tat org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)\n\tat org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)\n\tat org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)\n\tat org.junit.runners.ParentRunner.run(ParentRunner.java:363)\n\tat org.junit.runner.JUnitCore.run(JUnitCore.java:137)\n\tat org.junit.runner.JUnitCore.run(JUnitCore.java:115)\n\tat com.novocode.junit.JUnitRunner$1.execute(JUnitRunner.java:132)\n\tat sbt.ForkMain$Run$2.call(ForkMain.java:296)\n\tat sbt.ForkMain$Run$2.call(ForkMain.java:286)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n```\n\nThis happened when the second RPC message was being sent but the connection was closed at the same time.\n\n## How was this patch tested?\n\nJenkins\n\nCloses #23109 from zsxwing/SPARK-26069-2.\n\nAuthored-by: Shixiong Zhu <zsxwing@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/92fc0a8f9619a8e7f8382d6a5c288aeceb03a472"},{"sha":"466d011d3515723653e41d8b1d0b6150b9945f52","author":{"name":"caoxuewen","email":"eb7f0bd4caa8b82cb6828bfd54ff04d7dff1b79c@zte.com.cn"},"message":"[SPARK-26117][CORE][SQL] use SparkOutOfMemoryError instead of OutOfMemoryError when catch exception\n\n## What changes were proposed in this pull request?\n\nthe pr #20014 which introduced `SparkOutOfMemoryError` to avoid killing the entire executor when an `OutOfMemoryError `is thrown.\nso apply for memory using `MemoryConsumer. allocatePage `when  catch exception, use `SparkOutOfMemoryError `instead of `OutOfMemoryError`\n\n## How was this patch tested?\nN / A\n\nCloses #23084 from heary-cao/SparkOutOfMemoryError.\n\nAuthored-by: caoxuewen <cao.xuewen@zte.com.cn>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/466d011d3515723653e41d8b1d0b6150b9945f52"},{"sha":"8e8d1177e623d5f995fb9ba1d9574675e1e70d56","author":{"name":"Maxim Gekk","email":"b91a34c712a29cd91cde3972985e114f1bb54d88@gmail.com"},"message":"[SPARK-26108][SQL] Support custom lineSep in CSV datasource\n\n## What changes were proposed in this pull request?\n\nIn the PR,  I propose new options for CSV datasource - `lineSep` similar to Text and JSON datasource. The option allows to specify custom line separator of maximum length of 2 characters (because of a restriction in `uniVocity` parser). New option can be used in reading and writing CSV files.\n\n## How was this patch tested?\n\nAdded a few tests with custom `lineSep` for enabled/disabled `multiLine` in read as well as tests in write. Also I added roundtrip tests.\n\nCloses #23080 from MaxGekk/csv-line-sep.\n\nLead-authored-by: Maxim Gekk <max.gekk@gmail.com>\nCo-authored-by: Maxim Gekk <maxim.gekk@databricks.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/MaxGekk/spark-1/commits/8e8d1177e623d5f995fb9ba1d9574675e1e70d56"}]}
{"actor":{"display_login":"MaxGekk"}}