{"push_id":1362256480,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"3fbf5a58c236fc5d5fee39cb29e7f5c7e01c0ee7","before":"7a531e3054f8d4820216ed379433559f57f571b8","commits":[{"sha":"c1f344f1a09b8834bec70c1ece30b9bff63e55ea","author":{"name":"w00228970","email":"b7f66cd45f78ebd8439b28a88058866c6c775d1c@huawei.com"},"message":"[SPARK-17929][CORE] Fix deadlock when CoarseGrainedSchedulerBackend reset\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-17929\n\nNow `CoarseGrainedSchedulerBackend` reset will get the lock,\n```\n  protected def reset(): Unit = synchronized {\n    numPendingExecutors = 0\n    executorsPendingToRemove.clear()\n\n    // Remove all the lingering executors that should be removed but not yet. The reason might be\n    // because (1) disconnected event is not yet received; (2) executors die silently.\n    executorDataMap.toMap.foreach { case (eid, _) =>\n      driverEndpoint.askWithRetry[Boolean](\n        RemoveExecutor(eid, SlaveLost(\"Stale executor after cluster manager re-registered.\")))\n    }\n  }\n```\n but on removeExecutor also need the lock \"CoarseGrainedSchedulerBackend.this.synchronized\", this will cause deadlock.\n\n```\n   private def removeExecutor(executorId: String, reason: ExecutorLossReason): Unit = {\n      logDebug(s\"Asked to remove executor $executorId with reason $reason\")\n      executorDataMap.get(executorId) match {\n        case Some(executorInfo) =>\n          // This must be synchronized because variables mutated\n          // in this block are read when requesting executors\n          val killed = CoarseGrainedSchedulerBackend.this.synchronized {\n            addressToExecutorId -= executorInfo.executorAddress\n            executorDataMap -= executorId\n            executorsPendingLossReason -= executorId\n            executorsPendingToRemove.remove(executorId).getOrElse(false)\n          }\n     ...\n\n## How was this patch tested?\n\nmanual test.\n\nAuthor: w00228970 <wangfei1@huawei.com>\n\nCloses #15481 from scwf/spark-17929.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/c1f344f1a09b8834bec70c1ece30b9bff63e55ea"},{"sha":"140570252fd3739d6bdcadd6d4d5a180e480d3e0","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18044][STREAMING] FileStreamSource should not infer partitions in every batch\n\n## What changes were proposed in this pull request?\n\nIn `FileStreamSource.getBatch`, we will create a `DataSource` with specified schema, to avoid inferring the schema again and again. However, we don't pass the partition columns, and will infer the partition again and again.\n\nThis PR fixes it by keeping the partition columns in `FileStreamSource`, like schema.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15581 from cloud-fan/stream.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/140570252fd3739d6bdcadd6d4d5a180e480d3e0"},{"sha":"268ccb9a48dfefc4d7bc85155e7e20a2dfe89307","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-17812][SQL][KAFKA] Assign and specific startingOffsets for structured stream\n\n## What changes were proposed in this pull request?\n\nstartingOffsets takes specific per-topicpartition offsets as a json argument, usable with any consumer strategy\n\nassign with specific topicpartitions as a consumer strategy\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #15504 from koeninger/SPARK-17812.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/268ccb9a48dfefc4d7bc85155e7e20a2dfe89307"},{"sha":"c9720b2195a465653690b3e221ce789142217b0d","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[STREAMING][KAFKA][DOC] clarify kafka settings needed for larger batches\n\n## What changes were proposed in this pull request?\n\nMinor doc change to mention kafka configuration for larger spark batches.\n\n## How was this patch tested?\n\nDoc change only, confirmed via jekyll.\n\nThe configuration issue was discussed / confirmed with users on the mailing list.\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #15570 from koeninger/kafka-doc-heartbeat.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/c9720b2195a465653690b3e221ce789142217b0d"},{"sha":"3fbf5a58c236fc5d5fee39cb29e7f5c7e01c0ee7","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18042][SQL] OutputWriter should expose file path written\n\n## What changes were proposed in this pull request?\nThis patch adds a new \"path\" method on OutputWriter that returns the path of the file written by the OutputWriter. This is part of the necessary work to consolidate structured streaming and batch write paths.\n\nThe batch write path has a nice feature that each data source can define the extension of the files, and allow Spark to specify the staging directory and the prefix for the files. However, in the streaming path we need to collect the list of files written, and there is no interface right now to do that.\n\n## How was this patch tested?\nN/A - there is no behavior change and this should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15580 from rxin/SPARK-18042.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3fbf5a58c236fc5d5fee39cb29e7f5c7e01c0ee7"}]}
{"actor":{"display_login":"ueshin"}}