{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15481","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15481/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15481/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15481/events","html_url":"https://github.com/apache/spark/pull/15481","id":183004361,"number":15481,"title":"[SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSchedulerBackend reset","user":{"login":"scwf","id":7018048,"avatar_url":"https://avatars.githubusercontent.com/u/7018048?v=3","gravatar_id":"","url":"https://api.github.com/users/scwf","html_url":"https://github.com/scwf","followers_url":"https://api.github.com/users/scwf/followers","following_url":"https://api.github.com/users/scwf/following{/other_user}","gists_url":"https://api.github.com/users/scwf/gists{/gist_id}","starred_url":"https://api.github.com/users/scwf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scwf/subscriptions","organizations_url":"https://api.github.com/users/scwf/orgs","repos_url":"https://api.github.com/users/scwf/repos","events_url":"https://api.github.com/users/scwf/events{/privacy}","received_events_url":"https://api.github.com/users/scwf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-10-14T09:31:10Z","updated_at":"2016-10-19T03:21:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15481","html_url":"https://github.com/apache/spark/pull/15481","diff_url":"https://github.com/apache/spark/pull/15481.diff","patch_url":"https://github.com/apache/spark/pull/15481.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-17929\r\n\r\nNow `CoarseGrainedSchedulerBackend` reset will get the lock,\r\n```\r\n  protected def reset(): Unit = synchronized {\r\n    numPendingExecutors = 0\r\n    executorsPendingToRemove.clear()\r\n\r\n    // Remove all the lingering executors that should be removed but not yet. The reason might be\r\n    // because (1) disconnected event is not yet received; (2) executors die silently.\r\n    executorDataMap.toMap.foreach { case (eid, _) =>\r\n      driverEndpoint.askWithRetry[Boolean](\r\n        RemoveExecutor(eid, SlaveLost(\"Stale executor after cluster manager re-registered.\")))\r\n    }\r\n  }\r\n```\r\n but on removeExecutor also need the lock \"CoarseGrainedSchedulerBackend.this.synchronized\", this will cause deadlock.\r\n\r\n```\r\n   private def removeExecutor(executorId: String, reason: ExecutorLossReason): Unit = {\r\n      logDebug(s\"Asked to remove executor $executorId with reason $reason\")\r\n      executorDataMap.get(executorId) match {\r\n        case Some(executorInfo) =>\r\n          // This must be synchronized because variables mutated\r\n          // in this block are read when requesting executors\r\n          val killed = CoarseGrainedSchedulerBackend.this.synchronized {\r\n            addressToExecutorId -= executorInfo.executorAddress\r\n            executorDataMap -= executorId\r\n            executorsPendingLossReason -= executorId\r\n            executorsPendingToRemove.remove(executorId).getOrElse(false)\r\n          }\r\n     ...\r\n\r\n## How was this patch tested?\r\n\r\nmanual test.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/254700443","html_url":"https://github.com/apache/spark/pull/15481#issuecomment-254700443","issue_url":"https://api.github.com/repos/apache/spark/issues/15481","id":254700443,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-19T03:21:20Z","updated_at":"2016-10-19T03:21:20Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67155/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}