{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20998","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20998/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20998/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20998/events","html_url":"https://github.com/apache/spark/pull/20998","id":312030979,"number":20998,"title":"[SPARK-23888][CORE] speculative task should not run on a given host where another attempt is already running on","user":{"login":"Ngone51","id":16397174,"avatar_url":"https://avatars1.githubusercontent.com/u/16397174?v=4","gravatar_id":"","url":"https://api.github.com/users/Ngone51","html_url":"https://github.com/Ngone51","followers_url":"https://api.github.com/users/Ngone51/followers","following_url":"https://api.github.com/users/Ngone51/following{/other_user}","gists_url":"https://api.github.com/users/Ngone51/gists{/gist_id}","starred_url":"https://api.github.com/users/Ngone51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ngone51/subscriptions","organizations_url":"https://api.github.com/users/Ngone51/orgs","repos_url":"https://api.github.com/users/Ngone51/repos","events_url":"https://api.github.com/users/Ngone51/events{/privacy}","received_events_url":"https://api.github.com/users/Ngone51/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-06T16:02:05Z","updated_at":"2018-04-06T16:02:39Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20998","html_url":"https://github.com/apache/spark/pull/20998","diff_url":"https://github.com/apache/spark/pull/20998.diff","patch_url":"https://github.com/apache/spark/pull/20998.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThere's a bug in:\r\n```\r\n/** Check whether a task is currently running an attempt on a given host */\r\n private def hasAttemptOnHost(taskIndex: Int, host: String): Boolean = {\r\n   taskAttempts(taskIndex).exists(_.host == host)\r\n }\r\n```\r\nThis will ignore hosts which only have finished attempts, so we should check whether the attempt is currently running on the given host. \r\n\r\nAnd it is possible for a speculative task to run on a host where another attempt failed here before.\r\nAssume we have only two machines: host1, host2.  We first run task0.0 on host1. Then, due to  a long time waiting for task0.0, we launch a speculative task0.1 on host2. And, task0.1 finally failed on host1, but it can not re-run since there's already  a copy running on host2. After another long time waiting, we launch a new  speculative task0.2. And, now, we can run task0.2 on host1 again, since there's no more running attempt on host1.\r\n\r\n## How was this patch tested?\r\n\r\nAdded."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/379299235","html_url":"https://github.com/apache/spark/pull/20998#issuecomment-379299235","issue_url":"https://api.github.com/repos/apache/spark/issues/20998","id":379299235,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-04-06T16:02:39Z","updated_at":"2018-04-06T16:02:39Z","author_association":"NONE","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}