{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22221","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22221/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22221/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22221/events","html_url":"https://github.com/apache/spark/pull/22221","id":353831448,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEwNzc1NTE1","number":22221,"title":"[SPARK-25231] : Fix synchronization of executor heartbeat receiver in TaskSchedulerImpl","user":{"login":"pgandhi999","id":22228190,"node_id":"MDQ6VXNlcjIyMjI4MTkw","avatar_url":"https://avatars0.githubusercontent.com/u/22228190?v=4","gravatar_id":"","url":"https://api.github.com/users/pgandhi999","html_url":"https://github.com/pgandhi999","followers_url":"https://api.github.com/users/pgandhi999/followers","following_url":"https://api.github.com/users/pgandhi999/following{/other_user}","gists_url":"https://api.github.com/users/pgandhi999/gists{/gist_id}","starred_url":"https://api.github.com/users/pgandhi999/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pgandhi999/subscriptions","organizations_url":"https://api.github.com/users/pgandhi999/orgs","repos_url":"https://api.github.com/users/pgandhi999/repos","events_url":"https://api.github.com/users/pgandhi999/events{/privacy}","received_events_url":"https://api.github.com/users/pgandhi999/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-08-24T15:33:17Z","updated_at":"2018-08-24T20:40:57Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22221","html_url":"https://github.com/apache/spark/pull/22221","diff_url":"https://github.com/apache/spark/pull/22221.diff","patch_url":"https://github.com/apache/spark/pull/22221.patch"},"body":"Running a large Spark job with speculation turned on was causing executor heartbeats to time out on the driver end after sometime and eventually, after hitting the max number of executor failures, the job would fail. \r\n\r\n## What changes were proposed in this pull request?\r\n\r\nThe main reason for the heartbeat timeouts was that the heartbeat-receiver-event-loop-thread was blocked waiting on the TaskSchedulerImpl object which was being held by one of the dispatcher-event-loop threads executing the method dequeueSpeculativeTasks() in TaskSetManager.scala. On further analysis of the heartbeat receiver method, it turns out there is no need to hold the lock on the whole object. The block of code in the method only uses  one global HashMap taskIdToTaskSetManager. Making that map a ConcurrentHashMap, we are ensuring atomicity of operations and speeding up the heartbeat receiver thread operation.\r\n\r\n## How was this patch tested?\r\n\r\nScreenshots of the thread dump have been attached below:\r\n**heartbeat-receiver-event-loop-thread:**\r\n\r\n<img width=\"1409\" alt=\"screen shot 2018-08-24 at 9 19 57 am\" src=\"https://user-images.githubusercontent.com/22228190/44593413-e25df780-a788-11e8-9520-176a18401a59.png\">\r\n\r\n**dispatcher-event-loop-thread:**\r\n\r\n<img width=\"1409\" alt=\"screen shot 2018-08-24 at 9 21 56 am\" src=\"https://user-images.githubusercontent.com/22228190/44593484-13d6c300-a789-11e8-8d88-34b1d51d4541.png\">\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/415876661","html_url":"https://github.com/apache/spark/pull/22221#issuecomment-415876661","issue_url":"https://api.github.com/repos/apache/spark/issues/22221","id":415876661,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTg3NjY2MQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-08-24T20:40:57Z","updated_at":"2018-08-24T20:40:57Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95216/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}