{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22173","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22173/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22173/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22173/events","html_url":"https://github.com/apache/spark/pull/22173","id":352640710,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA5OTAyMjgz","number":22173,"title":"[SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests.","user":{"login":"redsanket","id":8295799,"node_id":"MDQ6VXNlcjgyOTU3OTk=","avatar_url":"https://avatars3.githubusercontent.com/u/8295799?v=4","gravatar_id":"","url":"https://api.github.com/users/redsanket","html_url":"https://github.com/redsanket","followers_url":"https://api.github.com/users/redsanket/followers","following_url":"https://api.github.com/users/redsanket/following{/other_user}","gists_url":"https://api.github.com/users/redsanket/gists{/gist_id}","starred_url":"https://api.github.com/users/redsanket/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redsanket/subscriptions","organizations_url":"https://api.github.com/users/redsanket/orgs","repos_url":"https://api.github.com/users/redsanket/repos","events_url":"https://api.github.com/users/redsanket/events{/privacy}","received_events_url":"https://api.github.com/users/redsanket/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":40,"created_at":"2018-08-21T17:36:26Z","updated_at":"2018-09-20T03:33:41Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22173","html_url":"https://github.com/apache/spark/pull/22173","diff_url":"https://github.com/apache/spark/pull/22173.diff","patch_url":"https://github.com/apache/spark/pull/22173.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nDescription:\r\nRight now, the default server side netty handler threads is 2 * # cores, and can be further configured with parameter spark.shuffle.io.serverThreads.\r\nIn order to process a client request, it would require one available server netty handler thread.\r\nHowever, when the server netty handler threads start to process ChunkFetchRequests, they will be blocked on disk I/O, mostly due to disk contentions from the random read operations initiated by all the ChunkFetchRequests received from clients.\r\nAs a result, when the shuffle server is serving many concurrent ChunkFetchRequests, the server side netty handler threads could all be blocked on reading shuffle files, thus leaving no handler thread available to process other types of requests which should all be very quick to process.\r\n\r\nThis issue could potentially be fixed by limiting the number of netty handler threads that could get blocked when processing ChunkFetchRequest. We have a patch to do this by using a separate EventLoopGroup with a dedicated ChannelHandler to process ChunkFetchRequest. This enables shuffle server to reserve netty handler threads for non-ChunkFetchRequest, thus enabling consistent processing time for these requests which are fast to process. After deploying the patch in our infrastructure, we no longer see timeout issues with either executor registration with local shuffle server or shuffle client establishing connection with remote shuffle server.\r\n(Please fill in changes proposed in this fix)\r\n\r\nFor Original PR please refer here\r\nhttps://github.com/apache/spark/pull/21402\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests and stress testing.\r\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\r\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\r\n\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/423029159","html_url":"https://github.com/apache/spark/pull/22173#issuecomment-423029159","issue_url":"https://api.github.com/repos/apache/spark/issues/22173","id":423029159,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzAyOTE1OQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T03:33:41Z","updated_at":"2018-09-20T03:33:41Z","author_association":"NONE","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}