{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11746","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11746/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11746/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11746/events","html_url":"https://github.com/apache/spark/pull/11746","id":141135520,"number":11746,"title":"[SPARK-13602][CORE] Add shutdown hook to DriverRunner to prevent driver process leak","user":{"login":"BryanCutler","id":4534389,"avatar_url":"https://avatars.githubusercontent.com/u/4534389?v=3","gravatar_id":"","url":"https://api.github.com/users/BryanCutler","html_url":"https://github.com/BryanCutler","followers_url":"https://api.github.com/users/BryanCutler/followers","following_url":"https://api.github.com/users/BryanCutler/following{/other_user}","gists_url":"https://api.github.com/users/BryanCutler/gists{/gist_id}","starred_url":"https://api.github.com/users/BryanCutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BryanCutler/subscriptions","organizations_url":"https://api.github.com/users/BryanCutler/orgs","repos_url":"https://api.github.com/users/BryanCutler/repos","events_url":"https://api.github.com/users/BryanCutler/events{/privacy}","received_events_url":"https://api.github.com/users/BryanCutler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-03-16T00:25:46Z","updated_at":"2016-04-15T21:02:05Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11746","html_url":"https://github.com/apache/spark/pull/11746","diff_url":"https://github.com/apache/spark/pull/11746.diff","patch_url":"https://github.com/apache/spark/pull/11746.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAdded shutdown hook to DriverRunner to kill the driver process in case the Worker JVM exits suddenly and the `WorkerWatcher` was unable to properly catch this.  Did some cleanup to consolidate driver state management and setting of finalized vars within the running thread.\r\n\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit tests to verify that final state and exception variables are set accordingly for successfull, failed, and errors in the driver process.  Retrofitted existing test to verify killing of mocked process ends with the correct state and stops properly\r\n\r\nManually tested (with deploy-mode=cluster) that the shutdown hook is called by forcibly exiting the `Worker` and various points in the code with the `WorkerWatcher` both disabled and enabled.  Also, manually killed the driver through the ui and verified that the `DriverRunner` interrupted, killed the process and exited properly.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/210645244","html_url":"https://github.com/apache/spark/pull/11746#issuecomment-210645244","issue_url":"https://api.github.com/repos/apache/spark/issues/11746","id":210645244,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-04-15T21:02:05Z","updated_at":"2016-04-15T21:02:05Z","body":"**[Test build #55946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55946/consoleFull)** for PR 11746 at commit [`d8effc2`](https://github.com/apache/spark/commit/d8effc2bb2a1c0bc7bc481b2172e5fcc8a56efdc).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}