{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15867","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15867/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15867/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15867/events","html_url":"https://github.com/apache/spark/pull/15867","id":188933324,"number":15867,"title":"[SPARK-18423][Streaming] ReceiverTracker should close checkpoint dir when stopped even if it was not started","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-11-12T20:36:55Z","updated_at":"2016-11-14T14:02:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15867","html_url":"https://github.com/apache/spark/pull/15867","diff_url":"https://github.com/apache/spark/pull/15867.diff","patch_url":"https://github.com/apache/spark/pull/15867.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSeveral tests are being failed on Windows due to the failure of removing the checkpoint dir between each tests.\r\n\r\nThis is caused by not closed file in `ReceiverTracker`. When it is not started, it does not close it even if `stop()` is called.\r\n\r\n```\r\nTest org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery started\r\nTest org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery failed: java.io.IOException: Failed to delete: C:\\projects\\spark\\target\\tmp\\1478983663710-0, took 3.828 sec\r\n    at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)\r\n    at org.apache.spark.util.Utils.deleteRecursively(Utils.scala)\r\n    at org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery(JavaAPISuite.java:1809)\r\n    ...\r\n```\r\n\r\n```\r\n- mapWithState - basic operations with simple API (7 seconds, 640 milliseconds)\r\nException encountered when attempting to run a suite with class name: org.apache.spark.streaming.MapWithStateSuite *** ABORTED *** (12 seconds, 688 milliseconds)\r\n  java.io.IOException: Failed to delete: C:\\projects\\spark\\streaming\\checkpoint\\spark-b8486e2b-6468-4e6f-bb24-88277d2c033c\r\n  ...\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nTests in `JavaAPISuite` and `MapWithStateSuite`.\r\n\r\nManually tested via AppVeyor:\r\n\r\n**Before**\r\n\r\n- `org.apache.spark.streaming.JavaAPISuite`\r\n  Build: https://ci.appveyor.com/project/spark-test/spark/build/71-MapWithStateSuite-1\r\n  Diff: https://github.com/apache/spark/compare/master...spark-test:188c828e682ec45b75d15c3dfc782bcdc8ce024c\r\n\r\n- `org.apache.spark.streaming.MapWithStateSuite`\r\n  Build: https://ci.appveyor.com/project/spark-test/spark/build/72-MapWithStateSuite-1\r\n  Diff: https://github.com/apache/spark/compare/master...spark-test:8f6945d0ccde022a23d3848f6b7fe6da1e7c902e\r\n\r\n**After**\r\n\r\n- `org.apache.spark.streaming.JavaAPISuite`\r\n  Build started: [Streaming] `org.apache.spark.streaming.JavaAPISuite` [![PR-15867](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=3D74F2D5-B0D5-4E1D-874C-685AE694FD37&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/3D74F2D5-B0D5-4E1D-874C-685AE694FD37)\r\n  Diff: https://github.com/apache/spark/compare/master...spark-test:3D74F2D5-B0D5-4E1D-874C-685AE694FD37\r\n\r\n- `org.apache.spark.streaming.MapWithStateSuite`\r\n  Build started: [Streaming] `org.apache.spark.streaming.MapWithStateSuite` [![PR-15867](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=C8E88B64-49F0-4157-9AFA-FC3ACC442351&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/C8E88B64-49F0-4157-9AFA-FC3ACC442351)\r\n  Diff: https://github.com/apache/spark/compare/master...spark-test:C8E88B64-49F0-4157-9AFA-FC3ACC442351"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/260342207","html_url":"https://github.com/apache/spark/pull/15867#issuecomment-260342207","issue_url":"https://api.github.com/repos/apache/spark/issues/15867","id":260342207,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-11-14T14:02:09Z","updated_at":"2016-11-14T14:02:09Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}