{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/857","repository_url":"https://api.github.com/repos/linkedin/gobblin","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/857/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/857/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/857/events","html_url":"https://github.com/linkedin/gobblin/issues/857","id":141370921,"number":857,"title":"AbstractJobLauncher failing to clean leftover staging data","user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/linkedin/gobblin/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2016-03-16T19:02:35Z","updated_at":"2016-03-24T18:19:28Z","closed_at":null,"body":"Before a job starts to run, `AbstractJobLauncher` attempts to clean left over staging data (which is usually due to the previous run failed) in `cleanLeftoverStagingData()`.\r\n\r\nHowever, since the staging folder path contains the job ID and each run will have a unique job ID, the staging folder it is trying to clean is different from the staging folder the previous run used, which means the leftover staging data is not cleaned.\r\n\r\nTwo possible approaches: (1) change the job ID component in the staging folder path to job Name, so that different runs of the same job will have the same staging folder; (2) change `cleanLeftoverStagingData()` so that it somehow finds the staging folder of the previous run (e.g., using a glob pattern) and clean it."},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/200958328","html_url":"https://github.com/linkedin/gobblin/issues/857#issuecomment-200958328","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/857","id":200958328,"user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"created_at":"2016-03-24T18:19:28Z","updated_at":"2016-03-24T18:19:28Z","body":"@sahilTakiar so the vast majority of use cases shouldn't need to have two runs at the same time. So how about making the task staging directory path configurable with two options: using job id and using job name, the latter of which will ensure different runs have the same staging directory."}}