{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/1665","repository_url":"https://api.github.com/repos/spotify/luigi","labels_url":"https://api.github.com/repos/spotify/luigi/issues/1665/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/1665/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/1665/events","html_url":"https://github.com/spotify/luigi/issues/1665","id":150161451,"number":1665,"title":"SGEJobTask can fail on NFS due to use of shutil.rmtree","user":{"login":"brcopeland","id":13629943,"avatar_url":"https://avatars.githubusercontent.com/u/13629943?v=3","gravatar_id":"","url":"https://api.github.com/users/brcopeland","html_url":"https://github.com/brcopeland","followers_url":"https://api.github.com/users/brcopeland/followers","following_url":"https://api.github.com/users/brcopeland/following{/other_user}","gists_url":"https://api.github.com/users/brcopeland/gists{/gist_id}","starred_url":"https://api.github.com/users/brcopeland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brcopeland/subscriptions","organizations_url":"https://api.github.com/users/brcopeland/orgs","repos_url":"https://api.github.com/users/brcopeland/repos","events_url":"https://api.github.com/users/brcopeland/events{/privacy}","received_events_url":"https://api.github.com/users/brcopeland/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-21T18:57:47Z","updated_at":"2016-06-10T13:16:09Z","closed_at":null,"body":"First off, thank you @jfeala for the sge module.  I'm using luigi.contrib.sge.SGEJobTask to run jobs on my cluster and my temporary drive is on an NFS.  Not uncommonly, I get a traceback of the sort:\r\n\r\nTraceback (most recent call last):\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/site-packages/luigi/worker.py\", line 175, in run\r\n    new_deps = self._run_get_new_deps()\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/site-packages/luigi/worker.py\", line 117, in _run_get_new_deps\r\n    task_gen = self.task.run()\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/site-packages/luigi/contrib/sge.py\", line 215, in run\r\n    self._run_job()\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/site-packages/luigi/contrib/sge.py\", line 265, in _run_job\r\n    shutil.rmtree(self.tmp_dir)\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/shutil.py\", line 247, in rmtree\r\n    rmtree(fullname, ignore_errors, onerror)\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/shutil.py\", line 256, in rmtree\r\n    onerror(os.rmdir, path, sys.exc_info())\r\n  File \"/nfs/goldstein/software/python2.7.7/lib/python2.7/shutil.py\", line 254, in rmtree\r\n    os.rmdir(path)\r\nOSError: [Errno 39] Directory not empty: '/nfs/seqscratch09/bc2675/luigi_test/SimulateGeneBatchData__nfs_central_hom_63__nfs_central_hom_0feed55ade-8fa44d4fc0b45350/luigi'\r\n\r\nI've read that shutil.rmtree can fail on an NFS due to too slow of a response.  I believe that changing line 265, `shutil.rmtree(self.tmp_dir)` to `subprocess.call([\"rm\", \"-rf\", self.tmp_dir])` should yield the same results while alleviating this problem.  This solution is obviously not portable to Windows, so this could be used if an OSError is thrown as a backup."},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/225178958","html_url":"https://github.com/spotify/luigi/issues/1665#issuecomment-225178958","issue_url":"https://api.github.com/repos/spotify/luigi/issues/1665","id":225178958,"user":{"login":"jfeala","id":2364366,"avatar_url":"https://avatars.githubusercontent.com/u/2364366?v=3","gravatar_id":"","url":"https://api.github.com/users/jfeala","html_url":"https://github.com/jfeala","followers_url":"https://api.github.com/users/jfeala/followers","following_url":"https://api.github.com/users/jfeala/following{/other_user}","gists_url":"https://api.github.com/users/jfeala/gists{/gist_id}","starred_url":"https://api.github.com/users/jfeala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jfeala/subscriptions","organizations_url":"https://api.github.com/users/jfeala/orgs","repos_url":"https://api.github.com/users/jfeala/repos","events_url":"https://api.github.com/users/jfeala/events{/privacy}","received_events_url":"https://api.github.com/users/jfeala/received_events","type":"User","site_admin":false},"created_at":"2016-06-10T13:16:09Z","updated_at":"2016-06-10T13:16:09Z","body":"on second thought, since this always has to run on SGE master node, it should never need windows compatibility. Does your proposed fix work? If so, please submit a PR because I've heard of similar issues with NFS race conditions."}}
{"actor":{"display_login":"jfeala"}}