{"action":"created","issue":{"url":"https://api.github.com/repos/BD2KGenomics/toil/issues/928","repository_url":"https://api.github.com/repos/BD2KGenomics/toil","labels_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/928/labels{/name}","comments_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/928/comments","events_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/928/events","html_url":"https://github.com/BD2KGenomics/toil/issues/928","id":158335188,"number":928,"title":"slurm currently does not support shared caching","user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-06-03T10:13:26Z","updated_at":"2016-06-03T12:01:20Z","closed_at":null,"body":"Hi all -- this extracts the caching discussion from #918 (cc'ing @arkal) and provides more details so that PR can focus on Slurm.\r\n\r\nRunning cwltoil with Slurm using:\r\n```\r\ncwltoil --batchSystem slurm --jobStore $JOBSTORE --logFile $LOGFILE --workDir $WORKDIR --preserve-environment PATH HOME --no-container run_info-cwl-workflow/main-run_info-cwl.cwl run_info-cwl-workflow/main-run_info-cwl-samples.json\r\n```\r\n(The CWL is from https://s3.amazonaws.com/bcbio/cwl/test_bcbio_cwl.tar.gz but not sure this has anything to do with CWL versus toil cache usage).\r\n\r\nAfter applying the fix from #910 (which masks the real error), we see:\r\n```\r\nnode04: 2016-06-03 10:43:16,323 INFO: toil.lib.bioio: Logging set at level: INFO\r\nnode04: 2016-06-03 10:43:16,323 INFO: toil.lib.bioio: Logging to file: cwltoil.log\r\nnode04: 2016-06-03 10:43:57,411 INFO: toil.lib.bioio: Logging set at level: INFO\r\nnode04: 2016-06-03 10:43:57,412 INFO: toil.lib.bioio: Logging to file: cwltoil.log\r\nnode04: 2016-06-03 10:43:57,476 INFO: toil.jobStores.fileJobStore: Jobstore directory is: /scratch/bchapman/cwl_testing/test_bcbio_cwl/cwltoil_jobstore\r\nnode04: 2016-06-03 10:43:57,477 INFO: toil.jobStores.abstractJobStore: The workflow ID is: 'e93d192a-a3ab-4027-9902-44db0f0837ad'\r\nnode04: 2016-06-03 10:43:57,685 INFO: toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.azureJobStore'\r\nTraceback (most recent call last):\r\n  File \"/scratch/bchapman/cwl_testing/test_bcbio_cwl/../anaconda/bin/cwltoil\", line 9, in <module>\r\n    load_entry_point('toil==3.2.0a2', 'console_scripts', 'cwltoil')()\r\n  File \"/scratch/bchapman/cwl_testing/anaconda/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/cwl/cwltoil.py\", line 595, in main\r\n    outobj = toil.start(wf1)\r\n  File \"/scratch/bchapman/cwl_testing/anaconda/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/common.py\", line 527, in start\r\n    userScript=rootJob.getUserScript())\r\n  File \"/scratch/bchapman/cwl_testing/anaconda/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/common.py\", line 664, in createBatchSystem\r\n    raise RuntimeError('%s currently does not support shared caching.' % config.batchSystem)\r\nRuntimeError: slurm currently does not support shared caching.\r\n```\r\nI used `--disableSharedCache` to work around the problem, but don't know if that's the best fix or what the underlying incompatibility with caching it. Happy for any suggestions. Thanks much."},"comment":{"url":"https://api.github.com/repos/BD2KGenomics/toil/issues/comments/223561036","html_url":"https://github.com/BD2KGenomics/toil/issues/928#issuecomment-223561036","issue_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/928","id":223561036,"user":{"login":"arkal","id":11653944,"avatar_url":"https://avatars.githubusercontent.com/u/11653944?v=3","gravatar_id":"","url":"https://api.github.com/users/arkal","html_url":"https://github.com/arkal","followers_url":"https://api.github.com/users/arkal/followers","following_url":"https://api.github.com/users/arkal/following{/other_user}","gists_url":"https://api.github.com/users/arkal/gists{/gist_id}","starred_url":"https://api.github.com/users/arkal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arkal/subscriptions","organizations_url":"https://api.github.com/users/arkal/orgs","repos_url":"https://api.github.com/users/arkal/repos","events_url":"https://api.github.com/users/arkal/events{/privacy}","received_events_url":"https://api.github.com/users/arkal/received_events","type":"User","site_admin":false},"created_at":"2016-06-03T12:01:20Z","updated_at":"2016-06-03T12:01:20Z","body":"so i'm not too familiar with slurm but looking at this, i can tell you why it's failing. With caching, what happens is Toil creates a directory toil-<workfole id> under --workDir or in the $TMPDIR and that contains the job wrapper temp dirs, and the cache. If you don't have a way of cleaning up the directory on each node,  in your batch system shutdown method, you might need to set one up since that's the only way the cache dir will get cleaned up.\r\n\r\nTo overcome the caching issue, add this to the SlurmBatchSystem Class\r\n```\r\n    @classmethod\r\n    def supportsWorkerCleanup(cls):\r\n        return True```"}}
{"actor":{"display_login":"arkal"}}