{"action":"created","issue":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1541","repository_url":"https://api.github.com/repos/cooperative-computing-lab/cctools","labels_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1541/labels{/name}","comments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1541/comments","events_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1541/events","html_url":"https://github.com/cooperative-computing-lab/cctools/issues/1541","id":190117315,"number":1541,"title":"Take advantage of shared filesystems with Makeflow","user":{"login":"annawoodard","id":5114833,"avatar_url":"https://avatars.githubusercontent.com/u/5114833?v=3","gravatar_id":"","url":"https://api.github.com/users/annawoodard","html_url":"https://github.com/annawoodard","followers_url":"https://api.github.com/users/annawoodard/followers","following_url":"https://api.github.com/users/annawoodard/following{/other_user}","gists_url":"https://api.github.com/users/annawoodard/gists{/gist_id}","starred_url":"https://api.github.com/users/annawoodard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/annawoodard/subscriptions","organizations_url":"https://api.github.com/users/annawoodard/orgs","repos_url":"https://api.github.com/users/annawoodard/repos","events_url":"https://api.github.com/users/annawoodard/events{/privacy}","received_events_url":"https://api.github.com/users/annawoodard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-17T17:51:51Z","updated_at":"2016-11-17T17:58:32Z","closed_at":null,"body":"This is a feature request. I am running an HEP analysis with Makeflow+WQ+Condor. Part of my workflow runs over a large collection of input files which are several gigabytes each, stored in an HDFS filesystem which is accessible to all machines in our cluster. I would like to be able to tell Makeflow about these files by specifying them as inputs, so I can take advantage of nice Makeflow features like re-running downstream rules if one if its input files are modified. Running over this collection of inputs is an \"embarrassingly parallel\" problem, but parallelization is broken because absolute paths are forbidden-- WQ+condor must be run in transfer mode, so the input files are transferred in series. @dthain suggested you might be able to add a command line switch which would allow me to tell WQ that any file under a given path is globally accessible to the workers and does not need to be transferred. His idea has the advantage that it preserves the status of the Makeflow script as a complete description of the workflow which is independent of the batch system.\r\n\r\nIn case it's helpful, you can find an example Makeflow script for the workflow I'm referring to here: `/afs/crc.nd.edu/user/a/awoodard/www/ttV/6/1/1/Makeflow`. "},"comment":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/comments/261320145","html_url":"https://github.com/cooperative-computing-lab/cctools/issues/1541#issuecomment-261320145","issue_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1541","id":261320145,"user":{"login":"hmeng-19","id":5217421,"avatar_url":"https://avatars.githubusercontent.com/u/5217421?v=3","gravatar_id":"","url":"https://api.github.com/users/hmeng-19","html_url":"https://github.com/hmeng-19","followers_url":"https://api.github.com/users/hmeng-19/followers","following_url":"https://api.github.com/users/hmeng-19/following{/other_user}","gists_url":"https://api.github.com/users/hmeng-19/gists{/gist_id}","starred_url":"https://api.github.com/users/hmeng-19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmeng-19/subscriptions","organizations_url":"https://api.github.com/users/hmeng-19/orgs","repos_url":"https://api.github.com/users/hmeng-19/repos","events_url":"https://api.github.com/users/hmeng-19/events{/privacy}","received_events_url":"https://api.github.com/users/hmeng-19/received_events","type":"User","site_admin":false},"created_at":"2016-11-17T17:58:32Z","updated_at":"2016-11-17T17:58:32Z","body":"@annawoodard , thanks. I will give it a try. "}}
{"actor":{"display_login":"hmeng-19"}}