{"action":"created","issue":{"url":"https://api.github.com/repos/ropensci/drake/issues/366","repository_url":"https://api.github.com/repos/ropensci/drake","labels_url":"https://api.github.com/repos/ropensci/drake/issues/366/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/drake/issues/366/comments","events_url":"https://api.github.com/repos/ropensci/drake/issues/366/events","html_url":"https://github.com/ropensci/drake/issues/366","id":317006667,"number":366,"title":"How to speed up construction of a (very) large plan?","user":{"login":"bmchorse","id":20344778,"avatar_url":"https://avatars1.githubusercontent.com/u/20344778?v=4","gravatar_id":"","url":"https://api.github.com/users/bmchorse","html_url":"https://github.com/bmchorse","followers_url":"https://api.github.com/users/bmchorse/followers","following_url":"https://api.github.com/users/bmchorse/following{/other_user}","gists_url":"https://api.github.com/users/bmchorse/gists{/gist_id}","starred_url":"https://api.github.com/users/bmchorse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmchorse/subscriptions","organizations_url":"https://api.github.com/users/bmchorse/orgs","repos_url":"https://api.github.com/users/bmchorse/repos","events_url":"https://api.github.com/users/bmchorse/events{/privacy}","received_events_url":"https://api.github.com/users/bmchorse/received_events","type":"User","site_admin":false},"labels":[{"id":812913904,"url":"https://api.github.com/repos/ropensci/drake/labels/frequently%20asked%20question","name":"frequently asked question","color":"fbca04","default":false},{"id":794454041,"url":"https://api.github.com/repos/ropensci/drake/labels/topic:%20documentation","name":"topic: documentation","color":"bfd4f2","default":false},{"id":577254680,"url":"https://api.github.com/repos/ropensci/drake/labels/topic:%20efficiency","name":"topic: efficiency","color":"0052cc","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-04-23T22:33:48Z","updated_at":"2018-05-02T20:05:28Z","closed_at":"2018-05-02T03:28:26Z","author_association":"CONTRIBUTOR","body":"I'm tracking an analysis where a single run (of `analysis.slurm`) leads to ~300 output files (`dataset_NUMBER_VARIABLE`, where `NUMBER` is 1-100 replicates and `VARIABLE` is 4 different results files). They're just text files, but there are a lot of them when you multiply over 8 different datasets. \r\n\r\nTo handle the connection, I'm going about it as suggested in the [FAQ about multiple output files](https://github.com/ropensci/drake/issues/257#issuecomment-366379134), using wildcard templating to expand as follows:\r\n\r\n```\r\n# Note that I'm not using drake's HPC abilities to submit to slurm; \r\n# that's a project for another day.  Just trying to connect the in and \r\n# out files so that drake knows there's some dependency happening.\r\n\r\noutfile_plan <- drake_plan(\r\n      c(file_out(\"results/dataset1_NUMBER_VARIABLE\"), \r\n       file_in(\"scripts/analysis1.slurm\")),\r\n      c(file_out(\"results/dataset2_NUMBER_VARIABLE\"), \r\n       file_in(\"scripts/analysis2.slurm\")))  # etc for the 8 datasets\r\n\r\noutplan <- evaluate_plan(outfile_plan, \r\n                             rules=list(NUMBER = rep(1:100), \r\n                                        VARIABLE = c(\"var1.log\", \"var2.log\",\r\n                                                 \"var3.log\", \"summary.txt\")))\r\n```\r\n\r\nAs you can imagine, this expansion takes quite awhile, as it leads to a plan tibble that's 2800 lines long. I think this plan evaluation has to happen every time I run the project, so it's not an insignificant time cost.\r\n\r\nAre there ways to speed this up? A less structured way of doing this could be to simply link each `analysis.slurm` file to only a few output files (say, only files from replicate 1, or only var1.log) but that seems not ideal for reproducibility. "},"comment":{"url":"https://api.github.com/repos/ropensci/drake/issues/comments/386103729","html_url":"https://github.com/ropensci/drake/issues/366#issuecomment-386103729","issue_url":"https://api.github.com/repos/ropensci/drake/issues/366","id":386103729,"user":{"login":"bmchorse","id":20344778,"avatar_url":"https://avatars1.githubusercontent.com/u/20344778?v=4","gravatar_id":"","url":"https://api.github.com/users/bmchorse","html_url":"https://github.com/bmchorse","followers_url":"https://api.github.com/users/bmchorse/followers","following_url":"https://api.github.com/users/bmchorse/following{/other_user}","gists_url":"https://api.github.com/users/bmchorse/gists{/gist_id}","starred_url":"https://api.github.com/users/bmchorse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmchorse/subscriptions","organizations_url":"https://api.github.com/users/bmchorse/orgs","repos_url":"https://api.github.com/users/bmchorse/repos","events_url":"https://api.github.com/users/bmchorse/events{/privacy}","received_events_url":"https://api.github.com/users/bmchorse/received_events","type":"User","site_admin":false},"created_at":"2018-05-02T20:05:28Z","updated_at":"2018-05-02T20:05:28Z","author_association":"CONTRIBUTOR","body":"Looks great!"}}
{"actor":{"display_login":"bmchorse"}}