{"action":"created","issue":{"url":"https://api.github.com/repos/richfitz/remake/issues/84","repository_url":"https://api.github.com/repos/richfitz/remake","labels_url":"https://api.github.com/repos/richfitz/remake/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/richfitz/remake/issues/84/comments","events_url":"https://api.github.com/repos/richfitz/remake/issues/84/events","html_url":"https://github.com/richfitz/remake/issues/84","id":147596657,"number":84,"title":"Parallelise tasks with `parallel`","user":{"login":"dfalster","id":3589590,"avatar_url":"https://avatars.githubusercontent.com/u/3589590?v=3","gravatar_id":"","url":"https://api.github.com/users/dfalster","html_url":"https://github.com/dfalster","followers_url":"https://api.github.com/users/dfalster/followers","following_url":"https://api.github.com/users/dfalster/following{/other_user}","gists_url":"https://api.github.com/users/dfalster/gists{/gist_id}","starred_url":"https://api.github.com/users/dfalster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfalster/subscriptions","organizations_url":"https://api.github.com/users/dfalster/orgs","repos_url":"https://api.github.com/users/dfalster/repos","events_url":"https://api.github.com/users/dfalster/events{/privacy}","received_events_url":"https://api.github.com/users/dfalster/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-04-12T00:31:02Z","updated_at":"2016-06-06T02:19:32Z","closed_at":null,"body":"i know proper parallelisation is something that is a pretty big, longer-term challenge for remake. But I wonder about this as a relatively easily implementation for the use case of iterating over a list. The idea came to me via @kunstler. \r\n\r\nLets's say you have a target of sort\r\n\r\n```\r\nret:\r\n    command: my_wrapper(my_list)\r\n```\r\n\r\nwhere\r\n```\r\nmy_wrapper <- function(input) {\r\n    lapply(input, my_fun)\r\n```\r\n\r\nIf this is slow, we might want to do some parallel compute. At present, we can force (trick) remake into doing some parallel compute using a function like (based on `parallel` package):\r\n\r\n```\r\nmy_wrapper_parallel <- function(input, ncores= detectCores()-2) {\r\n  cl <- makeCluster(ncores)\r\n  on.exit(stopCluster(cl))\r\n  parLapply(input, my_fun)\r\n}\r\n```\r\nThen the remake target would be \r\n\r\n```\r\nret:\r\n    command: my_wrapper_parallel(my_list)\r\n    packages: parallel\r\n```\r\n\r\nSo we can already do this. But then we'll be writing lots of wrapper functions, so why not make it a remake option?\r\n\r\nWould probably need to come as part of a general list target (#8). E.g.\r\n```\r\nret:\r\n    command: remake_list_target(my_list, my_fun)\r\n    parallel: parallel \r\n```\r\nEventually one might addd other (more complicated) backends (like farming out to `AWS`), but starting with `parallel` seems like an at least potentially manageable shorter term goal. \r\n"},"comment":{"url":"https://api.github.com/repos/richfitz/remake/issues/comments/223855564","html_url":"https://github.com/richfitz/remake/issues/84#issuecomment-223855564","issue_url":"https://api.github.com/repos/richfitz/remake/issues/84","id":223855564,"user":{"login":"wlandau","id":1580860,"avatar_url":"https://avatars.githubusercontent.com/u/1580860?v=3","gravatar_id":"","url":"https://api.github.com/users/wlandau","html_url":"https://github.com/wlandau","followers_url":"https://api.github.com/users/wlandau/followers","following_url":"https://api.github.com/users/wlandau/following{/other_user}","gists_url":"https://api.github.com/users/wlandau/gists{/gist_id}","starred_url":"https://api.github.com/users/wlandau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wlandau/subscriptions","organizations_url":"https://api.github.com/users/wlandau/orgs","repos_url":"https://api.github.com/users/wlandau/repos","events_url":"https://api.github.com/users/wlandau/events{/privacy}","received_events_url":"https://api.github.com/users/wlandau/received_events","type":"User","site_admin":false},"created_at":"2016-06-06T02:19:32Z","updated_at":"2016-06-06T02:19:32Z","body":"Both [`parallelRemake`](https://github.com/wlandau/parallelRemake) and [`workflowHelper`](https://github.com/wlandau/workflowHelper) now implement the suggestion by @dfalster on master. That was a quicker update than I thought it would be."}}
{"actor":{"display_login":"wlandau"}}