{"action":"created","issue":{"url":"https://api.github.com/repos/psolymos/pbapply/issues/9","repository_url":"https://api.github.com/repos/psolymos/pbapply","labels_url":"https://api.github.com/repos/psolymos/pbapply/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/psolymos/pbapply/issues/9/comments","events_url":"https://api.github.com/repos/psolymos/pbapply/issues/9/events","html_url":"https://github.com/psolymos/pbapply/issues/9","id":160435932,"number":9,"title":"Feature request: parallel functions","user":{"login":"kendonB","id":5327505,"avatar_url":"https://avatars.githubusercontent.com/u/5327505?v=3","gravatar_id":"","url":"https://api.github.com/users/kendonB","html_url":"https://github.com/kendonB","followers_url":"https://api.github.com/users/kendonB/followers","following_url":"https://api.github.com/users/kendonB/following{/other_user}","gists_url":"https://api.github.com/users/kendonB/gists{/gist_id}","starred_url":"https://api.github.com/users/kendonB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kendonB/subscriptions","organizations_url":"https://api.github.com/users/kendonB/orgs","repos_url":"https://api.github.com/users/kendonB/repos","events_url":"https://api.github.com/users/kendonB/events{/privacy}","received_events_url":"https://api.github.com/users/kendonB/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/psolymos/pbapply/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-06-15T14:26:55Z","updated_at":"2016-09-08T16:16:29Z","closed_at":null,"body":"Currently, there's no nice way to get progress bars for parallel::*apply functions - the best I'm able to do on Windows is write to a .txt log file from each process, which is cumbersome."},"comment":{"url":"https://api.github.com/repos/psolymos/pbapply/issues/comments/245651896","html_url":"https://github.com/psolymos/pbapply/issues/9#issuecomment-245651896","issue_url":"https://api.github.com/repos/psolymos/pbapply/issues/9","id":245651896,"user":{"login":"psolymos","id":831636,"avatar_url":"https://avatars.githubusercontent.com/u/831636?v=3","gravatar_id":"","url":"https://api.github.com/users/psolymos","html_url":"https://github.com/psolymos","followers_url":"https://api.github.com/users/psolymos/followers","following_url":"https://api.github.com/users/psolymos/following{/other_user}","gists_url":"https://api.github.com/users/psolymos/gists{/gist_id}","starred_url":"https://api.github.com/users/psolymos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/psolymos/subscriptions","organizations_url":"https://api.github.com/users/psolymos/orgs","repos_url":"https://api.github.com/users/psolymos/repos","events_url":"https://api.github.com/users/psolymos/events{/privacy}","received_events_url":"https://api.github.com/users/psolymos/received_events","type":"User","site_admin":false},"created_at":"2016-09-08T16:16:29Z","updated_at":"2016-09-08T16:16:29Z","body":"@kendonB : see my take on a possible solution 9bf861b0dd64aa0b4f51d0a7ad7f3c3ada244738 . The same idea can be carried forward for similar functions. By adding the `cl` argument after `...` I can see that I add the option for parallel processing to `pblapply` instead of having it in a separate function.\r\n\r\nThe main difference relative to what `parallel::parLapply` does is this:\r\n\r\n```\r\n> parallel::splitIndices(10, 4)\r\n[[1]]\r\n[1] 1 2 3\r\n\r\n[[2]]\r\n[1] 4 5\r\n\r\n[[3]]\r\n[1] 6 7\r\n\r\n[[4]]\r\n[1]  8  9 10\r\n\r\n> splitpb(10, 4)\r\n[[1]]\r\n[1] 1 2 3 4\r\n\r\n[[2]]\r\n[1] 5 6 7 8\r\n\r\n[[3]]\r\n[1]  9 10\r\n```\r\n\r\nwhich means that instead of passing the chunks to the workers at once, we do it multiple times while updating the progress bar. This means increased communication overhead between the master and workers, which is a price one has to pay for a progress bar. Currently I can't see any work-around to speed things up even more. See a little example in the commit cited above for timings.\r\n\r\n`mclapply` can be added in a similar manner, even as `cl` defined as an integer. I would rather remove the cluster-auto-detect feature as I find it quite dangerous (e.g. you might have to push objects to the workers anyways due to lack of shared memory in a non-forking situation, but more importantly, setting up RNGs safely cannot be done when the cluster is created AND destroyed within the function).\r\n"}}
{"actor":{"display_login":"psolymos"}}