{"action":"created","comment":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/66076613","id":66076613,"diff_hunk":"@@ -1,11 +1,22 @@\n # This file is a part of Julia. License is MIT: http://julialang.org/license\n \n-type WorkerPool\n+abstract AbstractWorkerPool\n+\n+# An AbstractWorkerPool should implement\n+#\n+# `append!` - add a new worker into the pool","path":"base/workerpool.jl","position":8,"original_position":8,"commit_id":"aa54642ec3bb045cb6b4d6183e2ff92a046517be","original_commit_id":"aa54642ec3bb045cb6b4d6183e2ff92a046517be","user":{"login":"tkelman","id":5934628,"avatar_url":"https://avatars.githubusercontent.com/u/5934628?v=3","gravatar_id":"","url":"https://api.github.com/users/tkelman","html_url":"https://github.com/tkelman","followers_url":"https://api.github.com/users/tkelman/followers","following_url":"https://api.github.com/users/tkelman/following{/other_user}","gists_url":"https://api.github.com/users/tkelman/gists{/gist_id}","starred_url":"https://api.github.com/users/tkelman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkelman/subscriptions","organizations_url":"https://api.github.com/users/tkelman/orgs","repos_url":"https://api.github.com/users/tkelman/repos","events_url":"https://api.github.com/users/tkelman/events{/privacy}","received_events_url":"https://api.github.com/users/tkelman/received_events","type":"User","site_admin":false},"body":"is a worker considered a scalar entity? then `push!` might be more consistent","created_at":"2016-06-07T14:05:54Z","updated_at":"2016-06-07T14:05:54Z","html_url":"https://github.com/JuliaLang/julia/pull/16808#discussion_r66076613","pull_request_url":"https://api.github.com/repos/JuliaLang/julia/pulls/16808","_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/66076613"},"html":{"href":"https://github.com/JuliaLang/julia/pull/16808#discussion_r66076613"},"pull_request":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/16808"}}},"pull_request":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/16808","id":72840250,"html_url":"https://github.com/JuliaLang/julia/pull/16808","diff_url":"https://github.com/JuliaLang/julia/pull/16808.diff","patch_url":"https://github.com/JuliaLang/julia/pull/16808.patch","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/16808","number":16808,"state":"open","locked":false,"title":"RFC: Introduce AbstractWorkerPool, CachingPool and a means of broadcasting variables","user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"body":"This PR does the following:\r\n\r\n- Introduce `AbstractWorkerPool`. `WorkerPool` is an implementation of `AbstractWorkerPool`\r\n- `CachingPool <:  AbstractWorkerPool` provides two functionalities\r\n  1) caches functions executed via `remote`, `remotecall_fetch(f, cp::CachingPool.....)` and others.\r\n      This helps in efficiently executing closures with large closed data multiple times on remote workers.\r\n\r\n  2) implements a `define(wp::CachingPool; kwargs...)` method which broadcasts all the arguments specified onto workers in the pool. These are bound under `Main` with the same name as the keyword arg.\r\n\r\n- `empty!(wp::CachingPool)` releases the closure cache as well as set all previously bound names to `nothing`. We cannot \"unbind\" a name in Julia yet.\r\n\r\nAddresses #10438, #16345 and other issues on easily broadcasting variables.\r\n\r\nExample of large closure:\r\n```\r\nconst wp = Base.CachingPool(workers())\r\n\r\nfunction foo(c, n)\r\n    a = ones(n)\r\n    f = x->sum(a) * x\r\n    t = @elapsed results = pmap(wp, f, 1:c)\r\n    @assert results == map(f, 1:c)\r\n    t\r\nend\r\n\r\nfor coll_sz in [10, 10^2, 10^4]\r\n    for data_sz in [10, 10^4, 10^6]\r\n        println(\"coll_sz:\", coll_sz, \", data_sz:\", data_sz)\r\n        tt = foo(coll_sz, data_sz)\r\n        println(tt)\r\n    end\r\nend\r\n```\r\n\r\nExample of broadcast (from the test file):\r\n```\r\n# CachingPool tests\r\nwp = CachingPool(workers())\r\n\r\n# test variable broadcasting\r\ndefine(wp; wp_foo=1, wp_bar=\"foobar\", wp_baz=ones(10^6))\r\nfor p in workers()\r\n    @test remotecall_fetch(()->wp_foo, p) == 1\r\n    @test remotecall_fetch(()->wp_bar, p) == \"foobar\"\r\n    @test remotecall_fetch(()->wp_baz, p) == ones(10^6)\r\nend\r\n\r\nempty!(wp)\r\n\r\nfor p in workers()\r\n    @test remotecall_fetch(()->wp_foo, p) == nothing\r\n    @test remotecall_fetch(()->wp_bar, p) == nothing\r\n    @test remotecall_fetch(()->wp_baz, p) == nothing\r\nend\r\n```\r\n\r\nDoc updates are pending.","created_at":"2016-06-07T09:12:24Z","updated_at":"2016-06-07T14:05:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"af11f6bea756d06f31eb27210045133e6231d4ab","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/JuliaLang/julia/pulls/16808/commits","review_comments_url":"https://api.github.com/repos/JuliaLang/julia/pulls/16808/comments","review_comment_url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/16808/comments","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/aa54642ec3bb045cb6b4d6183e2ff92a046517be","head":{"label":"JuliaLang:amitm/cachingpool","ref":"amitm/cachingpool","sha":"aa54642ec3bb045cb6b4d6183e2ff92a046517be","user":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"repo":{"id":1644196,"name":"julia","full_name":"JuliaLang/julia","owner":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/JuliaLang/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":false,"url":"https://api.github.com/repos/JuliaLang/julia","forks_url":"https://api.github.com/repos/JuliaLang/julia/forks","keys_url":"https://api.github.com/repos/JuliaLang/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaLang/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaLang/julia/teams","hooks_url":"https://api.github.com/repos/JuliaLang/julia/hooks","issue_events_url":"https://api.github.com/repos/JuliaLang/julia/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaLang/julia/events","assignees_url":"https://api.github.com/repos/JuliaLang/julia/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaLang/julia/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaLang/julia/tags","blobs_url":"https://api.github.com/repos/JuliaLang/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaLang/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaLang/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaLang/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaLang/julia/languages","stargazers_url":"https://api.github.com/repos/JuliaLang/julia/stargazers","contributors_url":"https://api.github.com/repos/JuliaLang/julia/contributors","subscribers_url":"https://api.github.com/repos/JuliaLang/julia/subscribers","subscription_url":"https://api.github.com/repos/JuliaLang/julia/subscription","commits_url":"https://api.github.com/repos/JuliaLang/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaLang/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaLang/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaLang/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaLang/julia/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaLang/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaLang/julia/merges","archive_url":"https://api.github.com/repos/JuliaLang/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaLang/julia/downloads","issues_url":"https://api.github.com/repos/JuliaLang/julia/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaLang/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaLang/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaLang/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaLang/julia/labels{/name}","releases_url":"https://api.github.com/repos/JuliaLang/julia/releases{/id}","deployments_url":"https://api.github.com/repos/JuliaLang/julia/deployments","created_at":"2011-04-21T07:01:50Z","updated_at":"2016-06-07T13:26:26Z","pushed_at":"2016-06-07T13:47:45Z","git_url":"git://github.com/JuliaLang/julia.git","ssh_url":"git@github.com:JuliaLang/julia.git","clone_url":"https://github.com/JuliaLang/julia.git","svn_url":"https://github.com/JuliaLang/julia","homepage":"http://julialang.org/","size":393442,"stargazers_count":7108,"watchers_count":7108,"language":"Julia","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1644,"mirror_url":null,"open_issues_count":1804,"forks":1644,"open_issues":1804,"watchers":7108,"default_branch":"master"}},"base":{"label":"JuliaLang:master","ref":"master","sha":"3142819516da68d5d928bf66456ad633bc53c915","user":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"repo":{"id":1644196,"name":"julia","full_name":"JuliaLang/julia","owner":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/JuliaLang/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":false,"url":"https://api.github.com/repos/JuliaLang/julia","forks_url":"https://api.github.com/repos/JuliaLang/julia/forks","keys_url":"https://api.github.com/repos/JuliaLang/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaLang/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaLang/julia/teams","hooks_url":"https://api.github.com/repos/JuliaLang/julia/hooks","issue_events_url":"https://api.github.com/repos/JuliaLang/julia/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaLang/julia/events","assignees_url":"https://api.github.com/repos/JuliaLang/julia/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaLang/julia/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaLang/julia/tags","blobs_url":"https://api.github.com/repos/JuliaLang/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaLang/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaLang/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaLang/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaLang/julia/languages","stargazers_url":"https://api.github.com/repos/JuliaLang/julia/stargazers","contributors_url":"https://api.github.com/repos/JuliaLang/julia/contributors","subscribers_url":"https://api.github.com/repos/JuliaLang/julia/subscribers","subscription_url":"https://api.github.com/repos/JuliaLang/julia/subscription","commits_url":"https://api.github.com/repos/JuliaLang/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaLang/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaLang/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaLang/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaLang/julia/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaLang/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaLang/julia/merges","archive_url":"https://api.github.com/repos/JuliaLang/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaLang/julia/downloads","issues_url":"https://api.github.com/repos/JuliaLang/julia/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaLang/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaLang/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaLang/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaLang/julia/labels{/name}","releases_url":"https://api.github.com/repos/JuliaLang/julia/releases{/id}","deployments_url":"https://api.github.com/repos/JuliaLang/julia/deployments","created_at":"2011-04-21T07:01:50Z","updated_at":"2016-06-07T13:26:26Z","pushed_at":"2016-06-07T13:47:45Z","git_url":"git://github.com/JuliaLang/julia.git","ssh_url":"git@github.com:JuliaLang/julia.git","clone_url":"https://github.com/JuliaLang/julia.git","svn_url":"https://github.com/JuliaLang/julia","homepage":"http://julialang.org/","size":393442,"stargazers_count":7108,"watchers_count":7108,"language":"Julia","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1644,"mirror_url":null,"open_issues_count":1804,"forks":1644,"open_issues":1804,"watchers":7108,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/16808"},"html":{"href":"https://github.com/JuliaLang/julia/pull/16808"},"issue":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/16808"},"comments":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/16808/comments"},"review_comments":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/16808/comments"},"review_comment":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/16808/commits"},"statuses":{"href":"https://api.github.com/repos/JuliaLang/julia/statuses/aa54642ec3bb045cb6b4d6183e2ff92a046517be"}}}}
{"actor":{"display_login":"tkelman"}}