{"action":"created","comment":{"url":"https://api.github.com/repos/stripe/rainier/pulls/comments/217823979","pull_request_review_id":155638156,"id":217823979,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxNzgyMzk3OQ==","diff_hunk":"@@ -18,6 +18,45 @@ class Target(val real: Real, val placeholders: Map[Variable, Array[Double]]) {\n \n   private def nRows(data: Map[Variable, Array[Double]]): Int =\n     data.head._2.size\n+\n+  def batched(nBatches: Int): (Target, Real) =\n+    if (placeholders.isEmpty)\n+      (this, Real.zero)\n+    else if (nBatches == 1)\n+      (Target.empty, inlined)\n+    else {\n+      val n = nRows(placeholders)\n+      val variables = placeholders.keys.toList\n+      val nColumns = n / nBatches\n+      val copies = 0.until(nColumns).map { i =>\n+        val columnVariables = variables.map { v =>\n+          v -> new Variable\n+        }.toMap\n+        val columnReal = PartialEvaluator.inline(real, columnVariables)\n+        (i, columnReal, columnVariables)\n+      }\n+      val newReal = Real.sum(copies.map(_._2))","path":"rainier-core/src/main/scala/com/stripe/rainier/compute/Target.scala","position":30,"original_position":30,"commit_id":"613082f1fe73ba689e615961fefe7fa3cd506f7f","original_commit_id":"613082f1fe73ba689e615961fefe7fa3cd506f7f","user":{"login":"mio-stripe","id":21321145,"node_id":"MDQ6VXNlcjIxMzIxMTQ1","avatar_url":"https://avatars2.githubusercontent.com/u/21321145?v=4","gravatar_id":"","url":"https://api.github.com/users/mio-stripe","html_url":"https://github.com/mio-stripe","followers_url":"https://api.github.com/users/mio-stripe/followers","following_url":"https://api.github.com/users/mio-stripe/following{/other_user}","gists_url":"https://api.github.com/users/mio-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/mio-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mio-stripe/subscriptions","organizations_url":"https://api.github.com/users/mio-stripe/orgs","repos_url":"https://api.github.com/users/mio-stripe/repos","events_url":"https://api.github.com/users/mio-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/mio-stripe/received_events","type":"User","site_admin":false},"body":"I understand the \"make a copy of the original `real` with the cloned variables substituted for the original ones\" but don't totally get the \"and them sum these to make the `newReal`.\" I get that we want a single `newReal` at the end and don't _really_ see another way to make one, but why is the sum what we want?","created_at":"2018-09-14T19:49:23Z","updated_at":"2018-09-14T19:49:24Z","html_url":"https://github.com/stripe/rainier/pull/204#discussion_r217823979","pull_request_url":"https://api.github.com/repos/stripe/rainier/pulls/204","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/stripe/rainier/pulls/comments/217823979"},"html":{"href":"https://github.com/stripe/rainier/pull/204#discussion_r217823979"},"pull_request":{"href":"https://api.github.com/repos/stripe/rainier/pulls/204"}}},"pull_request":{"url":"https://api.github.com/repos/stripe/rainier/pulls/204","id":215117407,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE1MTE3NDA3","html_url":"https://github.com/stripe/rainier/pull/204","diff_url":"https://github.com/stripe/rainier/pull/204.diff","patch_url":"https://github.com/stripe/rainier/pull/204.patch","issue_url":"https://api.github.com/repos/stripe/rainier/issues/204","number":204,"state":"open","locked":false,"title":"WIP: batched target and context","user":{"login":"avibryant","id":9892,"node_id":"MDQ6VXNlcjk4OTI=","avatar_url":"https://avatars1.githubusercontent.com/u/9892?v=4","gravatar_id":"","url":"https://api.github.com/users/avibryant","html_url":"https://github.com/avibryant","followers_url":"https://api.github.com/users/avibryant/followers","following_url":"https://api.github.com/users/avibryant/following{/other_user}","gists_url":"https://api.github.com/users/avibryant/gists{/gist_id}","starred_url":"https://api.github.com/users/avibryant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avibryant/subscriptions","organizations_url":"https://api.github.com/users/avibryant/orgs","repos_url":"https://api.github.com/users/avibryant/repos","events_url":"https://api.github.com/users/avibryant/events{/privacy}","received_events_url":"https://api.github.com/users/avibryant/received_events","type":"User","site_admin":false},"body":"This is WIP because it doesn't pass tests yet, but @mio-stripe you may still want to start looking over it. Warning: even this description is quite dense.\r\n\r\nThis enables batched evaluation of log-density for simple non-gradient cases like `Walkers`. The major pieces of functionality are:\r\n\r\n* `Target.batched(k)`. Given a `Target` with a `placeholders: Map[Variable, Array[Double]]`, we assume that the `Array[Double]` values all have the same length, `n`. If we divide each of these into `k` batches, we end up with `m = n/k` items per batch. We can represent this by a dense `k x m` matrix, with a \"remainder\" of `n - (k*m)` items that don't fit inside it. We want to ultimately do one evaluation for each row of this matrix (ie each batch); to achieve this, we copy each `Variable` in the original `placeholders` map `m` times, and produce a new copy of the target's `real` that has these cloned `Variable`s substituted for the original ones, and then create a new `Target` whose `real` is the sum of those copies, and whose `placeholders` has the new `Variable` objects and the appropriate matrix columns as its arrays. At the same time, we inline the `remainder` as constants into `real` to produce a fully inlined (no placeholders) version that covers those items.\r\n* `Context.apply(k,targets)`. Given a list of targets and a number of batches, this uses `batched` on all of them to produce a new target with the correct number of batches, as well as a remainder. It can then sum all of the remainders, and, because all of the `Targets` have been normalized to have the same number of items in their placeholder arrays, it can union all of them also, leading to a `Context(base: Real, batched: Target)`.\r\n* `Context.variables` has been modified to look at both the `base` and the `real` inside the batched target, not *not* to consider the variables in the target's placeholders (since we don't want them in the gradient).\r\n* `compileBatched` produces an `Array[Double] => Double` function for the target whose input is just the parameters array (ie the same as `variables`), but that internally compiles a function that takes the `placeholders` as input also, and then divides up the placeholders into batches and evaluates that compiled function once per batch, summing the results and returning that. (This summation is one place we could use kahan summation btw). It's worth noting some potential inefficiency here in that any intermediate values in that compiled function that do *not* depend on the placeholder variables still get recomputed for each batch; a smarter implementation of this which coordinated more deeply with the compiler could avoid that.\r\n","created_at":"2018-09-12T22:45:11Z","updated_at":"2018-09-14T19:49:24Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cb2d329cf36f8363715d0696d757dadde2e2cf94","assignee":{"login":"mio-stripe","id":21321145,"node_id":"MDQ6VXNlcjIxMzIxMTQ1","avatar_url":"https://avatars2.githubusercontent.com/u/21321145?v=4","gravatar_id":"","url":"https://api.github.com/users/mio-stripe","html_url":"https://github.com/mio-stripe","followers_url":"https://api.github.com/users/mio-stripe/followers","following_url":"https://api.github.com/users/mio-stripe/following{/other_user}","gists_url":"https://api.github.com/users/mio-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/mio-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mio-stripe/subscriptions","organizations_url":"https://api.github.com/users/mio-stripe/orgs","repos_url":"https://api.github.com/users/mio-stripe/repos","events_url":"https://api.github.com/users/mio-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/mio-stripe/received_events","type":"User","site_admin":false},"assignees":[{"login":"mio-stripe","id":21321145,"node_id":"MDQ6VXNlcjIxMzIxMTQ1","avatar_url":"https://avatars2.githubusercontent.com/u/21321145?v=4","gravatar_id":"","url":"https://api.github.com/users/mio-stripe","html_url":"https://github.com/mio-stripe","followers_url":"https://api.github.com/users/mio-stripe/followers","following_url":"https://api.github.com/users/mio-stripe/following{/other_user}","gists_url":"https://api.github.com/users/mio-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/mio-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mio-stripe/subscriptions","organizations_url":"https://api.github.com/users/mio-stripe/orgs","repos_url":"https://api.github.com/users/mio-stripe/repos","events_url":"https://api.github.com/users/mio-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/mio-stripe/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/stripe/rainier/pulls/204/commits","review_comments_url":"https://api.github.com/repos/stripe/rainier/pulls/204/comments","review_comment_url":"https://api.github.com/repos/stripe/rainier/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stripe/rainier/issues/204/comments","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/613082f1fe73ba689e615961fefe7fa3cd506f7f","head":{"label":"stripe:scalewalkers","ref":"scalewalkers","sha":"613082f1fe73ba689e615961fefe7fa3cd506f7f","user":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":124964747,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQ5NjQ3NDc=","name":"rainier","full_name":"stripe/rainier","owner":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/rainier","description":"Bayesian inference in Scala.","fork":false,"url":"https://api.github.com/repos/stripe/rainier","forks_url":"https://api.github.com/repos/stripe/rainier/forks","keys_url":"https://api.github.com/repos/stripe/rainier/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/rainier/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/rainier/teams","hooks_url":"https://api.github.com/repos/stripe/rainier/hooks","issue_events_url":"https://api.github.com/repos/stripe/rainier/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/rainier/events","assignees_url":"https://api.github.com/repos/stripe/rainier/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/rainier/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/rainier/tags","blobs_url":"https://api.github.com/repos/stripe/rainier/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/rainier/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/rainier/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/rainier/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/rainier/languages","stargazers_url":"https://api.github.com/repos/stripe/rainier/stargazers","contributors_url":"https://api.github.com/repos/stripe/rainier/contributors","subscribers_url":"https://api.github.com/repos/stripe/rainier/subscribers","subscription_url":"https://api.github.com/repos/stripe/rainier/subscription","commits_url":"https://api.github.com/repos/stripe/rainier/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/rainier/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/rainier/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/rainier/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/rainier/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/rainier/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/rainier/merges","archive_url":"https://api.github.com/repos/stripe/rainier/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/rainier/downloads","issues_url":"https://api.github.com/repos/stripe/rainier/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/rainier/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/rainier/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/rainier/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/rainier/labels{/name}","releases_url":"https://api.github.com/repos/stripe/rainier/releases{/id}","deployments_url":"https://api.github.com/repos/stripe/rainier/deployments","created_at":"2018-03-12T23:35:29Z","updated_at":"2018-09-14T07:12:21Z","pushed_at":"2018-09-13T16:12:55Z","git_url":"git://github.com/stripe/rainier.git","ssh_url":"git@github.com:stripe/rainier.git","clone_url":"https://github.com/stripe/rainier.git","svn_url":"https://github.com/stripe/rainier","homepage":"","size":1204,"stargazers_count":173,"watchers_count":173,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13,"mirror_url":null,"archived":false,"open_issues_count":26,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":13,"open_issues":26,"watchers":173,"default_branch":"develop"}},"base":{"label":"stripe:develop","ref":"develop","sha":"f9a0194500c26f4b5caea69064e7b56f8c7c3466","user":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":124964747,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQ5NjQ3NDc=","name":"rainier","full_name":"stripe/rainier","owner":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/rainier","description":"Bayesian inference in Scala.","fork":false,"url":"https://api.github.com/repos/stripe/rainier","forks_url":"https://api.github.com/repos/stripe/rainier/forks","keys_url":"https://api.github.com/repos/stripe/rainier/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/rainier/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/rainier/teams","hooks_url":"https://api.github.com/repos/stripe/rainier/hooks","issue_events_url":"https://api.github.com/repos/stripe/rainier/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/rainier/events","assignees_url":"https://api.github.com/repos/stripe/rainier/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/rainier/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/rainier/tags","blobs_url":"https://api.github.com/repos/stripe/rainier/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/rainier/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/rainier/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/rainier/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/rainier/languages","stargazers_url":"https://api.github.com/repos/stripe/rainier/stargazers","contributors_url":"https://api.github.com/repos/stripe/rainier/contributors","subscribers_url":"https://api.github.com/repos/stripe/rainier/subscribers","subscription_url":"https://api.github.com/repos/stripe/rainier/subscription","commits_url":"https://api.github.com/repos/stripe/rainier/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/rainier/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/rainier/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/rainier/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/rainier/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/rainier/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/rainier/merges","archive_url":"https://api.github.com/repos/stripe/rainier/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/rainier/downloads","issues_url":"https://api.github.com/repos/stripe/rainier/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/rainier/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/rainier/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/rainier/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/rainier/labels{/name}","releases_url":"https://api.github.com/repos/stripe/rainier/releases{/id}","deployments_url":"https://api.github.com/repos/stripe/rainier/deployments","created_at":"2018-03-12T23:35:29Z","updated_at":"2018-09-14T07:12:21Z","pushed_at":"2018-09-13T16:12:55Z","git_url":"git://github.com/stripe/rainier.git","ssh_url":"git@github.com:stripe/rainier.git","clone_url":"https://github.com/stripe/rainier.git","svn_url":"https://github.com/stripe/rainier","homepage":"","size":1204,"stargazers_count":173,"watchers_count":173,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13,"mirror_url":null,"archived":false,"open_issues_count":26,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":13,"open_issues":26,"watchers":173,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/stripe/rainier/pulls/204"},"html":{"href":"https://github.com/stripe/rainier/pull/204"},"issue":{"href":"https://api.github.com/repos/stripe/rainier/issues/204"},"comments":{"href":"https://api.github.com/repos/stripe/rainier/issues/204/comments"},"review_comments":{"href":"https://api.github.com/repos/stripe/rainier/pulls/204/comments"},"review_comment":{"href":"https://api.github.com/repos/stripe/rainier/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stripe/rainier/pulls/204/commits"},"statuses":{"href":"https://api.github.com/repos/stripe/rainier/statuses/613082f1fe73ba689e615961fefe7fa3cd506f7f"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"mio-stripe"}}