{"action":"created","comment":{"url":"https://api.github.com/repos/stripe/rainier/pulls/comments/192940065","pull_request_review_id":125812493,"id":192940065,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5Mjk0MDA2NQ==","diff_hunk":"@@ -0,0 +1,173 @@\n+package com.stripe.rainier.core\n+\n+import com.stripe.rainier.sampler._\n+import com.stripe.rainier.compute._\n+\n+//implements Simulation-Based Calibration from https://arxiv.org/abs/1804.06788\n+final case class SBC[T](priorGenerators: Seq[Generator[Double]],\n+                        priorParams: Seq[Real],\n+                        posterior: RandomVariable[(Distribution[T], Real)]) {\n+\n+  import SBC._\n+\n+  require(priorParams.forall(_.variables.size == 1))\n+  val priorGenerator = Generator.traverse(priorGenerators)\n+  val emptyEvaluator = new Evaluator(Map.empty)\n+\n+  def prepare(sampler: Sampler, warmupIterations: Int, syntheticSamples: Int)(\n+      implicit rng: RNG): Run = {\n+    val initialThin = 2","path":"rainier-core/src/main/scala/com/stripe/rainier/core/SBC.scala","position":19,"original_position":19,"commit_id":"29ed382a68d3f9ce2811cff711c358008f34661e","original_commit_id":"969ae2aaa905899286d10ac4c7bfe350c220f6ca","user":{"login":"seantalts","id":805356,"node_id":"MDQ6VXNlcjgwNTM1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/805356?v=4","gravatar_id":"","url":"https://api.github.com/users/seantalts","html_url":"https://github.com/seantalts","followers_url":"https://api.github.com/users/seantalts/followers","following_url":"https://api.github.com/users/seantalts/following{/other_user}","gists_url":"https://api.github.com/users/seantalts/gists{/gist_id}","starred_url":"https://api.github.com/users/seantalts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seantalts/subscriptions","organizations_url":"https://api.github.com/users/seantalts/orgs","repos_url":"https://api.github.com/users/seantalts/repos","events_url":"https://api.github.com/users/seantalts/events{/privacy}","received_events_url":"https://api.github.com/users/seantalts/received_events","type":"User","site_admin":false},"body":"You should be able to calculate how much to thin based on the number of effective samples. There's an example in the paper in section 4.","created_at":"2018-06-05T04:01:09Z","updated_at":"2018-06-05T04:01:09Z","html_url":"https://github.com/stripe/rainier/pull/107#discussion_r192940065","pull_request_url":"https://api.github.com/repos/stripe/rainier/pulls/107","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/stripe/rainier/pulls/comments/192940065"},"html":{"href":"https://github.com/stripe/rainier/pull/107#discussion_r192940065"},"pull_request":{"href":"https://api.github.com/repos/stripe/rainier/pulls/107"}},"in_reply_to_id":192809656},"pull_request":{"url":"https://api.github.com/repos/stripe/rainier/pulls/107","id":191892114,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkxODkyMTE0","html_url":"https://github.com/stripe/rainier/pull/107","diff_url":"https://github.com/stripe/rainier/pull/107.diff","patch_url":"https://github.com/stripe/rainier/pull/107.patch","issue_url":"https://api.github.com/repos/stripe/rainier/issues/107","number":107,"state":"open","locked":false,"title":"Simulation-based calibration","user":{"login":"avi-stripe","id":23175169,"node_id":"MDQ6VXNlcjIzMTc1MTY5","avatar_url":"https://avatars1.githubusercontent.com/u/23175169?v=4","gravatar_id":"","url":"https://api.github.com/users/avi-stripe","html_url":"https://github.com/avi-stripe","followers_url":"https://api.github.com/users/avi-stripe/followers","following_url":"https://api.github.com/users/avi-stripe/following{/other_user}","gists_url":"https://api.github.com/users/avi-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/avi-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avi-stripe/subscriptions","organizations_url":"https://api.github.com/users/avi-stripe/orgs","repos_url":"https://api.github.com/users/avi-stripe/repos","events_url":"https://api.github.com/users/avi-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/avi-stripe/received_events","type":"User","site_admin":false},"body":"This implements Simulation-based calibration as described in https://arxiv.org/abs/1804.06788 and http://andrewgelman.com/2018/04/18/better-check-yo-self-wreck-yo-self/ . The API works like this:\r\n\r\n* you construct a `SBC`instance using one of the two `apply` methods on the `SBC` companion object. The univariate form is `SBC(priorDist){x => buildNoiseDist(x)}` where `priorDist` is a `Continuous` distribution, `x` is assumed to be drawn from `priorDist`, and `buildNoiseDist` returns any `Distribution` parameterized by `x`, which will be used first to generate synthetic data and then as a likelihood function for that data.\r\n* you start with the `prepare` method on `SBC` which will return a `SBC.Run` instance after running a single 4-chain diagnostic trace on the model; this lets you inspect the rHat for convergence before proceeding, and inspect how much SBC needs to thin your trace to get a sufficiently large effective sample size for running the calibration\r\n* you can then call `simulate` (for programmatic use) or, better, `animate` (for interactive use) to run the actual calibration, which will run a large number of traces and check the results for uniformity per the referenced paper. Both of these methods take a `logBins` parameter, which must be between 1 and 7, and will produce histograms with 2^logBins bins. The `animate` version will animate the histograms growing in your terminal (which can take quite a long time, so this acts as a kind of progress bar), as well as giving you other diagnostic info like the rHat, and drawing confidence bounds on the histogram bars so you can see  whether the non-uniformity is within reasonable limits.\r\n\r\nTo see this in action, try the `SBCNormal` example.\r\n\r\nThe next step would be to use this from the test cases, but I plan to do that as a separate PR.\r\n","created_at":"2018-05-31T22:40:32Z","updated_at":"2018-06-05T04:01:09Z","closed_at":null,"merged_at":null,"merge_commit_sha":"89e0ebff40fb6d9c1b2dc6ed561bd90206e842b8","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/stripe/rainier/pulls/107/commits","review_comments_url":"https://api.github.com/repos/stripe/rainier/pulls/107/comments","review_comment_url":"https://api.github.com/repos/stripe/rainier/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stripe/rainier/issues/107/comments","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/29ed382a68d3f9ce2811cff711c358008f34661e","head":{"label":"stripe:sbc2","ref":"sbc2","sha":"29ed382a68d3f9ce2811cff711c358008f34661e","user":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":124964747,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQ5NjQ3NDc=","name":"rainier","full_name":"stripe/rainier","owner":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/rainier","description":"Bayesian inference in Scala.","fork":false,"url":"https://api.github.com/repos/stripe/rainier","forks_url":"https://api.github.com/repos/stripe/rainier/forks","keys_url":"https://api.github.com/repos/stripe/rainier/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/rainier/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/rainier/teams","hooks_url":"https://api.github.com/repos/stripe/rainier/hooks","issue_events_url":"https://api.github.com/repos/stripe/rainier/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/rainier/events","assignees_url":"https://api.github.com/repos/stripe/rainier/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/rainier/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/rainier/tags","blobs_url":"https://api.github.com/repos/stripe/rainier/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/rainier/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/rainier/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/rainier/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/rainier/languages","stargazers_url":"https://api.github.com/repos/stripe/rainier/stargazers","contributors_url":"https://api.github.com/repos/stripe/rainier/contributors","subscribers_url":"https://api.github.com/repos/stripe/rainier/subscribers","subscription_url":"https://api.github.com/repos/stripe/rainier/subscription","commits_url":"https://api.github.com/repos/stripe/rainier/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/rainier/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/rainier/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/rainier/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/rainier/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/rainier/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/rainier/merges","archive_url":"https://api.github.com/repos/stripe/rainier/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/rainier/downloads","issues_url":"https://api.github.com/repos/stripe/rainier/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/rainier/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/rainier/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/rainier/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/rainier/labels{/name}","releases_url":"https://api.github.com/repos/stripe/rainier/releases{/id}","deployments_url":"https://api.github.com/repos/stripe/rainier/deployments","created_at":"2018-03-12T23:35:29Z","updated_at":"2018-06-05T03:48:58Z","pushed_at":"2018-06-05T02:49:32Z","git_url":"git://github.com/stripe/rainier.git","ssh_url":"git@github.com:stripe/rainier.git","clone_url":"https://github.com/stripe/rainier.git","svn_url":"https://github.com/stripe/rainier","homepage":"","size":828,"stargazers_count":88,"watchers_count":88,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":13,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":5,"open_issues":13,"watchers":88,"default_branch":"master"}},"base":{"label":"stripe:master","ref":"master","sha":"2295c07e3047dc79c5bcad65ffd020de72f72e02","user":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"repo":{"id":124964747,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQ5NjQ3NDc=","name":"rainier","full_name":"stripe/rainier","owner":{"login":"stripe","id":856813,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg1NjgxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/856813?v=4","gravatar_id":"","url":"https://api.github.com/users/stripe","html_url":"https://github.com/stripe","followers_url":"https://api.github.com/users/stripe/followers","following_url":"https://api.github.com/users/stripe/following{/other_user}","gists_url":"https://api.github.com/users/stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stripe/subscriptions","organizations_url":"https://api.github.com/users/stripe/orgs","repos_url":"https://api.github.com/users/stripe/repos","events_url":"https://api.github.com/users/stripe/events{/privacy}","received_events_url":"https://api.github.com/users/stripe/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stripe/rainier","description":"Bayesian inference in Scala.","fork":false,"url":"https://api.github.com/repos/stripe/rainier","forks_url":"https://api.github.com/repos/stripe/rainier/forks","keys_url":"https://api.github.com/repos/stripe/rainier/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stripe/rainier/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stripe/rainier/teams","hooks_url":"https://api.github.com/repos/stripe/rainier/hooks","issue_events_url":"https://api.github.com/repos/stripe/rainier/issues/events{/number}","events_url":"https://api.github.com/repos/stripe/rainier/events","assignees_url":"https://api.github.com/repos/stripe/rainier/assignees{/user}","branches_url":"https://api.github.com/repos/stripe/rainier/branches{/branch}","tags_url":"https://api.github.com/repos/stripe/rainier/tags","blobs_url":"https://api.github.com/repos/stripe/rainier/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stripe/rainier/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stripe/rainier/git/refs{/sha}","trees_url":"https://api.github.com/repos/stripe/rainier/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stripe/rainier/statuses/{sha}","languages_url":"https://api.github.com/repos/stripe/rainier/languages","stargazers_url":"https://api.github.com/repos/stripe/rainier/stargazers","contributors_url":"https://api.github.com/repos/stripe/rainier/contributors","subscribers_url":"https://api.github.com/repos/stripe/rainier/subscribers","subscription_url":"https://api.github.com/repos/stripe/rainier/subscription","commits_url":"https://api.github.com/repos/stripe/rainier/commits{/sha}","git_commits_url":"https://api.github.com/repos/stripe/rainier/git/commits{/sha}","comments_url":"https://api.github.com/repos/stripe/rainier/comments{/number}","issue_comment_url":"https://api.github.com/repos/stripe/rainier/issues/comments{/number}","contents_url":"https://api.github.com/repos/stripe/rainier/contents/{+path}","compare_url":"https://api.github.com/repos/stripe/rainier/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stripe/rainier/merges","archive_url":"https://api.github.com/repos/stripe/rainier/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stripe/rainier/downloads","issues_url":"https://api.github.com/repos/stripe/rainier/issues{/number}","pulls_url":"https://api.github.com/repos/stripe/rainier/pulls{/number}","milestones_url":"https://api.github.com/repos/stripe/rainier/milestones{/number}","notifications_url":"https://api.github.com/repos/stripe/rainier/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stripe/rainier/labels{/name}","releases_url":"https://api.github.com/repos/stripe/rainier/releases{/id}","deployments_url":"https://api.github.com/repos/stripe/rainier/deployments","created_at":"2018-03-12T23:35:29Z","updated_at":"2018-06-05T03:48:58Z","pushed_at":"2018-06-05T02:49:32Z","git_url":"git://github.com/stripe/rainier.git","ssh_url":"git@github.com:stripe/rainier.git","clone_url":"https://github.com/stripe/rainier.git","svn_url":"https://github.com/stripe/rainier","homepage":"","size":828,"stargazers_count":88,"watchers_count":88,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":13,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":5,"open_issues":13,"watchers":88,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/stripe/rainier/pulls/107"},"html":{"href":"https://github.com/stripe/rainier/pull/107"},"issue":{"href":"https://api.github.com/repos/stripe/rainier/issues/107"},"comments":{"href":"https://api.github.com/repos/stripe/rainier/issues/107/comments"},"review_comments":{"href":"https://api.github.com/repos/stripe/rainier/pulls/107/comments"},"review_comment":{"href":"https://api.github.com/repos/stripe/rainier/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stripe/rainier/pulls/107/commits"},"statuses":{"href":"https://api.github.com/repos/stripe/rainier/statuses/29ed382a68d3f9ce2811cff711c358008f34661e"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"seantalts"}}