{"action":"created","comment":{"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments/238625909","pull_request_review_id":181211035,"id":238625909,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzODYyNTkwOQ==","diff_hunk":"@@ -0,0 +1,334 @@\n+---\n+layout: post\n+title:  \"Monte Carlo Power Analysis\"\n+authors:\n+  - \"Daniel Nee\"\n+  - \"Jamie Edgecombe\"\n+  - \"Jared Conway\"\n+excerpt: >\n+  Take advantage of computing power and empirical data to use Monte Carlo simulation to perform experiment power analysis.\n+---\n+<script type=\"text/javascript\" async\n+  src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML\">\n+</script>\n+\n+Experimentation is a key part of shipping new features here at Deliveroo. Each of our product teams has an embedded data scientist to help design the experiments and analyse the results. When we have a new feature to ship, we will design an experiment to test if it improves a small number of pertinent metrics. We will need to decide what metrics we will be tracking and how we define success of an experiment.\n+\n+Prior to starting an experiment, the data scientist will have to ask a number of questions:\n+1. What is the minimum effect size (change in our metric) that we want to be able to detect?\n+2. What sample size do we need in order to have a good chance of detecting this minimum effect size?\n+\n+Obtaining an answer to the second question is called Power Analysis. Deciding the minimum effect size is often a business judgement and is an input into Power Analysis. Usually we only have to answer one of these questions: we choose the minimum effect size and need to calculate the sample size (duration of the experiment) or we have a fixed sample size and observe the minimum effect size that we expect would give us a significant result. While there are a number of [online](http://www.evanmiller.org/ab-testing/sample-size.html) calculators and [analytical methods](https://www.statsmodels.org/stable/stats.html#power-and-sample-size-calculations) (closed-form) to help us do this, we find using Monte Carlo simulation is far more flexible when running experiments with different designs and potentially multiple metrics. In this blog post we will talk through how and why to use Monte Carlo simulation for power analysis and briefly introduce our internal experimentation framework Echidna.\n+\n+## What is statistical power?\n+\n+We use frequentist hypothesis testing approaches to analyse the impact of any experiment. The statistical power of an experiment for a given alternative hypothesis is the probability we will get a statistically significant result (reject the null hypothesis) when the alternative is true for any one realisation of the experiment:\n+\n+$$power = 1 - \\beta = P(\\textrm{Reject }H_0|H_1\\textrm{ is true})$$","path":"_posts/2018-11-29-monte-carlo-power-analysis.md","position":27,"original_position":27,"commit_id":"6c5a82e4f1f09b66e5dc9aefcab719473be1dd69","original_commit_id":"6c5a82e4f1f09b66e5dc9aefcab719473be1dd69","user":{"login":"marianeum","id":1529791,"node_id":"MDQ6VXNlcjE1Mjk3OTE=","avatar_url":"https://avatars0.githubusercontent.com/u/1529791?v=4","gravatar_id":"","url":"https://api.github.com/users/marianeum","html_url":"https://github.com/marianeum","followers_url":"https://api.github.com/users/marianeum/followers","following_url":"https://api.github.com/users/marianeum/following{/other_user}","gists_url":"https://api.github.com/users/marianeum/gists{/gist_id}","starred_url":"https://api.github.com/users/marianeum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marianeum/subscriptions","organizations_url":"https://api.github.com/users/marianeum/orgs","repos_url":"https://api.github.com/users/marianeum/repos","events_url":"https://api.github.com/users/marianeum/events{/privacy}","received_events_url":"https://api.github.com/users/marianeum/received_events","type":"User","site_admin":false},"body":"maybe the formulas could be rendered as code? e.g. \r\n```\r\n$$power = 1 - \\beta = P(\\textrm{Reject }H_0|H_1\\textrm{ is true})$$\r\n```\r\n\r\n(use triple back tips around the code)","created_at":"2018-12-04T11:34:25Z","updated_at":"2018-12-04T11:40:32Z","html_url":"https://github.com/deliveroo/deliveroo.engineering/pull/191#discussion_r238625909","pull_request_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments/238625909"},"html":{"href":"https://github.com/deliveroo/deliveroo.engineering/pull/191#discussion_r238625909"},"pull_request":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191"}}},"pull_request":{"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191","id":234568321,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM0NTY4MzIx","html_url":"https://github.com/deliveroo/deliveroo.engineering/pull/191","diff_url":"https://github.com/deliveroo/deliveroo.engineering/pull/191.diff","patch_url":"https://github.com/deliveroo/deliveroo.engineering/pull/191.patch","issue_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/191","number":191,"state":"open","locked":false,"title":"Blog posts/monte carlo power analysis","user":{"login":"danielnee","id":1815441,"node_id":"MDQ6VXNlcjE4MTU0NDE=","avatar_url":"https://avatars1.githubusercontent.com/u/1815441?v=4","gravatar_id":"","url":"https://api.github.com/users/danielnee","html_url":"https://github.com/danielnee","followers_url":"https://api.github.com/users/danielnee/followers","following_url":"https://api.github.com/users/danielnee/following{/other_user}","gists_url":"https://api.github.com/users/danielnee/gists{/gist_id}","starred_url":"https://api.github.com/users/danielnee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielnee/subscriptions","organizations_url":"https://api.github.com/users/danielnee/orgs","repos_url":"https://api.github.com/users/danielnee/repos","events_url":"https://api.github.com/users/danielnee/events{/privacy}","received_events_url":"https://api.github.com/users/danielnee/received_events","type":"User","site_admin":false},"body":"New blog post by the Data Science team. \r\n\r\nOnly thing I've noticed is that is doesn't like display three authors. At the top, it doesn't add a comma between 1st and 2nd author. At the bottom the 3rd author's 'About' is not aligned like the rest.\r\n\r\n\r\n![screenshot 2018-11-29 at 09 39 36](https://user-images.githubusercontent.com/1815441/49212810-d3f02f00-f3ba-11e8-9e40-b700b3880b67.png)\r\n![screenshot 2018-11-29 at 09 39 54](https://user-images.githubusercontent.com/1815441/49212813-d3f02f00-f3ba-11e8-91f1-3617e331477d.png)\r\n\r\n","created_at":"2018-11-29T09:42:17Z","updated_at":"2018-12-04T11:40:32Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e1e5932f954d17b388e89fcea4cc8634cf3733c5","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":411178414,"node_id":"MDU6TGFiZWw0MTExNzg0MTQ=","url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/labels/ready%20for%20editor%20review","name":"ready for editor review","color":"5319e7","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191/commits","review_comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191/comments","review_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments{/number}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/191/comments","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/6c5a82e4f1f09b66e5dc9aefcab719473be1dd69","head":{"label":"deliveroo:blog-posts/monte-carlo-power-analysis","ref":"blog-posts/monte-carlo-power-analysis","sha":"6c5a82e4f1f09b66e5dc9aefcab719473be1dd69","user":{"login":"deliveroo","id":10514109,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNTE0MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/10514109?v=4","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"repo":{"id":32532933,"node_id":"MDEwOlJlcG9zaXRvcnkzMjUzMjkzMw==","name":"deliveroo.engineering","full_name":"deliveroo/deliveroo.engineering","private":false,"owner":{"login":"deliveroo","id":10514109,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNTE0MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/10514109?v=4","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/deliveroo/deliveroo.engineering","description":"Guidelines and conventions for good software at Deliveroo","fork":false,"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering","forks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/forks","keys_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/teams","hooks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/hooks","issue_events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/events{/number}","events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/events","assignees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/assignees{/user}","branches_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/branches{/branch}","tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/tags","blobs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/refs{/sha}","trees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/{sha}","languages_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/languages","stargazers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/stargazers","contributors_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contributors","subscribers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscribers","subscription_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscription","commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/commits{/sha}","git_commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/commits{/sha}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/comments{/number}","issue_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/comments{/number}","contents_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contents/{+path}","compare_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/merges","archive_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/downloads","issues_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues{/number}","pulls_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls{/number}","milestones_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/milestones{/number}","notifications_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/labels{/name}","releases_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/releases{/id}","deployments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/deployments","created_at":"2015-03-19T16:23:28Z","updated_at":"2018-12-03T10:08:11Z","pushed_at":"2018-12-04T10:09:09Z","git_url":"git://github.com/deliveroo/deliveroo.engineering.git","ssh_url":"git@github.com:deliveroo/deliveroo.engineering.git","clone_url":"https://github.com/deliveroo/deliveroo.engineering.git","svn_url":"https://github.com/deliveroo/deliveroo.engineering","homepage":"","size":46775,"stargazers_count":32,"watchers_count":32,"language":"CSS","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":null,"forks":12,"open_issues":17,"watchers":32,"default_branch":"gh-pages"}},"base":{"label":"deliveroo:gh-pages","ref":"gh-pages","sha":"684458468615fec2aba4212faf38f068d69e1f08","user":{"login":"deliveroo","id":10514109,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNTE0MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/10514109?v=4","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"repo":{"id":32532933,"node_id":"MDEwOlJlcG9zaXRvcnkzMjUzMjkzMw==","name":"deliveroo.engineering","full_name":"deliveroo/deliveroo.engineering","private":false,"owner":{"login":"deliveroo","id":10514109,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNTE0MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/10514109?v=4","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/deliveroo/deliveroo.engineering","description":"Guidelines and conventions for good software at Deliveroo","fork":false,"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering","forks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/forks","keys_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/teams","hooks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/hooks","issue_events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/events{/number}","events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/events","assignees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/assignees{/user}","branches_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/branches{/branch}","tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/tags","blobs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/refs{/sha}","trees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/{sha}","languages_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/languages","stargazers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/stargazers","contributors_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contributors","subscribers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscribers","subscription_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscription","commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/commits{/sha}","git_commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/commits{/sha}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/comments{/number}","issue_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/comments{/number}","contents_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contents/{+path}","compare_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/merges","archive_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/downloads","issues_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues{/number}","pulls_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls{/number}","milestones_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/milestones{/number}","notifications_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/labels{/name}","releases_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/releases{/id}","deployments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/deployments","created_at":"2015-03-19T16:23:28Z","updated_at":"2018-12-03T10:08:11Z","pushed_at":"2018-12-04T10:09:09Z","git_url":"git://github.com/deliveroo/deliveroo.engineering.git","ssh_url":"git@github.com:deliveroo/deliveroo.engineering.git","clone_url":"https://github.com/deliveroo/deliveroo.engineering.git","svn_url":"https://github.com/deliveroo/deliveroo.engineering","homepage":"","size":46775,"stargazers_count":32,"watchers_count":32,"language":"CSS","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":null,"forks":12,"open_issues":17,"watchers":32,"default_branch":"gh-pages"}},"_links":{"self":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191"},"html":{"href":"https://github.com/deliveroo/deliveroo.engineering/pull/191"},"issue":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/191"},"comments":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/191/comments"},"review_comments":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191/comments"},"review_comment":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/191/commits"},"statuses":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/6c5a82e4f1f09b66e5dc9aefcab719473be1dd69"}},"author_association":"NONE"}}
{"actor":{"display_login":"marianeum"}}