{"action":"created","comment":{"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments/71485794","id":71485794,"diff_hunk":"@@ -0,0 +1,352 @@\n+---\n+layout: post\n+title:  \"Running A/B tests on our hosting infrastructure\"\n+date:   2016-07-13 00:00:00\n+exerpt: >\n+  Split testing is a cornerstone of how we improve our products. While we\n+  usually run such tests for user-visible interface changes, this is an example\n+  of running a (successful!) multivariate test between CDNs.\n+\n+---\n+\n+TL,DR:\n+\n+- Fastly is generally more performant than AWS Cloudfront — for our userbase.\n+- A/B testing CDNs is easy (with out toolset).\n+- Synthetic performance checks (e.g. with Pingdom) do not reflect end user\n+  performance experience.\n+\n+We split tested Cloudfront vs. Fastly across 1,000,000 users, in 12\n+countries, on 3 continents, over a significant time period. Using Fastly\n+improved page load times by up to 50% and site conversion by up to a whopping\n+3.9% (with _p_ < 0.01). Read on for the caveats!","path":"_posts/2016-07-13-split-testing-cdns.md","position":22,"original_position":22,"commit_id":"8a743ab19b326152843947c43edbccc6aa63b50c","original_commit_id":"8a743ab19b326152843947c43edbccc6aa63b50c","user":{"login":"mezis","id":231698,"avatar_url":"https://avatars.githubusercontent.com/u/231698?v=3","gravatar_id":"","url":"https://api.github.com/users/mezis","html_url":"https://github.com/mezis","followers_url":"https://api.github.com/users/mezis/followers","following_url":"https://api.github.com/users/mezis/following{/other_user}","gists_url":"https://api.github.com/users/mezis/gists{/gist_id}","starred_url":"https://api.github.com/users/mezis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mezis/subscriptions","organizations_url":"https://api.github.com/users/mezis/orgs","repos_url":"https://api.github.com/users/mezis/repos","events_url":"https://api.github.com/users/mezis/events{/privacy}","received_events_url":"https://api.github.com/users/mezis/received_events","type":"User","site_admin":false},"body":"Actually, the performance improvement is pretty massive locally, and very good globally.\r\nIt's on _conversion_ that we don't see the expected improvements.\r\n\r\nI'll try to rephrase a bit, see if I can convince you.","created_at":"2016-07-20T08:38:52Z","updated_at":"2016-07-20T08:38:52Z","html_url":"https://github.com/deliveroo/deliveroo.engineering/pull/8#discussion_r71485794","pull_request_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8","_links":{"self":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments/71485794"},"html":{"href":"https://github.com/deliveroo/deliveroo.engineering/pull/8#discussion_r71485794"},"pull_request":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8"}}},"pull_request":{"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8","id":77946529,"html_url":"https://github.com/deliveroo/deliveroo.engineering/pull/8","diff_url":"https://github.com/deliveroo/deliveroo.engineering/pull/8.diff","patch_url":"https://github.com/deliveroo/deliveroo.engineering/pull/8.patch","issue_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/8","number":8,"state":"open","locked":false,"title":"Article cdn fastly cloudfront","user":{"login":"mezis","id":231698,"avatar_url":"https://avatars.githubusercontent.com/u/231698?v=3","gravatar_id":"","url":"https://api.github.com/users/mezis","html_url":"https://github.com/mezis","followers_url":"https://api.github.com/users/mezis/followers","following_url":"https://api.github.com/users/mezis/following{/other_user}","gists_url":"https://api.github.com/users/mezis/gists{/gist_id}","starred_url":"https://api.github.com/users/mezis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mezis/subscriptions","organizations_url":"https://api.github.com/users/mezis/orgs","repos_url":"https://api.github.com/users/mezis/repos","events_url":"https://api.github.com/users/mezis/events{/privacy}","received_events_url":"https://api.github.com/users/mezis/received_events","type":"User","site_admin":false},"body":"First post!\r\n\r\ncc @jphastings, our beloved Editor-in-Chief. Feel free to rewrite any part of this.\r\n\r\nTo read the article locally:\r\n\r\n- Add `future: true` to `_config.yml`\r\n- Run `bundle exec jekyll serve`","created_at":"2016-07-19T09:45:15Z","updated_at":"2016-07-20T08:38:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"54a834e471c098c3a3b1875081209650d5556fde","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8/commits","review_comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8/comments","review_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments{/number}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/8/comments","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/8a743ab19b326152843947c43edbccc6aa63b50c","head":{"label":"deliveroo:article-cdn-fastly-cloudfront","ref":"article-cdn-fastly-cloudfront","sha":"8a743ab19b326152843947c43edbccc6aa63b50c","user":{"login":"deliveroo","id":10514109,"avatar_url":"https://avatars.githubusercontent.com/u/10514109?v=3","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"repo":{"id":32532933,"name":"deliveroo.engineering","full_name":"deliveroo/deliveroo.engineering","owner":{"login":"deliveroo","id":10514109,"avatar_url":"https://avatars.githubusercontent.com/u/10514109?v=3","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/deliveroo/deliveroo.engineering","description":"Guidelines and conventions for good software at Deliveroo","fork":true,"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering","forks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/forks","keys_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/teams","hooks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/hooks","issue_events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/events{/number}","events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/events","assignees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/assignees{/user}","branches_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/branches{/branch}","tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/tags","blobs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/refs{/sha}","trees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/{sha}","languages_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/languages","stargazers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/stargazers","contributors_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contributors","subscribers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscribers","subscription_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscription","commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/commits{/sha}","git_commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/commits{/sha}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/comments{/number}","issue_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/comments{/number}","contents_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contents/{+path}","compare_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/merges","archive_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/downloads","issues_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues{/number}","pulls_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls{/number}","milestones_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/milestones{/number}","notifications_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/labels{/name}","releases_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/releases{/id}","deployments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/deployments","created_at":"2015-03-19T16:23:28Z","updated_at":"2016-07-20T08:23:55Z","pushed_at":"2016-07-19T09:45:15Z","git_url":"git://github.com/deliveroo/deliveroo.engineering.git","ssh_url":"git@github.com:deliveroo/deliveroo.engineering.git","clone_url":"https://github.com/deliveroo/deliveroo.engineering.git","svn_url":"https://github.com/deliveroo/deliveroo.engineering","homepage":"","size":877,"stargazers_count":6,"watchers_count":6,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":1,"mirror_url":null,"open_issues_count":6,"forks":1,"open_issues":6,"watchers":6,"default_branch":"gh-pages"}},"base":{"label":"deliveroo:gh-pages","ref":"gh-pages","sha":"cf4255224b105ce11fcd5cbad78092f603d7fea2","user":{"login":"deliveroo","id":10514109,"avatar_url":"https://avatars.githubusercontent.com/u/10514109?v=3","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"repo":{"id":32532933,"name":"deliveroo.engineering","full_name":"deliveroo/deliveroo.engineering","owner":{"login":"deliveroo","id":10514109,"avatar_url":"https://avatars.githubusercontent.com/u/10514109?v=3","gravatar_id":"","url":"https://api.github.com/users/deliveroo","html_url":"https://github.com/deliveroo","followers_url":"https://api.github.com/users/deliveroo/followers","following_url":"https://api.github.com/users/deliveroo/following{/other_user}","gists_url":"https://api.github.com/users/deliveroo/gists{/gist_id}","starred_url":"https://api.github.com/users/deliveroo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deliveroo/subscriptions","organizations_url":"https://api.github.com/users/deliveroo/orgs","repos_url":"https://api.github.com/users/deliveroo/repos","events_url":"https://api.github.com/users/deliveroo/events{/privacy}","received_events_url":"https://api.github.com/users/deliveroo/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/deliveroo/deliveroo.engineering","description":"Guidelines and conventions for good software at Deliveroo","fork":true,"url":"https://api.github.com/repos/deliveroo/deliveroo.engineering","forks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/forks","keys_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/teams","hooks_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/hooks","issue_events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/events{/number}","events_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/events","assignees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/assignees{/user}","branches_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/branches{/branch}","tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/tags","blobs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/refs{/sha}","trees_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/{sha}","languages_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/languages","stargazers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/stargazers","contributors_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contributors","subscribers_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscribers","subscription_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/subscription","commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/commits{/sha}","git_commits_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/git/commits{/sha}","comments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/comments{/number}","issue_comment_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/comments{/number}","contents_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/contents/{+path}","compare_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/merges","archive_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/downloads","issues_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues{/number}","pulls_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls{/number}","milestones_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/milestones{/number}","notifications_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/labels{/name}","releases_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/releases{/id}","deployments_url":"https://api.github.com/repos/deliveroo/deliveroo.engineering/deployments","created_at":"2015-03-19T16:23:28Z","updated_at":"2016-07-20T08:23:55Z","pushed_at":"2016-07-19T09:45:15Z","git_url":"git://github.com/deliveroo/deliveroo.engineering.git","ssh_url":"git@github.com:deliveroo/deliveroo.engineering.git","clone_url":"https://github.com/deliveroo/deliveroo.engineering.git","svn_url":"https://github.com/deliveroo/deliveroo.engineering","homepage":"","size":877,"stargazers_count":6,"watchers_count":6,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":1,"mirror_url":null,"open_issues_count":6,"forks":1,"open_issues":6,"watchers":6,"default_branch":"gh-pages"}},"_links":{"self":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8"},"html":{"href":"https://github.com/deliveroo/deliveroo.engineering/pull/8"},"issue":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/8"},"comments":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/issues/8/comments"},"review_comments":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8/comments"},"review_comment":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/pulls/8/commits"},"statuses":{"href":"https://api.github.com/repos/deliveroo/deliveroo.engineering/statuses/8a743ab19b326152843947c43edbccc6aa63b50c"}}}}
{"actor":{"display_login":"mezis"}}