{"action":"opened","number":1009,"pull_request":{"url":"https://api.github.com/repos/mozilla/testpilot/pulls/1009","id":74708293,"html_url":"https://github.com/mozilla/testpilot/pull/1009","diff_url":"https://github.com/mozilla/testpilot/pull/1009.diff","patch_url":"https://github.com/mozilla/testpilot/pull/1009.patch","issue_url":"https://api.github.com/repos/mozilla/testpilot/issues/1009","number":1009,"state":"open","locked":false,"title":"[WIP]: Add variant testing framework for experiments.","user":{"login":"chuckharmston","id":23885,"avatar_url":"https://avatars.githubusercontent.com/u/23885?v=3","gravatar_id":"","url":"https://api.github.com/users/chuckharmston","html_url":"https://github.com/chuckharmston","followers_url":"https://api.github.com/users/chuckharmston/followers","following_url":"https://api.github.com/users/chuckharmston/following{/other_user}","gists_url":"https://api.github.com/users/chuckharmston/gists{/gist_id}","starred_url":"https://api.github.com/users/chuckharmston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chuckharmston/subscriptions","organizations_url":"https://api.github.com/users/chuckharmston/orgs","repos_url":"https://api.github.com/users/chuckharmston/repos","events_url":"https://api.github.com/users/chuckharmston/events{/privacy}","received_events_url":"https://api.github.com/users/chuckharmston/received_events","type":"User","site_admin":false},"body":"So, here's a very rough sketch at what #477 could look like. Here's how it works:\r\n\r\n1. Upon initialization, an experiment spams Test Pilot with a [test and variant definition](https://gist.github.com/chuckharmston/b8b728844c44fa931ace881e92286b4d#experiment---test-pilot) using [`nslObserverService`](https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIObserverService) and the `testpilot::register-variants` topic. This is done in an interval, as there is no guarantee that experiments are loaded after Test Pilot is.\r\n2. The first time Test Pilot receives that message, it [does some magic](https://gist.github.com/chuckharmston/b8b728844c44fa931ace881e92286b4d) and deterministically (seeded by test name and `store.clientUUID`) places users into variants for each test.\r\n3. Test Pilot stores and ships a payload containing those results back to the experiment, using `testpilot::receive-variants`. When received, the experiment clears the interval and can do whatever it would like with the payload.\r\n4. In any future `testpilot::send-metric` messages sent by experiments, the variants are appended to the payload before `submitExternalPing` is called.\r\n\r\nIn addition to the code in the PR, we'd include something approximating the following in some sort of experiment developer SDK:\r\n\r\n```javascript\r\nconst {classes: Cc, interfaces: Ci, utils: Cu} = Components;\r\n\r\nfunction TestPilotObserver(topic, resolve) {\r\n  this.topic = topic;\r\n  this.resolve = resolve;\r\n  this.register();\r\n}\r\n\r\nTestPilotObserver.prototype = {\r\n  observe: function(subject, topic, data) {\r\n    this.resolve({\r\n      subject: subject,\r\n      topic: topic,\r\n      data: data\r\n    });\r\n    this.unregister();\r\n  },\r\n\r\n  register: function() {\r\n    let observerService = Cc[\"@mozilla.org/observer-service;1\"]\r\n                          .getService(Ci.nsIObserverService);\r\n    observerService.addObserver(this, this.topic, false);\r\n  },\r\n\r\n  unregister: function() {\r\n    let observerService = Cc[\"@mozilla.org/observer-service;1\"]\r\n                          .getService(Ci.nsIObserverService);\r\n    observerService.removeObserver(this, this.topic);\r\n  }\r\n};\r\n\r\nconst registerVariants = function(addonId, payload) {\r\n  let observerService = Cc[\"@mozilla.org/observer-service;1\"]\r\n                        .getService(Ci.nsIObserverService);\r\n\r\n  let receiveTopic = 'testpilot::receive-variants';\r\n  let registerTopic = 'testpilot::register-variants';\r\n\r\n  return new Promise((resolve, reject) => {\r\n    let intervalPing = win.setInterval(() => {\r\n      observerService.notifyObservers(\r\n        {\r\n          wrappedJSObject: {\r\n            observersModuleSubjectWrapper: true,\r\n            object: 'universal-search@mozilla.com'\r\n          }\r\n        },\r\n        registerTopic,\r\n        JSON.stringify(payload)\r\n      );\r\n    }, 100);\r\n    let observer = new TestPilotObserver(receiveTopic, data => {\r\n      win.clearInterval(intervalPing);\r\n      resolve(data);\r\n    });\r\n    observerService.addObserver(observer, receiveTopic, false);\r\n  });\r\n};\r\n```\r\n\r\nWhich the experiments themselves would use like this:\r\n\r\n```javascript\r\nregisterVariants('universal-search@mozilla.com', payload).then(payload => {\r\n  app.console.log(JSON.parse(payload.data));\r\n});\r\n```\r\n\r\nUsing the above code and [this sample JSON blob as the payload](https://gist.github.com/chuckharmston/b8b728844c44fa931ace881e92286b4d#experiment---test-pilot), Universal Search receives the following object:\r\n\r\n<img width=\"169\" alt=\"screen shot 2016-06-21 at 6 27 56 pm\" src=\"https://cloud.githubusercontent.com/assets/23885/16250849/06a92fb4-37de-11e6-97f8-7b0a00c0a819.png\">\r\n\r\nAnd the telemetry pings sent by Universal Search look like this:\r\n\r\n<img width=\"299\" alt=\"screen shot 2016-06-21 at 6 23 07 pm\" src=\"https://cloud.githubusercontent.com/assets/23885/16250856/0fb5f45c-37de-11e6-8fe6-770502adfb9f.png\">\r\n\r\nThis is all very messy, but I wanted to get eyes on it early. r? @lmorchard \r\n\r\nPotentially-interested parties: @k88hudson @rjweiss","created_at":"2016-06-22T00:39:15Z","updated_at":"2016-06-22T00:39:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/testpilot/pulls/1009/commits","review_comments_url":"https://api.github.com/repos/mozilla/testpilot/pulls/1009/comments","review_comment_url":"https://api.github.com/repos/mozilla/testpilot/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mozilla/testpilot/issues/1009/comments","statuses_url":"https://api.github.com/repos/mozilla/testpilot/statuses/4143c6c57fb8614240e01ba407cd3299387b5b6d","head":{"label":"chuckharmston:477-variant-testing","ref":"477-variant-testing","sha":"4143c6c57fb8614240e01ba407cd3299387b5b6d","user":{"login":"chuckharmston","id":23885,"avatar_url":"https://avatars.githubusercontent.com/u/23885?v=3","gravatar_id":"","url":"https://api.github.com/users/chuckharmston","html_url":"https://github.com/chuckharmston","followers_url":"https://api.github.com/users/chuckharmston/followers","following_url":"https://api.github.com/users/chuckharmston/following{/other_user}","gists_url":"https://api.github.com/users/chuckharmston/gists{/gist_id}","starred_url":"https://api.github.com/users/chuckharmston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chuckharmston/subscriptions","organizations_url":"https://api.github.com/users/chuckharmston/orgs","repos_url":"https://api.github.com/users/chuckharmston/repos","events_url":"https://api.github.com/users/chuckharmston/events{/privacy}","received_events_url":"https://api.github.com/users/chuckharmston/received_events","type":"User","site_admin":false},"repo":{"id":58287800,"name":"testpilot","full_name":"chuckharmston/testpilot","owner":{"login":"chuckharmston","id":23885,"avatar_url":"https://avatars.githubusercontent.com/u/23885?v=3","gravatar_id":"","url":"https://api.github.com/users/chuckharmston","html_url":"https://github.com/chuckharmston","followers_url":"https://api.github.com/users/chuckharmston/followers","following_url":"https://api.github.com/users/chuckharmston/following{/other_user}","gists_url":"https://api.github.com/users/chuckharmston/gists{/gist_id}","starred_url":"https://api.github.com/users/chuckharmston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chuckharmston/subscriptions","organizations_url":"https://api.github.com/users/chuckharmston/orgs","repos_url":"https://api.github.com/users/chuckharmston/repos","events_url":"https://api.github.com/users/chuckharmston/events{/privacy}","received_events_url":"https://api.github.com/users/chuckharmston/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/chuckharmston/testpilot","description":"Test Pilot is a platform for performing controlled tests of new product concepts in Firefox","fork":true,"url":"https://api.github.com/repos/chuckharmston/testpilot","forks_url":"https://api.github.com/repos/chuckharmston/testpilot/forks","keys_url":"https://api.github.com/repos/chuckharmston/testpilot/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chuckharmston/testpilot/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chuckharmston/testpilot/teams","hooks_url":"https://api.github.com/repos/chuckharmston/testpilot/hooks","issue_events_url":"https://api.github.com/repos/chuckharmston/testpilot/issues/events{/number}","events_url":"https://api.github.com/repos/chuckharmston/testpilot/events","assignees_url":"https://api.github.com/repos/chuckharmston/testpilot/assignees{/user}","branches_url":"https://api.github.com/repos/chuckharmston/testpilot/branches{/branch}","tags_url":"https://api.github.com/repos/chuckharmston/testpilot/tags","blobs_url":"https://api.github.com/repos/chuckharmston/testpilot/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chuckharmston/testpilot/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chuckharmston/testpilot/git/refs{/sha}","trees_url":"https://api.github.com/repos/chuckharmston/testpilot/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chuckharmston/testpilot/statuses/{sha}","languages_url":"https://api.github.com/repos/chuckharmston/testpilot/languages","stargazers_url":"https://api.github.com/repos/chuckharmston/testpilot/stargazers","contributors_url":"https://api.github.com/repos/chuckharmston/testpilot/contributors","subscribers_url":"https://api.github.com/repos/chuckharmston/testpilot/subscribers","subscription_url":"https://api.github.com/repos/chuckharmston/testpilot/subscription","commits_url":"https://api.github.com/repos/chuckharmston/testpilot/commits{/sha}","git_commits_url":"https://api.github.com/repos/chuckharmston/testpilot/git/commits{/sha}","comments_url":"https://api.github.com/repos/chuckharmston/testpilot/comments{/number}","issue_comment_url":"https://api.github.com/repos/chuckharmston/testpilot/issues/comments{/number}","contents_url":"https://api.github.com/repos/chuckharmston/testpilot/contents/{+path}","compare_url":"https://api.github.com/repos/chuckharmston/testpilot/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chuckharmston/testpilot/merges","archive_url":"https://api.github.com/repos/chuckharmston/testpilot/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chuckharmston/testpilot/downloads","issues_url":"https://api.github.com/repos/chuckharmston/testpilot/issues{/number}","pulls_url":"https://api.github.com/repos/chuckharmston/testpilot/pulls{/number}","milestones_url":"https://api.github.com/repos/chuckharmston/testpilot/milestones{/number}","notifications_url":"https://api.github.com/repos/chuckharmston/testpilot/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chuckharmston/testpilot/labels{/name}","releases_url":"https://api.github.com/repos/chuckharmston/testpilot/releases{/id}","deployments_url":"https://api.github.com/repos/chuckharmston/testpilot/deployments","created_at":"2016-05-07T23:00:39Z","updated_at":"2016-06-20T15:47:28Z","pushed_at":"2016-06-22T00:25:27Z","git_url":"git://github.com/chuckharmston/testpilot.git","ssh_url":"git@github.com:chuckharmston/testpilot.git","clone_url":"https://github.com/chuckharmston/testpilot.git","svn_url":"https://github.com/chuckharmston/testpilot","homepage":"https://wiki.mozilla.org/Test_Pilot","size":7881,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"3390f364e5acb1f6a6245e700f60082b5e8e9ca8","user":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":40625180,"name":"testpilot","full_name":"mozilla/testpilot","owner":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/testpilot","description":"Test Pilot is a platform for performing controlled tests of new product concepts in Firefox","fork":false,"url":"https://api.github.com/repos/mozilla/testpilot","forks_url":"https://api.github.com/repos/mozilla/testpilot/forks","keys_url":"https://api.github.com/repos/mozilla/testpilot/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/testpilot/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/testpilot/teams","hooks_url":"https://api.github.com/repos/mozilla/testpilot/hooks","issue_events_url":"https://api.github.com/repos/mozilla/testpilot/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/testpilot/events","assignees_url":"https://api.github.com/repos/mozilla/testpilot/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/testpilot/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/testpilot/tags","blobs_url":"https://api.github.com/repos/mozilla/testpilot/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/testpilot/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/testpilot/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/testpilot/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/testpilot/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/testpilot/languages","stargazers_url":"https://api.github.com/repos/mozilla/testpilot/stargazers","contributors_url":"https://api.github.com/repos/mozilla/testpilot/contributors","subscribers_url":"https://api.github.com/repos/mozilla/testpilot/subscribers","subscription_url":"https://api.github.com/repos/mozilla/testpilot/subscription","commits_url":"https://api.github.com/repos/mozilla/testpilot/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/testpilot/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/testpilot/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/testpilot/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/testpilot/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/testpilot/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/testpilot/merges","archive_url":"https://api.github.com/repos/mozilla/testpilot/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/testpilot/downloads","issues_url":"https://api.github.com/repos/mozilla/testpilot/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/testpilot/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/testpilot/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/testpilot/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/testpilot/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/testpilot/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/testpilot/deployments","created_at":"2015-08-12T21:27:27Z","updated_at":"2016-06-20T19:25:54Z","pushed_at":"2016-06-21T21:01:58Z","git_url":"git://github.com/mozilla/testpilot.git","ssh_url":"git@github.com:mozilla/testpilot.git","clone_url":"https://github.com/mozilla/testpilot.git","svn_url":"https://github.com/mozilla/testpilot","homepage":"https://testpilot.firefox.com/","size":12571,"stargazers_count":36,"watchers_count":36,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":119,"forks":21,"open_issues":119,"watchers":36,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/testpilot/pulls/1009"},"html":{"href":"https://github.com/mozilla/testpilot/pull/1009"},"issue":{"href":"https://api.github.com/repos/mozilla/testpilot/issues/1009"},"comments":{"href":"https://api.github.com/repos/mozilla/testpilot/issues/1009/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/testpilot/pulls/1009/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/testpilot/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mozilla/testpilot/pulls/1009/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/testpilot/statuses/4143c6c57fb8614240e01ba407cd3299387b5b6d"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":54,"deletions":3,"changed_files":3}}
{"actor":{"display_login":"chuckharmston"}}