{"action":"created","comment":{"url":"https://api.github.com/repos/twosigma/waiter/pulls/comments/192896688","pull_request_review_id":125761958,"id":192896688,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5Mjg5NjY4OA==","diff_hunk":"@@ -638,3 +642,145 @@\n           \"WAITER_PASSWORD\" (service-id->password-fn service-id)\n           \"WAITER_SERVICE_ID\" service-id\n           \"WAITER_USERNAME\" \"waiter\"}))\n+\n+;; Support for tracking service instance launching time stats\n+\n+(def instance-counts-zero {:requested 0 :started 0})\n+\n+(defn- make-launch-tracker-entry\n+  [service-id]\n+  [service-id\n+   {:instance-counts instance-counts-zero\n+    :known-instance-ids #{}\n+    :instance-scheduling-start-times []\n+    :service-schedule-timer (metrics/service-timer service-id \"launch-overhead\" \"schedule-time\")\n+    :service-startup-timer (metrics/service-timer service-id \"launch-overhead\" \"startup-time\")\n+    :starting-instance-ids []}])\n+\n+(defn update-launch-trackers\n+  \"Track metrics on app instances, specifically schedule and startup times.\"\n+  [service-id->launch-tracker new-service-ids removed-service-ids service-id->healthy-instances\n+   service-id->unhealthy-instances service-id->instance-counts waiter-schedule-timer]\n+  (let [now (t/now)\n+        new-tracker-entries (map make-launch-tracker-entry new-service-ids)\n+        tracker-entries' (->> service-id->launch-tracker\n+                              ;; Remove trackers from deleted services\n+                              (remove (comp removed-service-ids key))\n+                              ;; Add trackers for newly-discovered services\n+                              (concat new-tracker-entries))]\n+    ;; Update all trackers...\n+    (pc/for-map [[service-id\n+                  {:keys [known-instance-ids instance-counts instance-scheduling-start-times\n+                          service-schedule-timer service-startup-timer starting-instance-ids]\n+                   :as tracker-state}]\n+                 tracker-entries']\n+      service-id\n+      (let [healthy-instances (get service-id->healthy-instances service-id)\n+            known-instances' (->> service-id\n+                                  (get service-id->unhealthy-instances)\n+                                  (into healthy-instances))\n+            previously-known-instance? (comp known-instance-ids :id)\n+            new-instance-ids (->> known-instances'\n+                                  (remove previously-known-instance?)\n+                                  (sort-by :started-at)","path":"waiter/src/waiter/scheduler.clj","position":87,"original_position":87,"commit_id":"1010e9f785fb654c9010657938564b8043f60c16","original_commit_id":"1010e9f785fb654c9010657938564b8043f60c16","user":{"login":"DaoWen","id":1783674,"node_id":"MDQ6VXNlcjE3ODM2NzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1783674?v=4","gravatar_id":"","url":"https://api.github.com/users/DaoWen","html_url":"https://github.com/DaoWen","followers_url":"https://api.github.com/users/DaoWen/followers","following_url":"https://api.github.com/users/DaoWen/following{/other_user}","gists_url":"https://api.github.com/users/DaoWen/gists{/gist_id}","starred_url":"https://api.github.com/users/DaoWen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaoWen/subscriptions","organizations_url":"https://api.github.com/users/DaoWen/orgs","repos_url":"https://api.github.com/users/DaoWen/repos","events_url":"https://api.github.com/users/DaoWen/events{/privacy}","received_events_url":"https://api.github.com/users/DaoWen/received_events","type":"User","site_admin":false},"body":"> \"The comparison order is: service-id, started-at, and finally id.\"\r\n\r\nWe want `:started-at` as the primary sorting key, so that doesn't seem to work.","created_at":"2018-06-04T22:16:29Z","updated_at":"2018-06-04T22:16:30Z","html_url":"https://github.com/twosigma/waiter/pull/305#discussion_r192896688","pull_request_url":"https://api.github.com/repos/twosigma/waiter/pulls/305","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/comments/192896688"},"html":{"href":"https://github.com/twosigma/waiter/pull/305#discussion_r192896688"},"pull_request":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/305"}},"in_reply_to_id":192823015},"pull_request":{"url":"https://api.github.com/repos/twosigma/waiter/pulls/305","id":185842298,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg1ODQyMjk4","html_url":"https://github.com/twosigma/waiter/pull/305","diff_url":"https://github.com/twosigma/waiter/pull/305.diff","patch_url":"https://github.com/twosigma/waiter/pull/305.patch","issue_url":"https://api.github.com/repos/twosigma/waiter/issues/305","number":305,"state":"open","locked":false,"title":"App instance startup metrics","user":{"login":"DaoWen","id":1783674,"node_id":"MDQ6VXNlcjE3ODM2NzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1783674?v=4","gravatar_id":"","url":"https://api.github.com/users/DaoWen","html_url":"https://github.com/DaoWen","followers_url":"https://api.github.com/users/DaoWen/followers","following_url":"https://api.github.com/users/DaoWen/following{/other_user}","gists_url":"https://api.github.com/users/DaoWen/gists{/gist_id}","starred_url":"https://api.github.com/users/DaoWen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaoWen/subscriptions","organizations_url":"https://api.github.com/users/DaoWen/orgs","repos_url":"https://api.github.com/users/DaoWen/repos","events_url":"https://api.github.com/users/DaoWen/events{/privacy}","received_events_url":"https://api.github.com/users/DaoWen/received_events","type":"User","site_admin":false},"body":"## Changes proposed in this PR\r\n\r\nAdd codahale timers for three metrics:\r\n1. Time from a Waiter service's creation until the first instance appears (\"schedule time\").\r\n2. Time from the first instance appearing to when it's healthy (\"startup time\").\r\n3. Time from a Waiter service's creation until the first instance is healthy (\"start-to-healthy time\").\r\n\r\n<span>#</span>3 is basically <span>#</span>1 + <span>#</span>2. Each of these metrics are tracked both per-service and overall.\r\n\r\n## Why are we making these changes?\r\n\r\nWe want to have data on how long it takes to start up apps in production on our existing platforms (Mesos/Marathon) and future platforms (Kubernetes).","created_at":"2018-05-03T21:25:08Z","updated_at":"2018-06-04T22:16:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ac29d2bd4e37f8b26cb34e31c3ab0459170c1352","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":922028575,"node_id":"MDU6TGFiZWw5MjIwMjg1NzU=","url":"https://api.github.com/repos/twosigma/waiter/labels/metrics","name":"metrics","color":"ff88cc","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/twosigma/waiter/pulls/305/commits","review_comments_url":"https://api.github.com/repos/twosigma/waiter/pulls/305/comments","review_comment_url":"https://api.github.com/repos/twosigma/waiter/pulls/comments{/number}","comments_url":"https://api.github.com/repos/twosigma/waiter/issues/305/comments","statuses_url":"https://api.github.com/repos/twosigma/waiter/statuses/1010e9f785fb654c9010657938564b8043f60c16","head":{"label":"DaoWen:feature/app-startup-metrics","ref":"feature/app-startup-metrics","sha":"1010e9f785fb654c9010657938564b8043f60c16","user":{"login":"DaoWen","id":1783674,"node_id":"MDQ6VXNlcjE3ODM2NzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1783674?v=4","gravatar_id":"","url":"https://api.github.com/users/DaoWen","html_url":"https://github.com/DaoWen","followers_url":"https://api.github.com/users/DaoWen/followers","following_url":"https://api.github.com/users/DaoWen/following{/other_user}","gists_url":"https://api.github.com/users/DaoWen/gists{/gist_id}","starred_url":"https://api.github.com/users/DaoWen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaoWen/subscriptions","organizations_url":"https://api.github.com/users/DaoWen/orgs","repos_url":"https://api.github.com/users/DaoWen/repos","events_url":"https://api.github.com/users/DaoWen/events{/privacy}","received_events_url":"https://api.github.com/users/DaoWen/received_events","type":"User","site_admin":false},"repo":{"id":105570659,"node_id":"MDEwOlJlcG9zaXRvcnkxMDU1NzA2NTk=","name":"waiter","full_name":"DaoWen/waiter","owner":{"login":"DaoWen","id":1783674,"node_id":"MDQ6VXNlcjE3ODM2NzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1783674?v=4","gravatar_id":"","url":"https://api.github.com/users/DaoWen","html_url":"https://github.com/DaoWen","followers_url":"https://api.github.com/users/DaoWen/followers","following_url":"https://api.github.com/users/DaoWen/following{/other_user}","gists_url":"https://api.github.com/users/DaoWen/gists{/gist_id}","starred_url":"https://api.github.com/users/DaoWen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaoWen/subscriptions","organizations_url":"https://api.github.com/users/DaoWen/orgs","repos_url":"https://api.github.com/users/DaoWen/repos","events_url":"https://api.github.com/users/DaoWen/events{/privacy}","received_events_url":"https://api.github.com/users/DaoWen/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/DaoWen/waiter","description":"Runs, manages, and autoscales web services on Mesos","fork":true,"url":"https://api.github.com/repos/DaoWen/waiter","forks_url":"https://api.github.com/repos/DaoWen/waiter/forks","keys_url":"https://api.github.com/repos/DaoWen/waiter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DaoWen/waiter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DaoWen/waiter/teams","hooks_url":"https://api.github.com/repos/DaoWen/waiter/hooks","issue_events_url":"https://api.github.com/repos/DaoWen/waiter/issues/events{/number}","events_url":"https://api.github.com/repos/DaoWen/waiter/events","assignees_url":"https://api.github.com/repos/DaoWen/waiter/assignees{/user}","branches_url":"https://api.github.com/repos/DaoWen/waiter/branches{/branch}","tags_url":"https://api.github.com/repos/DaoWen/waiter/tags","blobs_url":"https://api.github.com/repos/DaoWen/waiter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DaoWen/waiter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DaoWen/waiter/git/refs{/sha}","trees_url":"https://api.github.com/repos/DaoWen/waiter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DaoWen/waiter/statuses/{sha}","languages_url":"https://api.github.com/repos/DaoWen/waiter/languages","stargazers_url":"https://api.github.com/repos/DaoWen/waiter/stargazers","contributors_url":"https://api.github.com/repos/DaoWen/waiter/contributors","subscribers_url":"https://api.github.com/repos/DaoWen/waiter/subscribers","subscription_url":"https://api.github.com/repos/DaoWen/waiter/subscription","commits_url":"https://api.github.com/repos/DaoWen/waiter/commits{/sha}","git_commits_url":"https://api.github.com/repos/DaoWen/waiter/git/commits{/sha}","comments_url":"https://api.github.com/repos/DaoWen/waiter/comments{/number}","issue_comment_url":"https://api.github.com/repos/DaoWen/waiter/issues/comments{/number}","contents_url":"https://api.github.com/repos/DaoWen/waiter/contents/{+path}","compare_url":"https://api.github.com/repos/DaoWen/waiter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DaoWen/waiter/merges","archive_url":"https://api.github.com/repos/DaoWen/waiter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DaoWen/waiter/downloads","issues_url":"https://api.github.com/repos/DaoWen/waiter/issues{/number}","pulls_url":"https://api.github.com/repos/DaoWen/waiter/pulls{/number}","milestones_url":"https://api.github.com/repos/DaoWen/waiter/milestones{/number}","notifications_url":"https://api.github.com/repos/DaoWen/waiter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DaoWen/waiter/labels{/name}","releases_url":"https://api.github.com/repos/DaoWen/waiter/releases{/id}","deployments_url":"https://api.github.com/repos/DaoWen/waiter/deployments","created_at":"2017-10-02T18:39:29Z","updated_at":"2018-05-15T20:18:17Z","pushed_at":"2018-06-04T04:01:18Z","git_url":"git://github.com/DaoWen/waiter.git","ssh_url":"git@github.com:DaoWen/waiter.git","clone_url":"https://github.com/DaoWen/waiter.git","svn_url":"https://github.com/DaoWen/waiter","homepage":"","size":2182,"stargazers_count":0,"watchers_count":0,"language":"Clojure","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":5,"watchers":0,"default_branch":"master"}},"base":{"label":"twosigma:master","ref":"master","sha":"c4264f22ce7dc583ae60b511950ff2845d78f25a","user":{"login":"twosigma","id":4049181,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQwNDkxODE=","avatar_url":"https://avatars3.githubusercontent.com/u/4049181?v=4","gravatar_id":"","url":"https://api.github.com/users/twosigma","html_url":"https://github.com/twosigma","followers_url":"https://api.github.com/users/twosigma/followers","following_url":"https://api.github.com/users/twosigma/following{/other_user}","gists_url":"https://api.github.com/users/twosigma/gists{/gist_id}","starred_url":"https://api.github.com/users/twosigma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twosigma/subscriptions","organizations_url":"https://api.github.com/users/twosigma/orgs","repos_url":"https://api.github.com/users/twosigma/repos","events_url":"https://api.github.com/users/twosigma/events{/privacy}","received_events_url":"https://api.github.com/users/twosigma/received_events","type":"Organization","site_admin":false},"repo":{"id":81378873,"node_id":"MDEwOlJlcG9zaXRvcnk4MTM3ODg3Mw==","name":"waiter","full_name":"twosigma/waiter","owner":{"login":"twosigma","id":4049181,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQwNDkxODE=","avatar_url":"https://avatars3.githubusercontent.com/u/4049181?v=4","gravatar_id":"","url":"https://api.github.com/users/twosigma","html_url":"https://github.com/twosigma","followers_url":"https://api.github.com/users/twosigma/followers","following_url":"https://api.github.com/users/twosigma/following{/other_user}","gists_url":"https://api.github.com/users/twosigma/gists{/gist_id}","starred_url":"https://api.github.com/users/twosigma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twosigma/subscriptions","organizations_url":"https://api.github.com/users/twosigma/orgs","repos_url":"https://api.github.com/users/twosigma/repos","events_url":"https://api.github.com/users/twosigma/events{/privacy}","received_events_url":"https://api.github.com/users/twosigma/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/twosigma/waiter","description":"Runs, manages, and autoscales web services on Mesos","fork":false,"url":"https://api.github.com/repos/twosigma/waiter","forks_url":"https://api.github.com/repos/twosigma/waiter/forks","keys_url":"https://api.github.com/repos/twosigma/waiter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/twosigma/waiter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/twosigma/waiter/teams","hooks_url":"https://api.github.com/repos/twosigma/waiter/hooks","issue_events_url":"https://api.github.com/repos/twosigma/waiter/issues/events{/number}","events_url":"https://api.github.com/repos/twosigma/waiter/events","assignees_url":"https://api.github.com/repos/twosigma/waiter/assignees{/user}","branches_url":"https://api.github.com/repos/twosigma/waiter/branches{/branch}","tags_url":"https://api.github.com/repos/twosigma/waiter/tags","blobs_url":"https://api.github.com/repos/twosigma/waiter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/twosigma/waiter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/twosigma/waiter/git/refs{/sha}","trees_url":"https://api.github.com/repos/twosigma/waiter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/twosigma/waiter/statuses/{sha}","languages_url":"https://api.github.com/repos/twosigma/waiter/languages","stargazers_url":"https://api.github.com/repos/twosigma/waiter/stargazers","contributors_url":"https://api.github.com/repos/twosigma/waiter/contributors","subscribers_url":"https://api.github.com/repos/twosigma/waiter/subscribers","subscription_url":"https://api.github.com/repos/twosigma/waiter/subscription","commits_url":"https://api.github.com/repos/twosigma/waiter/commits{/sha}","git_commits_url":"https://api.github.com/repos/twosigma/waiter/git/commits{/sha}","comments_url":"https://api.github.com/repos/twosigma/waiter/comments{/number}","issue_comment_url":"https://api.github.com/repos/twosigma/waiter/issues/comments{/number}","contents_url":"https://api.github.com/repos/twosigma/waiter/contents/{+path}","compare_url":"https://api.github.com/repos/twosigma/waiter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/twosigma/waiter/merges","archive_url":"https://api.github.com/repos/twosigma/waiter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/twosigma/waiter/downloads","issues_url":"https://api.github.com/repos/twosigma/waiter/issues{/number}","pulls_url":"https://api.github.com/repos/twosigma/waiter/pulls{/number}","milestones_url":"https://api.github.com/repos/twosigma/waiter/milestones{/number}","notifications_url":"https://api.github.com/repos/twosigma/waiter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/twosigma/waiter/labels{/name}","releases_url":"https://api.github.com/repos/twosigma/waiter/releases{/id}","deployments_url":"https://api.github.com/repos/twosigma/waiter/deployments","created_at":"2017-02-08T21:40:37Z","updated_at":"2018-06-04T21:25:59Z","pushed_at":"2018-06-04T21:25:58Z","git_url":"git://github.com/twosigma/waiter.git","ssh_url":"git@github.com:twosigma/waiter.git","clone_url":"https://github.com/twosigma/waiter.git","svn_url":"https://github.com/twosigma/waiter","homepage":"","size":2586,"stargazers_count":24,"watchers_count":24,"language":"Clojure","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7,"mirror_url":null,"archived":false,"open_issues_count":20,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":7,"open_issues":20,"watchers":24,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/305"},"html":{"href":"https://github.com/twosigma/waiter/pull/305"},"issue":{"href":"https://api.github.com/repos/twosigma/waiter/issues/305"},"comments":{"href":"https://api.github.com/repos/twosigma/waiter/issues/305/comments"},"review_comments":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/305/comments"},"review_comment":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/twosigma/waiter/pulls/305/commits"},"statuses":{"href":"https://api.github.com/repos/twosigma/waiter/statuses/1010e9f785fb654c9010657938564b8043f60c16"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"DaoWen"}}