{"action":"created","comment":{"url":"https://api.github.com/repos/airbnb/nerve/pulls/comments/70893229","id":70893229,"diff_hunk":"@@ -57,32 +59,59 @@ def initialize(service={})\n       # force an initial report on startup\n       @was_up = nil\n \n+      # when this watcher is started it will store the\n+      # thread here\n+      @run_thread = nil\n+      @should_finish = false\n+\n       log.debug \"nerve: created service watcher for #{@name} with #{@service_checks.size} checks\"\n     end\n \n+    def start()\n+      unless @run_thread\n+        @run_thread = Thread.new { self.run() }\n+      else\n+        log.error \"nerve: tried to double start a watcher\"\n+      end\n+    end\n+\n+    def stop()\n+      log.info \"nerve: stopping service watch #{@name}\"\n+      @should_finish = true\n+      return true if @run_thread.nil?\n+\n+      unclean_shutdown = @run_thread.join(10).nil?\n+      if unclean_shutdown\n+        log.error \"nerve: unclean shutdown of #{@name}, killing thread\"\n+        Thread.kill(@run_thread)\n+      end\n+      @run_thread = nil\n+      !unclean_shutdown\n+    end\n+\n+    def alive?()\n+      !@run_thread.nil? && @run_thread.alive?\n+    end\n+\n     def run()\n       log.info \"nerve: starting service watch #{@name}\"\n-\n       @reporter.start()\n \n-      until $EXIT\n+      until $EXIT or @should_finish","path":"lib/nerve/service_watcher.rb","position":62,"original_position":62,"commit_id":"3b436fbe1b36ff3ffdeafbcd11b3acfb46d469c4","original_commit_id":"3b436fbe1b36ff3ffdeafbcd11b3acfb46d469c4","user":{"login":"jolynch","id":1420460,"avatar_url":"https://avatars.githubusercontent.com/u/1420460?v=3","gravatar_id":"","url":"https://api.github.com/users/jolynch","html_url":"https://github.com/jolynch","followers_url":"https://api.github.com/users/jolynch/followers","following_url":"https://api.github.com/users/jolynch/following{/other_user}","gists_url":"https://api.github.com/users/jolynch/gists{/gist_id}","starred_url":"https://api.github.com/users/jolynch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jolynch/subscriptions","organizations_url":"https://api.github.com/users/jolynch/orgs","repos_url":"https://api.github.com/users/jolynch/repos","events_url":"https://api.github.com/users/jolynch/events{/privacy}","received_events_url":"https://api.github.com/users/jolynch/received_events","type":"User","site_admin":false},"body":"Done. Yea I know about the or thing I just sometimes get my python and ruby confused lol. Cheers.","created_at":"2016-07-14T22:08:01Z","updated_at":"2016-07-14T22:08:01Z","html_url":"https://github.com/airbnb/nerve/pull/80#discussion_r70893229","pull_request_url":"https://api.github.com/repos/airbnb/nerve/pulls/80","_links":{"self":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/comments/70893229"},"html":{"href":"https://github.com/airbnb/nerve/pull/80#discussion_r70893229"},"pull_request":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/80"}}},"pull_request":{"url":"https://api.github.com/repos/airbnb/nerve/pulls/80","id":57014676,"html_url":"https://github.com/airbnb/nerve/pull/80","diff_url":"https://github.com/airbnb/nerve/pull/80.diff","patch_url":"https://github.com/airbnb/nerve/pull/80.patch","issue_url":"https://api.github.com/repos/airbnb/nerve/issues/80","number":80,"state":"open","locked":false,"title":"Dynamic reconfiguration of service watchers","user":{"login":"jolynch","id":1420460,"avatar_url":"https://avatars.githubusercontent.com/u/1420460?v=3","gravatar_id":"","url":"https://api.github.com/users/jolynch","html_url":"https://github.com/jolynch","followers_url":"https://api.github.com/users/jolynch/followers","following_url":"https://api.github.com/users/jolynch/following{/other_user}","gists_url":"https://api.github.com/users/jolynch/gists{/gist_id}","starred_url":"https://api.github.com/users/jolynch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jolynch/subscriptions","organizations_url":"https://api.github.com/users/jolynch/orgs","repos_url":"https://api.github.com/users/jolynch/repos","events_url":"https://api.github.com/users/jolynch/events{/privacy}","received_events_url":"https://api.github.com/users/jolynch/received_events","type":"User","site_admin":false},"body":"@igor47 @jnb \r\nThis PR should enable Nerve to dynamically add and remove registrations in response to a SIGHUP\r\n    \r\nWhy?\r\nThe zero downtime Nerve restart method that relies on starting duplicate nerves and relying on Synapse to de-duplicate registrations works until you start getting to hundreds of services per box. Then for every new service you schedule you have to send hundreds of ZK push updates across the entire infrastructure just to have Synapse go \"oh yup it's the same\". For us at Yelp this manifests as lots of Zookeeper \"churn\" (causing lots of snapshots) that has caused some instability for us.\r\n\r\nHow?\r\nThis patch changes the main loop to listen for a SIGHUP, and when it receives one it diffs the new configuration against the in memory state, launching and reaping watchers as needed.\r\n\r\nAmazingly all the specs still pass without me changing them, which makes me think that I've got some tests to write. This diff is full of some subtle ordering constraints (signals + threads + methods with side effects galore), so I'd like to get some eyes on it as I'm writing specs.\r\n\r\nI also think I may have fixed a bug with shutdown where the main thread used to wait forever (.join without a timeout) if it couldn't reap a watcher... now we give watchers 10s to go away before we Thread.kill them.","created_at":"2016-01-24T21:40:51Z","updated_at":"2016-07-14T22:08:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7940f0b8a57e9f0ca54b7a256ed04b463345c68d","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/airbnb/nerve/pulls/80/commits","review_comments_url":"https://api.github.com/repos/airbnb/nerve/pulls/80/comments","review_comment_url":"https://api.github.com/repos/airbnb/nerve/pulls/comments{/number}","comments_url":"https://api.github.com/repos/airbnb/nerve/issues/80/comments","statuses_url":"https://api.github.com/repos/airbnb/nerve/statuses/3b436fbe1b36ff3ffdeafbcd11b3acfb46d469c4","head":{"label":"airbnb:dynamic_reconfiguration","ref":"dynamic_reconfiguration","sha":"3b436fbe1b36ff3ffdeafbcd11b3acfb46d469c4","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":7491726,"name":"nerve","full_name":"airbnb/nerve","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/nerve","description":"A service registration daemon that performs health checks; companion to airbnb/synapse","fork":false,"url":"https://api.github.com/repos/airbnb/nerve","forks_url":"https://api.github.com/repos/airbnb/nerve/forks","keys_url":"https://api.github.com/repos/airbnb/nerve/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/nerve/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/nerve/teams","hooks_url":"https://api.github.com/repos/airbnb/nerve/hooks","issue_events_url":"https://api.github.com/repos/airbnb/nerve/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/nerve/events","assignees_url":"https://api.github.com/repos/airbnb/nerve/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/nerve/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/nerve/tags","blobs_url":"https://api.github.com/repos/airbnb/nerve/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/nerve/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/nerve/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/nerve/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/nerve/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/nerve/languages","stargazers_url":"https://api.github.com/repos/airbnb/nerve/stargazers","contributors_url":"https://api.github.com/repos/airbnb/nerve/contributors","subscribers_url":"https://api.github.com/repos/airbnb/nerve/subscribers","subscription_url":"https://api.github.com/repos/airbnb/nerve/subscription","commits_url":"https://api.github.com/repos/airbnb/nerve/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/nerve/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/nerve/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/nerve/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/nerve/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/nerve/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/nerve/merges","archive_url":"https://api.github.com/repos/airbnb/nerve/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/nerve/downloads","issues_url":"https://api.github.com/repos/airbnb/nerve/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/nerve/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/nerve/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/nerve/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/nerve/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/nerve/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/nerve/deployments","created_at":"2013-01-07T23:16:00Z","updated_at":"2016-07-14T21:32:34Z","pushed_at":"2016-07-13T07:30:40Z","git_url":"git://github.com/airbnb/nerve.git","ssh_url":"git@github.com:airbnb/nerve.git","clone_url":"https://github.com/airbnb/nerve.git","svn_url":"https://github.com/airbnb/nerve","homepage":"","size":474,"stargazers_count":687,"watchers_count":687,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":100,"mirror_url":null,"open_issues_count":18,"forks":100,"open_issues":18,"watchers":687,"default_branch":"master"}},"base":{"label":"airbnb:master","ref":"master","sha":"754d8410da61e84d086b34c71a8c5b2972a00b68","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":7491726,"name":"nerve","full_name":"airbnb/nerve","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/nerve","description":"A service registration daemon that performs health checks; companion to airbnb/synapse","fork":false,"url":"https://api.github.com/repos/airbnb/nerve","forks_url":"https://api.github.com/repos/airbnb/nerve/forks","keys_url":"https://api.github.com/repos/airbnb/nerve/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/nerve/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/nerve/teams","hooks_url":"https://api.github.com/repos/airbnb/nerve/hooks","issue_events_url":"https://api.github.com/repos/airbnb/nerve/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/nerve/events","assignees_url":"https://api.github.com/repos/airbnb/nerve/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/nerve/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/nerve/tags","blobs_url":"https://api.github.com/repos/airbnb/nerve/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/nerve/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/nerve/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/nerve/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/nerve/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/nerve/languages","stargazers_url":"https://api.github.com/repos/airbnb/nerve/stargazers","contributors_url":"https://api.github.com/repos/airbnb/nerve/contributors","subscribers_url":"https://api.github.com/repos/airbnb/nerve/subscribers","subscription_url":"https://api.github.com/repos/airbnb/nerve/subscription","commits_url":"https://api.github.com/repos/airbnb/nerve/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/nerve/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/nerve/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/nerve/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/nerve/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/nerve/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/nerve/merges","archive_url":"https://api.github.com/repos/airbnb/nerve/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/nerve/downloads","issues_url":"https://api.github.com/repos/airbnb/nerve/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/nerve/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/nerve/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/nerve/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/nerve/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/nerve/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/nerve/deployments","created_at":"2013-01-07T23:16:00Z","updated_at":"2016-07-14T21:32:34Z","pushed_at":"2016-07-13T07:30:40Z","git_url":"git://github.com/airbnb/nerve.git","ssh_url":"git@github.com:airbnb/nerve.git","clone_url":"https://github.com/airbnb/nerve.git","svn_url":"https://github.com/airbnb/nerve","homepage":"","size":474,"stargazers_count":687,"watchers_count":687,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":100,"mirror_url":null,"open_issues_count":18,"forks":100,"open_issues":18,"watchers":687,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/80"},"html":{"href":"https://github.com/airbnb/nerve/pull/80"},"issue":{"href":"https://api.github.com/repos/airbnb/nerve/issues/80"},"comments":{"href":"https://api.github.com/repos/airbnb/nerve/issues/80/comments"},"review_comments":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/80/comments"},"review_comment":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/airbnb/nerve/pulls/80/commits"},"statuses":{"href":"https://api.github.com/repos/airbnb/nerve/statuses/3b436fbe1b36ff3ffdeafbcd11b3acfb46d469c4"}}}}
{"actor":{"display_login":"jolynch"}}