{"action":"created","issue":{"url":"https://api.github.com/repos/zendesk/radar/issues/220","repository_url":"https://api.github.com/repos/zendesk/radar","labels_url":"https://api.github.com/repos/zendesk/radar/issues/220/labels{/name}","comments_url":"https://api.github.com/repos/zendesk/radar/issues/220/comments","events_url":"https://api.github.com/repos/zendesk/radar/issues/220/events","html_url":"https://github.com/zendesk/radar/pull/220","id":133303709,"number":220,"title":"Sentry check \"tilt mode\"","user":{"login":"jden","id":1106489,"avatar_url":"https://avatars.githubusercontent.com/u/1106489?v=3","gravatar_id":"","url":"https://api.github.com/users/jden","html_url":"https://github.com/jden","followers_url":"https://api.github.com/users/jden/followers","following_url":"https://api.github.com/users/jden/following{/other_user}","gists_url":"https://api.github.com/users/jden/gists{/gist_id}","starred_url":"https://api.github.com/users/jden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jden/subscriptions","organizations_url":"https://api.github.com/users/jden/orgs","repos_url":"https://api.github.com/users/jden/repos","events_url":"https://api.github.com/users/jden/events{/privacy}","received_events_url":"https://api.github.com/users/jden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-12T17:53:34Z","updated_at":"2016-02-12T18:47:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/zendesk/radar/pulls/220","html_url":"https://github.com/zendesk/radar/pull/220","diff_url":"https://github.com/zendesk/radar/pull/220.diff","patch_url":"https://github.com/zendesk/radar/pull/220.patch"},"body":"One of the symptoms during a sentry collapse is increased latency\nin the Node.js eventloop, which blocks over events like network IO.\nThis causes sentrys to be late sending their keepalives, but it also\ncauses the sentry system to be late checking the health of remote\nsentries in the cluster. If process A's latency exceeds the sentry\nhealthcheck interval, it may result in every other server in the\ncluster being reported as down, triggering many false client\ndisconnects and a flood of radar messages and redis commands.\n\nThis PR adds a check for clock skew, ensuring that the sentry check\nhappens no more than 1000ms after it was scheduled, otherwise it skips\nchecking on this interval and waits for the next (30s later).\n\nThe idea is borrowed from Redis' Sentinel system http://redis.io/topics/sentinel#tilt-mode\n\nFun fact, the name \"tilt mode\" comes from arcade pinball machines:\nto discourage cheating, they detect if there is an unstable environment\ncaused by the player bumping the table. If there is, it knows that the\nphysical state of the game cannot be trusted, so the controls seize\ntemporarily. Vending machines implement similar protections.\n\n![](http://i.giphy.com/R6bDgXEXCLcIw.gif)\n\nReferences\n========\n- [RADAR-638](https://zendesk.atlassian.net/browse/RADAR-638)\n\nRequired\n========\n- [ ] :+1: from @zendesk/radar\n\nRisks\n========\n- None"},"comment":{"url":"https://api.github.com/repos/zendesk/radar/issues/comments/183446943","html_url":"https://github.com/zendesk/radar/pull/220#issuecomment-183446943","issue_url":"https://api.github.com/repos/zendesk/radar/issues/220","id":183446943,"user":{"login":"vanchi-zendesk","id":1341591,"avatar_url":"https://avatars.githubusercontent.com/u/1341591?v=3","gravatar_id":"","url":"https://api.github.com/users/vanchi-zendesk","html_url":"https://github.com/vanchi-zendesk","followers_url":"https://api.github.com/users/vanchi-zendesk/followers","following_url":"https://api.github.com/users/vanchi-zendesk/following{/other_user}","gists_url":"https://api.github.com/users/vanchi-zendesk/gists{/gist_id}","starred_url":"https://api.github.com/users/vanchi-zendesk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanchi-zendesk/subscriptions","organizations_url":"https://api.github.com/users/vanchi-zendesk/orgs","repos_url":"https://api.github.com/users/vanchi-zendesk/repos","events_url":"https://api.github.com/users/vanchi-zendesk/events{/privacy}","received_events_url":"https://api.github.com/users/vanchi-zendesk/received_events","type":"User","site_admin":false},"created_at":"2016-02-12T18:47:01Z","updated_at":"2016-02-12T18:47:01Z","body":"One good way to deal with this would be to adjust our own keep-alive expiry based on the lag we are seeing. If we see high lag, we double our own expiry when we update it. Another thing we can do is : increase our sentry keep-alive frequency as well."}}