{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28387","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28387/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28387/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28387/events","html_url":"https://github.com/cockroachdb/cockroach/issues/28387","id":348838909,"node_id":"MDU6SXNzdWUzNDg4Mzg5MDk=","number":28387,"title":"stability: clock skew on GCE","user":{"login":"mberhault","id":5901480,"node_id":"MDQ6VXNlcjU5MDE0ODA=","avatar_url":"https://avatars3.githubusercontent.com/u/5901480?v=4","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-08T18:23:23Z","updated_at":"2018-08-08T18:58:28Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Multiple instances of drifting clocks have been occuring on GCE.\r\n\r\nThe first one was `cockroach-omega-0005`:\r\n![cockroach-omega-0005-skew](https://user-images.githubusercontent.com/5901480/43855850-e0705d1c-9b14-11e8-9047-6430cb5f4ad3.png)\r\n\r\nThis started around 2018-08-01T11:42 UTC. This is suspiciously close to a 11:30 migration.\r\n\r\nSecond: `cockroach-omega-0001`:\r\n![cockroach-omega-0001-skew](https://user-images.githubusercontent.com/5901480/43856076-7c523a20-9b15-11e8-96dd-54c69d13d7d6.png)\r\n\r\nThis started around 2018-08-08T06:07 UTC. Also close to a 05:52 migration.\r\n\r\nThird is `cockroach-rush-us-central-0001`:\r\n![cockroach-rush-0001-skew](https://user-images.githubusercontent.com/5901480/43856172-c9887a5c-9b15-11e8-9241-7d2debec56ad.png)\r\n\r\nThis started around 2018-08-08T14:32 UTC. Migration at 14:25.\r\n\r\nNote that other nodes we migrated around that time for the same clusters.\r\n`cockroach-rush-us-central-0003` almost at the same time as `cockroach-rush-us-central-0001`.\r\nMultiple `cockroach-omega` nodes around the same times.\r\n\r\nOther nodes did not exhibit clock skew. "},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/411515409","html_url":"https://github.com/cockroachdb/cockroach/issues/28387#issuecomment-411515409","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28387","id":411515409,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTUxNTQwOQ==","user":{"login":"mberhault","id":5901480,"node_id":"MDQ6VXNlcjU5MDE0ODA=","avatar_url":"https://avatars3.githubusercontent.com/u/5901480?v=4","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T18:58:28Z","updated_at":"2018-08-08T18:58:28Z","author_association":"CONTRIBUTOR","body":"Interestingly, stopping ntpd **and** running ntpdate does not fix the clock right away (ie: no big jump), but after ntpdate returns, the skew slowly decreases. Did this somehow unwedge the clock skew daemon? There's absolutely nothing in the logs."}}
{"actor":{"display_login":"mberhault"}}