{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3779","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3779/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3779/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3779/events","html_url":"https://github.com/cockroachdb/cockroach/issues/3779","id":126200514,"number":3779,"title":"Clock skew on EC2 leads to cluster death after 13h.","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-12T15:12:05Z","updated_at":"2016-01-12T15:27:32Z","closed_at":null,"body":"3 nodes running on m3-large machines on EC2 (2cpu, 7GB ram, 1x30GB SSD).\r\n\r\nFirst node crashed after 7 hours, the other two at 13h.\r\n\r\nPanic message form each:\r\n```\r\nF0112 01:43:28.873120 7102 rpc/clock_offset.go:179  clock offset from the cluster time for remote clocks: map[172.31.58.172:46131:off=-0.000021365s, err=0.000061756s, at=2016-01-12 01:43:17 +0000 UTC ip-172-31-58-172:26257:off=0.000021365s, err=0.000061756s, at=2016-01-12 01:43:17 +0000 UTC 172.31.58.173:51288:off=-0.500369683s, err=0.000224381s, at=2016-01-12 01:43:19 +0000 UTC 172.31.58.173:51820:off=-0.500390491s, err=0.000233755s, at=2016-01-12 01:43:20 +0000 UTC ip-172-31-58-174:26257:off=-0.193694690s, err=0.000269777s, at=2016-01-12 01:43:20 +0000 UTC 172.31.58.174:39641:off=-0.193795403s, err=0.000280246s, at=2016-01-12 01:43:17 +0000 UTC 172.31.58.174:40138:off=-0.193703714s, err=0.000289655s, at=2016-01-12 01:43:09 +0000 UTC ip-172-31-58-173:26257:off=-0.500349900s, err=0.000272122s, at=2016-01-12 01:43:19 +0000 UTC] is in interval: {-443.964467ms, -250.156736ms}, which indicates that the true offset is greater than 250ms\r\n```\r\n\r\n```\r\nF0112 05:15:59.291313 4884 rpc/clock_offset.go:172  clock offset from the cluster time for remote clocks map[172.31.58.174:46853:off=0.500712869s, err=0.000249557s, at=2016-01-12 05:15:47 +0000 UTC ip-172-31-58-174:26257:off=0.500667038s, err=0.000243563s, at=2016-01-12 05:15:41 +0000 UTC 172.31.58.173:48039:off=-0.000027485s, err=0.000069450s, at=2016-01-12 05:15:46 +0000 UTC ip-172-31-58-173:26257:off=0.000027485s, err=0.000069450s, at=2016-01-12 05:15:46 +0000 UTC] could not be determined: a majority of connected remote clocks have failed to encompass the true time for the cluster\r\n```\r\n\r\n```\r\nF0112 05:16:00.371407 5060 rpc/clock_offset.go:179  clock offset from the cluster time for remote clocks: map[ip-172-31-58-173:26257:off=-0.500712869s, err=0.000249557s, at=2016-01-12 05:15:47 +0000 UTC 172.31.58.174:34904:off=-0.000031205s, err=0.000072812s, at=2016-01-12 05:15:55 +0000 UTC ip-172-31-58-174:26257:off=0.000031205s, err=0.000072812s, at=2016-01-12 05:15:55 +0000 UTC 172.31.58.173:33075:off=-0.500644422s, err=0.000267458s, at=2016-01-12 05:15:41 +0000 UTC 172.31.58.173:59476:off=-0.500667038s, err=0.000243563s, at=2016-01-12 05:15:41 +0000 UTC] is in interval: {-750.910601ms, -250.463312ms}, which indicates that the true offset is greater than 250ms\r\n```\r\n\r\nI have full logs as well."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/170946905","html_url":"https://github.com/cockroachdb/cockroach/issues/3779#issuecomment-170946905","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3779","id":170946905,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-01-12T15:27:32Z","updated_at":"2016-01-12T15:27:32Z","body":"and it looks like the second guy uses a different time server than the other two - maybe doesn't help"}}