{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7224","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7224/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7224/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7224/events","html_url":"https://github.com/cockroachdb/cockroach/issues/7224","id":160277204,"number":7224,"title":"stability: replicaDescCache: clobbering {2 2 5} with {6 6 5}","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/stability","name":"stability","color":"b60205"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":51,"created_at":"2016-06-14T20:33:26Z","updated_at":"2016-06-20T00:31:16Z","closed_at":null,"body":"summary:\r\n* `gamma` cluster upgraded from 093e1e3 to 0b950ce crashed multiple nodes with #7185 \r\n* attempted to roll back to 093e1e3, Nodes 4 (`104.196.98.24`) and 5 (`104.196.39.184`) crashed with:\r\n```\r\nF160613 16:02:44.438105 storage/store.go:2186  store=5:5 ([]=/mnt/data) replicaDescCache: clobbering {6 6 5} with {2 2 5}\r\ngoroutine 104 [running]:\r\ngithub.com/cockroachdb/cockroach/util/log.getStacks(0xecef0d001, 0x0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:862 +0xb8\r\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).outputLogEntry(0x2879de0, 0x3, 0x228d6f3, 0x10, 0x88a, 0xc8249602d0, 0x4a)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:728 +0x9c5\r\ngithub.com/cockroachdb/cockroach/util/log.addStructured(0x0, 0x0, 0xc800000003, 0x2, 0x1b617a0, 0x2a, 0xc822f7b980, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/structured.go:57 +0xfc\r\ngithub.com/cockroachdb/cockroach/util/log.logDepth(0x0, 0x0, 0x1, 0xc800000003, 0x1b617a0, 0x2a, 0xc822f7b980, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/log.go:73 +0x7b\r\ngithub.com/cockroachdb/cockroach/util/log.Fatalf(0x1b617a0, 0x2a, 0xc822f7b980, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/log.go:179 +0x66\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).cacheReplicaDescriptorLocked(0xc82018e280, 0xe53, 0x200000002, 0xc800000005)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2186 +0x288\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).handleRaftMessage(0xc82018e280, 0xc8243dc7e0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:1985 +0x4e5\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2066 +0x552\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820343570, 0xc82139cbb0)\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\r\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\r\n```\r\n\r\nAttempted upgrade to 0b950ce, still crashing with same trace.\r\n\r\nNode 4 is crashing with:\r\n```\r\nF160614 20:26:27.555255 storage/store.go:2243  store=4:4 ([]=/mnt/data) replicaDescCache: clobbering {2 2 5} with {6 6 5}\r\n```\r\n\r\nNode 5 is crashing with:\r\n```\r\nF160614 20:26:31.004285 storage/store.go:2243  store=5:5 ([]=/mnt/data) replicaDescCache: clobbering {6 6 5} with {2 2 5}\r\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/227029848","html_url":"https://github.com/cockroachdb/cockroach/issues/7224#issuecomment-227029848","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7224","id":227029848,"user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2016-06-20T00:31:16Z","updated_at":"2016-06-20T00:31:16Z","body":"Is there a TL;DR summary here? Any ideas of what could be causing this problem?"}}
{"actor":{"display_login":"petermattis"}}