{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5658","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5658/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5658/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5658/events","html_url":"https://github.com/cockroachdb/cockroach/issues/5658","id":144269710,"number":5658,"title":"stability: crash on checksum mismatch","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1","html_url":"https://github.com/cockroachdb/cockroach/milestones/Beta","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1/labels","id":778698,"number":1,"title":"Beta","description":"","creator":{"login":"andybons","id":365204,"avatar_url":"https://avatars.githubusercontent.com/u/365204?v=3","gravatar_id":"","url":"https://api.github.com/users/andybons","html_url":"https://github.com/andybons","followers_url":"https://api.github.com/users/andybons/followers","following_url":"https://api.github.com/users/andybons/following{/other_user}","gists_url":"https://api.github.com/users/andybons/gists{/gist_id}","starred_url":"https://api.github.com/users/andybons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andybons/subscriptions","organizations_url":"https://api.github.com/users/andybons/orgs","repos_url":"https://api.github.com/users/andybons/repos","events_url":"https://api.github.com/users/andybons/events{/privacy}","received_events_url":"https://api.github.com/users/andybons/received_events","type":"User","site_admin":false},"open_issues":64,"closed_issues":612,"state":"open","created_at":"2014-09-04T16:42:18Z","updated_at":"2016-04-01T15:01:43Z","due_on":null,"closed_at":null},"comments":9,"created_at":"2016-03-29T13:46:08Z","updated_at":"2016-04-01T15:12:02Z","closed_at":null,"body":"scenario:\r\n* 4 nodes running with 4 photos apps. 2 died overnight of OOM and #5621.\r\n* stopped all remaining nodes\r\n* upgraded from sha a52f363 to de433e9\r\n* restarted all nodes (no load yet)\r\n\r\nafter about 6 minutes, two nodes crashed with:\r\n```\r\nF160329 13:34:56.024740 storage/replica_command.go:1617  replica: range=1 [/Min-/System/\"tsd\\x12cr.node.txn.durations-1m-p50\\x00\\x01\\x89\\xf8\\x06/a2\"), checksum mismatch: e = 44a2ac0be2ce7548f90f9ab2730424812aaa49463a8ae4d69441f61494ba289b8c2418128b48d25984b5cad3c96c8cdd5519366b8de949b7154ca8fc1710835b, v = d7a555600dc25fb1ec0ebd8cde1717f229d872d09d6dafc5f4328e302fe1facb5fccc90ac4a1ae2474421661c7881dabd1cc1ec6e476c9d828bd1dcf716ed72f\r\ngoroutine 51 [running]:\r\ngithub.com/cockroachdb/cockroach/util/log.getStacks(0xece8c7b00, 0x0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:858 +0xb8\r\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).outputLogEntry(0x2638060, 0x3, 0x203986a, 0x1a, 0x651, 0xc821886f00, 0x17b)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:722 +0x178\r\ngithub.com/cockroachdb/cockroach/util/log.addStructured(0x0, 0x0, 0xe00000003, 0x2, 0x1a0b8e0, 0x2e, 0xc82210fb38, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/structured.go:57 +0xfc\r\ngithub.com/cockroachdb/cockroach/util/log.logDepth(0x0, 0x0, 0x1, 0xc800000003, 0x1a0b8e0, 0x2e, 0xc82210fb38, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/log.go:73 +0x7b\r\ngithub.com/cockroachdb/cockroach/util/log.Fatalf(0x1a0b8e0, 0x2e, 0xc82210fb38, 0x3, 0x3)\r\n        /go/src/github.com/cockroachdb/cockroach/util/log/log.go:179 +0x66\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).VerifyChecksum(0xc8202b20e0, 0x7f981efb6fa8, 0xc823c3de90, 0xc822110fe8, 0x144053de6628126d, 0x0, 0x0, 0x0, 0x1, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica_command.go:1617 +0x617\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).executeCmd(0xc8202b20e0, 0x7f981f0027c0, 0xc823c3de60, 0xc823c39a00, 0x8, 0x0, 0x7f981efb6fa8, 0xc823c3de90, 0xc822110fe8, 0x144053de6628126d, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica_command.go:168 +0x1b10\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).executeBatch(0xc8202b20e0, 0x7f981f0027c0, 0xc823c3de60, 0xc823c39a00, 0x8, 0x7f981efb6fa8, 0xc823c3de90, 0xc822110fe8, 0x144053de6628126d, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1694 +0x237\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).applyRaftCommandInBatch(0xc8202b20e0, 0x7f981f0027c0, 0xc823c3de60, 0xc823c39a00, 0x8, 0x300000003, 0x6, 0x144053de6628126d, 0x0, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1574 +0x495\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).applyRaftCommand(0xc8202b20e0, 0xc823c39a00, 0x8, 0x7f981f0027c0, 0xc823c3de60, 0x21676, 0x300000003, 0x6, 0x144053de6628126d, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1470 +0x4a7\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).processRaftCommand(0xc8202b20e0, 0xc823c39a00, 0x8, 0x21676, 0x1, 0x300000003, 0x6, 0x144053de6628126d, 0x0, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1435 +0x2be\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc8202b20e0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1299 +0xa49\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:1861 +0x35a\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8202b71f0, 0xc820954910)\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\r\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\r\n```\r\n\r\nThe second node died with the almost the same message, just different `v`:\r\n```\r\nF160329 13:34:53.933187 storage/replica_command.go:1617  replica: range=1 [/Min-/System/\"tsd\\x12cr.node.txn.durations-1m-p50\\x00\\x01\\x89\\xf8\\x06/a2\"), checksum mismatch: e = 44a2ac0be2ce7548f90f9ab2730424812aaa49463a8ae4d69441f61494ba289b8c2418128b48d25984b5cad3c96c8cdd5519366b8de949b7154ca8fc1710835b, v = 59518e3341bab7d6bbf348136faab8bcc951be2bc4b416a849139d3ee5a53683f7729e3f1f2555b4b66c95001930525fd39563ddd3f3bd7aa4a69cce8a8c4a92\r\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/204433496","html_url":"https://github.com/cockroachdb/cockroach/issues/5658#issuecomment-204433496","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5658","id":204433496,"user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"created_at":"2016-04-01T15:12:02Z","updated_at":"2016-04-01T15:12:02Z","body":"Ditto:\r\nbuild sha:\r\n\r\npanics on three nodes:\r\n```\r\n0: ec2-54-209-133-121.compute-1.amazonaws.com\r\nF160401 14:17:37.977187 storage/replica_command.go:1618  replica: range=244 [/System/\"update-\\x8b\"-/System/\"update-\\x8c\"), checksum mismatch: e = f707718ba2218ec269c639997272d859af2bbaa183eb31b763458fda2526c31710c3d581913bc2498df75589220cc7cd3fe19b8fbf077598a7c72051f6797ad1, v = 41571eb11133e38b743604d1c9bd8b25c7747a4e38cc1672897e667c9a64a751fa5dc5193b01a424a4512d63580739f753206381a7d4866ff0cfb9986c9296a8\r\n1: ec2-54-209-69-52.compute-1.amazonaws.com\r\nF160401 13:52:41.551903 storage/replica_command.go:1618  replica: range=1 [/Min-/System/\"tsd\\x12cr.node.sys.cpu.user.ns\\x00\\x01\\x89\\xf8\\x06/\\xa02\"), checksum mismatch: e = 9ae2518eca99f2dded19ec7c40ffe6075cd3cf6857ac86e346baf98d4f2618c6a67f17d8636563a70dcbacf9be0a29b41b89c984420967d51b875755aa7e82f1, v = 009332ef240a94913713f7f5b592d632135a419551ede55ba4fbdb5607f608916bdc00f582252a6c2792deca270254c97407915cfac024b57b6c02f70abd6a6a\r\n2: ec2-54-208-199-194.compute-1.amazonaws.com\r\nF160401 13:52:41.993842 storage/replica_command.go:1618  replica: range=1 [/Min-/System/\"tsd\\x12cr.node.sys.cpu.user.ns\\x00\\x01\\x89\\xf8\\x06/\\xa02\"), checksum mismatch: e = 9ae2518eca99f2dded19ec7c40ffe6075cd3cf6857ac86e346baf98d4f2618c6a67f17d8636563a70dcbacf9be0a29b41b89c984420967d51b875755aa7e82f1, v = ed460fd220c59380d58d9d03de5de39716ea8255840d698c9eee10207d07eb8422feebdcb113182757f2f14130d5f754fdbc43e984f49e9b1c934f1c09ed1aa2\r\n```\r\n\r\nAll data/binary/logs copied with suffix `.5658-3`"}}