{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/35424","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/35424/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/35424/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/35424/events","html_url":"https://github.com/cockroachdb/cockroach/issues/35424","id":417346515,"node_id":"MDU6SXNzdWU0MTczNDY1MTU=","number":35424,"title":"Fatal consistency check failed with 1 inconsistent replicas","user":{"login":"awoods187","id":22278911,"node_id":"MDQ6VXNlcjIyMjc4OTEx","avatar_url":"https://avatars1.githubusercontent.com/u/22278911?v=4","gravatar_id":"","url":"https://api.github.com/users/awoods187","html_url":"https://github.com/awoods187","followers_url":"https://api.github.com/users/awoods187/followers","following_url":"https://api.github.com/users/awoods187/following{/other_user}","gists_url":"https://api.github.com/users/awoods187/gists{/gist_id}","starred_url":"https://api.github.com/users/awoods187/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awoods187/subscriptions","organizations_url":"https://api.github.com/users/awoods187/orgs","repos_url":"https://api.github.com/users/awoods187/repos","events_url":"https://api.github.com/users/awoods187/events{/privacy}","received_events_url":"https://api.github.com/users/awoods187/received_events","type":"User","site_admin":false},"labels":[{"id":371606733,"node_id":"MDU6TGFiZWwzNzE2MDY3MzM=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/S-1-stability","name":"S-1-stability","color":"d03300","default":false}],"state":"open","locked":false,"assignee":{"login":"andreimatei","id":377201,"node_id":"MDQ6VXNlcjM3NzIwMQ==","avatar_url":"https://avatars1.githubusercontent.com/u/377201?v=4","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"assignees":[{"login":"andreimatei","id":377201,"node_id":"MDQ6VXNlcjM3NzIwMQ==","avatar_url":"https://avatars1.githubusercontent.com/u/377201?v=4","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},{"login":"tbg","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tbg","html_url":"https://github.com/tbg","followers_url":"https://api.github.com/users/tbg/followers","following_url":"https://api.github.com/users/tbg/following{/other_user}","gists_url":"https://api.github.com/users/tbg/gists{/gist_id}","starred_url":"https://api.github.com/users/tbg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbg/subscriptions","organizations_url":"https://api.github.com/users/tbg/orgs","repos_url":"https://api.github.com/users/tbg/repos","events_url":"https://api.github.com/users/tbg/events{/privacy}","received_events_url":"https://api.github.com/users/tbg/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2019-03-05T15:10:27Z","updated_at":"2019-03-06T12:26:53Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Describe the problem**\r\nHit a fatal that killed a node while importing TPC-C 10k on a 7 node 72 cpu machine.\r\n\r\n```\r\nF190305 14:52:51.825973 686646 storage/replica_consistency.go:165  [n3,consistencyChecker,s3,r336/3:/Table/56/1/\"\\x04{\\x02…-W\\\"e…}] consistency check failed with 1 inconsistent replicas\r\ngoroutine 686646 [running]:\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000521100, 0xc0005211a0, 0x5374600, 0x1e)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1018 +0xd4\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x5b160a0, 0xc000000004, 0x5374685, 0x1e, 0xa5, 0xc0184c80a0, 0x7a)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:874 +0x95a\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3a20040, 0xc009930d20, 0x4, 0x2, 0x32ecb3f, 0x36, 0xc0087407d0, 0x1, 0x1)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:85 +0x2d5\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3a20040, 0xc009930d20, 0x1, 0xc000000004, 0x32ecb3f, 0x36, 0xc0087407d0, 0x1, 0x1)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:71 +0x8c\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x3a20040, 0xc009930d20, 0x32ecb3f, 0x36, 0xc0087407d0, 0x1, 0x1)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:182 +0x7e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).CheckConsistency(0xc04529f680, 0x3a20040, 0xc009930d20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_consistency.go:165 +0x863\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).CheckConsistency(0xc04529f680, 0x3a20040, 0xc009930d20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_consistency.go:174 +0x9ce\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*consistencyQueue).process(0xc001124480, 0x3a20040, 0xc009930d20, 0xc04529f680, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/consistency_queue.go:117 +0x1d8\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x3a20040, 0xc009930d20, 0xdf8475800, 0x3a20040)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:753 +0x24e\r\ngithub.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x3a20040, 0xc009930d20, 0xc0348426f0, 0x2c, 0xdf8475800, 0xc00117ee90, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:90 +0xb5\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000756d00, 0x3a20080, 0xc024f38540, 0xc04529f680, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:719 +0x215\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x3a20080, 0xc01d9bdef0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:646 +0xb8\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00055a7e0, 0x3a20080, 0xc01d9bdef0, 0xc00b4524c0, 0x36, 0x0, 0x0, 0xc01b5d09e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134\r\n```\r\n\r\n**To Reproduce**\r\nroachprod create $CLUSTER -n 8 --clouds=aws  --aws-machine-type-ssd=c5d.18xlarge\r\nroachprod run $CLUSTER --  \"DEV=$(mount | grep /mnt/data1 | awk '{print $1}'); sudo umount /mnt/data1; sudo mount -o discard,defaults,nobarrier ${DEV} /mnt/data1/; mount | grep /mnt/data1\"\r\nroachprod stage $CLUSTER:1-7 cockroach\r\nroachprod stage $CLUSTER:8 workload\r\nroachprod start $CLUSTER:1-7 -e COCKROACH_ENGINE_MAX_SYNC_DURATION=24h\r\nroachprod adminurl --open $CLUSTER:1\r\nroachprod run $CLUSTER:1 -- \"./cockroach workload fixtures import tpcc --warehouses=10000 --db=tpcc\"\r\n\r\n**Environment:**\r\nv19.1.0-beta.20190304-135-gd8f7e85\r\n[cockroach.log](https://github.com/cockroachdb/cockroach/files/2931504/cockroach.log)\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/470088878","html_url":"https://github.com/cockroachdb/cockroach/issues/35424#issuecomment-470088878","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/35424","id":470088878,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3MDA4ODg3OA==","user":{"login":"tbg","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tbg","html_url":"https://github.com/tbg","followers_url":"https://api.github.com/users/tbg/followers","following_url":"https://api.github.com/users/tbg/following{/other_user}","gists_url":"https://api.github.com/users/tbg/gists{/gist_id}","starred_url":"https://api.github.com/users/tbg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbg/subscriptions","organizations_url":"https://api.github.com/users/tbg/orgs","repos_url":"https://api.github.com/users/tbg/repos","events_url":"https://api.github.com/users/tbg/events{/privacy}","received_events_url":"https://api.github.com/users/tbg/received_events","type":"User","site_admin":false},"created_at":"2019-03-06T12:26:53Z","updated_at":"2019-03-06T12:26:53Z","author_association":"MEMBER","body":"> A second node died\r\n\r\nLooks like the same inconsistency. This time, the leaseholder had the incorrect stats and compared itself with n4 (the one with the correct stats), so it killed itself. The new leaseholder is also incorrect and so the newly added replica is also incorrect. Effectively the same situation as before."}}
{"actor":{"display_login":"tbg"}}