{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6693","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6693/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6693/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6693/events","html_url":"https://github.com/cockroachdb/cockroach/issues/6693","id":155017024,"number":6693,"title":"stability: kv inconsistency","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-05-16T12:10:23Z","updated_at":"2016-05-16T13:18:20Z","closed_at":null,"body":"Build sha: b65e564\r\n\r\nNode 0 (`104.196.125.166`) on the `gamma` cluster ran for about 2 days before failing with:\r\n```\r\nE160515 10:21:45.742759 storage/replica_command.go:1771  replica: range=1292 [/Table/51/1/5863144299210822694/\"9029e5fc-82e6-4cff-9054-6845439af677\"/1650601-/Table/51/1/5867701974934793846/\"3d6154fc-088e-4273-980d-20a926f4cd6e\"/281756), checksum mismatch: e = ccb3d763d6144c7718fc95ac0b6455bbc4316662e7c874f7dfd57d68b913a5f0523671bb257b4aecd3962ee380ffbc5336eee5bf2d2c7fc6c885848bbb1e19ef, v = 2ae408f8e1bca549c058a1f77e367f18f784d17a41a99c2fd6c74381ce189883ac01d54fb03dacf33dfa95b4294746dca1277df7a64f39ee8f4f117914969323\r\nI160515 10:21:49.166925 server/status/runtime.go:160  runtime stats: 2.5 GiB RSS, 178 goroutines, 421 MiB active, 39068.94cgo/sec, 1.78/0.07 %(u/s)time, 0.09 %gc (2x)\r\nE160515 10:21:53.978284 storage/replica_command.go:1761  inconsistent k:v = (/Local/RangeID/1292/r/RangeStats (/Local/RangeID/1292/r/RangeStats), 0.000000000,0, 120408001000180020002800327af6656a1e03090413b91d4a434e1411000000000000000019000000000000000021900a2d020000000029126b01000000000031031d61000000000039126b010000000000418dedcb010000000049126b01000000000051000000000000000059000000000000000061040a000000000000690900000000000000) not present on replica\r\nE160515 10:21:53.978332 storage/replica_command.go:1761  inconsistent k:v = (/Local/RangeID/1292/r/RangeStats (/Local/RangeID/1292/r/RangeStats), 0.000000000,0, 120408001000180020002800327a2259fad903090413b91d4a434e1411000000000000000019000000000000000021900a2d020000000029126b01000000000031031d61000000000039126b010000000000418dedcb010000000049126b01000000000051000000000000000059000000000000000061060a000000000000690900000000000000) not present on leader\r\nE160515 10:21:53.978461 storage/replica_command.go:1761  inconsistent k:v = (/Local/Range/\"\\xbb\\x89\\xfdQ^\\x12\\xad\\x05QT&\\x129029e5fc-82e6-4cff-9054-6845439af677\\x00\\x01\\xf8\\x19/\\xa9\"/Transaction/addrKey:/id:\"c2baf8d6-f2b7-4035-bc96-d5d6f45636ba\" (/Local/Range/\"\\xbb\\x89\\xfdQ^\\x12\\xad\\x05QT&\\x129029e5fc-82e6-4cff-9054-6845439af677\\x00\\x01\\xf8\\x19/\\xa9\"/Transaction/addrKey:/id:\"c2baf8d6-f2b7-4035-bc96-d5d6f45636ba\"), 0.000000000,0, 12040800100018002000280032f6014d80b6f2030a6e0a10c2baf8d6f2b74035bc96d5d6f45636ba10001a40016b12bb89fd515e12ad055154261239303239653566632d383265362d346366662d393035342d36383435343339616636373700ff01f8192fa900017264736320002a0c08cba5babcefa5dfa61410003088de3338034000123973746f726167652f7265706c6963615f636f6d6d616e642e676f3a3234353820282a5265706c696361292e4368616e67655265706c696361732002320c08cba5babcefa5dfa61410003a0c08cba5babcefa5dfa614100042100802120c08cba5babcefa5dfa614100042100804120c08e682ced3efa5dfa614100048016000) not present on replica\r\nE160515 10:21:53.978506 storage/replica_command.go:1761  inconsistent k:v = (/Local/Range/\"\\xbb\\x89\\xfdQ^\\x12\\xad\\x05QT&\\x129029e5fc-82e6-4cff-9054-6845439af677\\x00\\x01\\xf8\\x19/\\xa9\"/Transaction/addrKey:/id:\"c2baf8d6-f2b7-4035-bc96-d5d6f45636ba\" (/Local/Range/\"\\xbb\\x89\\xfdQ^\\x12\\xad\\x05QT&\\x129029e5fc-82e6-4cff-9054-6845439af677\\x00\\x01\\xf8\\x19/\\xa9\"/Transaction/addrKey:/id:\"c2baf8d6-f2b7-4035-bc96-d5d6f45636ba\"), 0.000000000,0, 12040800100018002000280032f801e8752b56030a6e0a10c2baf8d6f2b74035bc96d5d6f45636ba10001a40016b12bb89fd515e12ad055154261239303239653566632d383265362d346366662d393035342d36383435343339616636373700ff01f8192fa900017264736320002a0c08cba5babcefa5dfa61410003088de3338034000123973746f726167652f7265706c6963615f636f6d6d616e642e676f3a3234353820282a5265706c696361292e4368616e67655265706c696361732002320c08cba5babcefa5dfa61410003a0c08cba5babcefa5dfa614100042100802120c08cba5babcefa5dfa614100042100804120c08e682ced3efa5dfa6141000480160006800) not present on leader\r\nF160515 10:21:53.978552 storage/replica_command.go:1771  replica: range=1292 [/Table/51/1/5863144299210822694/\"9029e5fc-82e6-4cff-9054-6845439af677\"/1650601-/Table/51/1/5867701974934793846/\"3d6154fc-088e-4273-980d-20a926f4cd6e\"/281756), checksum mismatch: e = 6795007f016dcbfd5dd1d8ff0ee90cf703625b397b62d5635c2d4be1094eb29babe13113013c5532c308afd1a5be506ef3883deb6dd4c48f2c7aba72d4c046aa, v = 98a9aba9f0f643a605811d7d58b8e48cc1357e87f1fae83c82b85a588b90a373d6618937cf44a27c83fbea670c4e72f2ed59aec7abd85993af59bb5414995566\r\n```\r\n\r\nThis is most likely not an upgrade issue, unless it can manifest itself days later."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/219422421","html_url":"https://github.com/cockroachdb/cockroach/issues/6693#issuecomment-219422421","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6693","id":219422421,"user":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"created_at":"2016-05-16T13:18:20Z","updated_at":"2016-05-16T13:18:20Z","body":"Its happening on the same key \"transaction proto\". Are you sure this was a fresh cluster?"}}