{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10733","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10733/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10733/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10733/events","html_url":"https://github.com/cockroachdb/cockroach/issues/10733","id":189750215,"number":10733,"title":"stability: fresh delta cluster experiencing unavailable ranges","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"assignees":[{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2016-11-16T16:02:38Z","updated_at":"2016-11-17T15:30:38Z","closed_at":null,"body":"For the last few days (since we first wiped delta on Nov 14th), every fresh wipe and restart of delta has had issues as soon as block-writer is started.\r\n\r\neg: the latest run was a bit more conservative than before, letting things run for a while.\r\n* started all nodes (from scratch)\r\n* 15 minutes later: start block writer on the first node\r\n* 10 minutes later: start block writer on the second node\r\n* 10 minutes later: start block writer on the third node\r\n* a few minutes later: start block writer on all remaining nodes\r\n\r\nThe moment the last step was performed, unavailable ranges went to 5 on one of the nodes: `104.196.23.238`\r\n\r\nThere's a long-running `replicate` op on that node:\r\n```\r\n2016/11/16 15:24:17.409664\t1516.572132\treplicate\r\n15:24:17.409711\t .    47\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] processing replica\r\n15:24:17.409723\t .    12\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] got range lease\r\n15:24:17.409804\t .    81\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] considering a rebalance\r\n15:24:17.409875\t .    71\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] rebalancing to {NodeID:1 StoreID:1 ReplicaID:0}\r\n15:24:17.409890\t .    15\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 0\r\n15:24:17.409901\t .    12\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:24:17.409911\t .     9\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 1\r\n15:24:17.409919\t .     8\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:24:17.409929\t .    10\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 2\r\n15:24:17.409942\t .    13\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:24:17.409949\t .     7\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 3\r\n15:24:17.409958\t .     8\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:24:17.409965\t .     8\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 4\r\n15:24:17.409992\t .    27\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n...\r\n15:49:34.008687\t .     9\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:49:34.008726\t .     6\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:49:34.008742\t .     8\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot already running\r\n15:49:34.008772\t .     8\t... [n9,replicate,s18,r44/1:/Table/51/1/1{446004…-735854…}] snapshot retry loop pass 87437375\r\n```\r\n\r\nThis matches the time when the last set of block_writers were started.\r\nThis is nodeID 9.\r\n\r\nLooking at its `/debug/raft` page, that ranges shows:\r\n```\r\n    \"44\": {\r\n      \"rangeId\": \"44\",\r\n      \"errors\": [\r\n        {\r\n          \"message\": \"node 10 range descriptor does not match node 9\"\r\n        }\r\n      ],\r\n      \"nodes\": [\r\n        {\r\n          \"nodeId\": 9,\r\n          \"range\": {\r\n            \"span\": {\r\n              \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n              \"endKey\": \"/Table/51/1/1735854988129937498/\\\"12b68893-1568-425f-8d2f-3e3e441ec6ce\\\"/30880\"\r\n            },\r\n            \"raftState\": {\r\n              \"replicaId\": \"1\",\r\n              \"hardState\": {\r\n                \"term\": \"7\",\r\n                \"vote\": \"1\",\r\n                \"commit\": \"234337\"\r\n              },\r\n              \"lead\": \"1\",\r\n              \"state\": \"StateLeader\",\r\n              \"applied\": \"234271\",\r\n              \"progress\": {\r\n                \"1\": {\r\n                  \"match\": \"234337\",\r\n                  \"next\": \"234338\",\r\n                  \"state\": \"ProgressStateProbe\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                },\r\n                \"16\": {\r\n                  \"match\": \"234335\",\r\n                  \"next\": \"234338\",\r\n                  \"state\": \"ProgressStateReplicate\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                },\r\n                \"17\": {\r\n                  \"match\": \"234337\",\r\n                  \"next\": \"234338\",\r\n                  \"state\": \"ProgressStateReplicate\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                }\r\n              }\r\n            },\r\n            \"state\": {\r\n              \"state\": {\r\n                \"raftAppliedIndex\": \"234275\",\r\n                \"leaseAppliedIndex\": \"234143\",\r\n                \"desc\": {\r\n                  \"rangeId\": \"44\",\r\n                  \"startKey\": \"u4n9FBE9TNZcX54SZWE4YzFjMjctNTI0MC00NzY0LTk0MjgtMjI1NjFhNmIxMGJmAAH4AYmP\",\r\n                  \"endKey\": \"u4n9GBb+6i0TKFoSMTJiNjg4OTMtMTU2OC00MjVmLThkMmYtM2UzZTQ0MWVjNmNlAAH3eKA=\",\r\n                  \"replicas\": [\r\n                    {\r\n                      \"nodeId\": 9,\r\n                      \"storeId\": 18,\r\n                      \"replicaId\": 1\r\n                    },\r\n                    {\r\n                      \"nodeId\": 3,\r\n                      \"storeId\": 6,\r\n                      \"replicaId\": 16\r\n                    },\r\n                    {\r\n                      \"nodeId\": 4,\r\n                      \"storeId\": 8,\r\n                      \"replicaId\": 17\r\n                    }\r\n                  ],\r\n                  \"nextReplicaId\": 18\r\n                },\r\n                \"lease\": {\r\n                  \"start\": {\r\n                    \"wallTime\": \"1479309084380384195\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"startStasis\": {\r\n                    \"wallTime\": \"1479309873076966273\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"expiration\": {\r\n                    \"wallTime\": \"1479309873326966273\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"replica\": {\r\n                    \"nodeId\": 9,\r\n                    \"storeId\": 18,\r\n                    \"replicaId\": 1\r\n                  },\r\n                  \"proposedTs\": {\r\n                    \"wallTime\": \"1479309864076972237\",\r\n                    \"logical\": 0\r\n                  }\r\n                },\r\n                \"truncatedState\": {\r\n                  \"index\": \"231264\",\r\n                  \"term\": \"7\"\r\n                },\r\n                \"gcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"stats\": {\r\n                  \"containsEstimates\": false,\r\n                  \"lastUpdateNanos\": \"1479309864673732033\",\r\n                  \"intentAge\": \"782\",\r\n                  \"gcBytesAge\": \"0\",\r\n                  \"liveBytes\": \"51600565\",\r\n                  \"liveCount\": \"363407\",\r\n                  \"keyBytes\": \"24489799\",\r\n                  \"keyCount\": \"363407\",\r\n                  \"valBytes\": \"27110766\",\r\n                  \"valCount\": \"363407\",\r\n                  \"intentBytes\": \"17556\",\r\n                  \"intentCount\": \"201\",\r\n                  \"sysBytes\": \"30861\",\r\n                  \"sysCount\": \"50\"\r\n                },\r\n                \"txnSpanGcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"frozen\": 2\r\n              },\r\n              \"lastIndex\": \"234337\",\r\n              \"numPending\": \"62\",\r\n              \"numDropped\": \"0\",\r\n              \"raftLogSize\": \"4157078\"\r\n            }\r\n          }\r\n        },\r\n        {\r\n          \"nodeId\": 10,\r\n          \"range\": {\r\n            \"span\": {\r\n              \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n              \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n            },\r\n            \"raftState\": {\r\n              \"replicaId\": \"18\",\r\n              \"hardState\": {\r\n                \"term\": \"13\",\r\n                \"vote\": \"20\",\r\n                \"commit\": \"470943\"\r\n              },\r\n              \"lead\": \"20\",\r\n              \"state\": \"StateFollower\",\r\n              \"applied\": \"470943\"\r\n            },\r\n            \"state\": {\r\n              \"state\": {\r\n                \"raftAppliedIndex\": \"470943\",\r\n                \"leaseAppliedIndex\": \"470591\",\r\n                \"desc\": {\r\n                  \"rangeId\": \"44\",\r\n                  \"startKey\": \"u4n9FBE9TNZcX54SZWE4YzFjMjctNTI0MC00NzY0LTk0MjgtMjI1NjFhNmIxMGJmAAH4AYmP\",\r\n                  \"endKey\": \"u4n9FhUipypk8rYSNTZiODBjOTktNDJkMi00MzRiLThhOGYtYjYxMjI3MzUxN2RiAAH4AeJb\",\r\n                  \"replicas\": [\r\n                    {\r\n                      \"nodeId\": 10,\r\n                      \"storeId\": 20,\r\n                      \"replicaId\": 18\r\n                    },\r\n                    {\r\n                      \"nodeId\": 7,\r\n                      \"storeId\": 13,\r\n                      \"replicaId\": 20\r\n                    },\r\n                    {\r\n                      \"nodeId\": 4,\r\n                      \"storeId\": 8,\r\n                      \"replicaId\": 17\r\n                    }\r\n                  ],\r\n                  \"nextReplicaId\": 22\r\n                },\r\n                \"lease\": {\r\n                  \"start\": {\r\n                    \"wallTime\": \"1479311378880480247\",\r\n                    \"logical\": 1\r\n                  },\r\n                  \"startStasis\": {\r\n                    \"wallTime\": \"1479311389924558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"expiration\": {\r\n                    \"wallTime\": \"1479311390174558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"replica\": {\r\n                    \"nodeId\": 7,\r\n                    \"storeId\": 13,\r\n                    \"replicaId\": 20\r\n                  },\r\n                  \"proposedTs\": {\r\n                    \"wallTime\": \"1479311380924570592\",\r\n                    \"logical\": 0\r\n                  }\r\n                },\r\n                \"truncatedState\": {\r\n                  \"index\": \"470938\",\r\n                  \"term\": \"13\"\r\n                },\r\n                \"gcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"stats\": {\r\n                  \"containsEstimates\": false,\r\n                  \"lastUpdateNanos\": \"1479311381761426084\",\r\n                  \"intentAge\": \"343837\",\r\n                  \"gcBytesAge\": \"0\",\r\n                  \"liveBytes\": \"36150276\",\r\n                  \"liveCount\": \"254388\",\r\n                  \"keyBytes\": \"17170701\",\r\n                  \"keyCount\": \"254388\",\r\n                  \"valBytes\": \"18979575\",\r\n                  \"valCount\": \"254388\",\r\n                  \"intentBytes\": \"22419\",\r\n                  \"intentCount\": \"262\",\r\n                  \"sysBytes\": \"39413\",\r\n                  \"sysCount\": \"55\"\r\n                },\r\n                \"txnSpanGcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"frozen\": 2\r\n              },\r\n              \"lastIndex\": \"470943\",\r\n              \"numPending\": \"0\",\r\n              \"numDropped\": \"0\",\r\n              \"raftLogSize\": \"70021836\"\r\n            }\r\n          }\r\n        },\r\n        {\r\n          \"nodeId\": 7,\r\n          \"range\": {\r\n            \"span\": {\r\n              \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n              \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n            },\r\n            \"raftState\": {\r\n              \"replicaId\": \"20\",\r\n              \"hardState\": {\r\n                \"term\": \"13\",\r\n                \"vote\": \"20\",\r\n                \"commit\": \"470943\"\r\n              },\r\n              \"lead\": \"20\",\r\n              \"state\": \"StateLeader\",\r\n              \"applied\": \"470943\",\r\n              \"progress\": {\r\n                \"17\": {\r\n                  \"match\": \"470943\",\r\n                  \"next\": \"470944\",\r\n                  \"state\": \"ProgressStateReplicate\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                },\r\n                \"18\": {\r\n                  \"match\": \"470943\",\r\n                  \"next\": \"470944\",\r\n                  \"state\": \"ProgressStateReplicate\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                },\r\n                \"20\": {\r\n                  \"match\": \"470943\",\r\n                  \"next\": \"470944\",\r\n                  \"state\": \"ProgressStateProbe\",\r\n                  \"paused\": false,\r\n                  \"pendingSnapshot\": \"0\"\r\n                }\r\n              }\r\n            },\r\n            \"state\": {\r\n              \"state\": {\r\n                \"raftAppliedIndex\": \"470943\",\r\n                \"leaseAppliedIndex\": \"470591\",\r\n                \"desc\": {\r\n                  \"rangeId\": \"44\",\r\n                  \"startKey\": \"u4n9FBE9TNZcX54SZWE4YzFjMjctNTI0MC00NzY0LTk0MjgtMjI1NjFhNmIxMGJmAAH4AYmP\",\r\n                  \"endKey\": \"u4n9FhUipypk8rYSNTZiODBjOTktNDJkMi00MzRiLThhOGYtYjYxMjI3MzUxN2RiAAH4AeJb\",\r\n                  \"replicas\": [\r\n                    {\r\n                      \"nodeId\": 10,\r\n                      \"storeId\": 20,\r\n                      \"replicaId\": 18\r\n                    },\r\n                    {\r\n                      \"nodeId\": 7,\r\n                      \"storeId\": 13,\r\n                      \"replicaId\": 20\r\n                    },\r\n                    {\r\n                      \"nodeId\": 4,\r\n                      \"storeId\": 8,\r\n                      \"replicaId\": 17\r\n                    }\r\n                  ],\r\n                  \"nextReplicaId\": 22\r\n                },\r\n                \"lease\": {\r\n                  \"start\": {\r\n                    \"wallTime\": \"1479311378880480247\",\r\n                    \"logical\": 1\r\n                  },\r\n                  \"startStasis\": {\r\n                    \"wallTime\": \"1479311389924558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"expiration\": {\r\n                    \"wallTime\": \"1479311390174558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"replica\": {\r\n                    \"nodeId\": 7,\r\n                    \"storeId\": 13,\r\n                    \"replicaId\": 20\r\n                  },\r\n                  \"proposedTs\": {\r\n                    \"wallTime\": \"1479311380924570592\",\r\n                    \"logical\": 0\r\n                  }\r\n                },\r\n                \"truncatedState\": {\r\n                  \"index\": \"470938\",\r\n                  \"term\": \"13\"\r\n                },\r\n                \"gcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"stats\": {\r\n                  \"containsEstimates\": false,\r\n                  \"lastUpdateNanos\": \"1479311381761426084\",\r\n                  \"intentAge\": \"343837\",\r\n                  \"gcBytesAge\": \"0\",\r\n                  \"liveBytes\": \"36150276\",\r\n                  \"liveCount\": \"254388\",\r\n                  \"keyBytes\": \"17170701\",\r\n                  \"keyCount\": \"254388\",\r\n                  \"valBytes\": \"18979575\",\r\n                  \"valCount\": \"254388\",\r\n                  \"intentBytes\": \"22419\",\r\n                  \"intentCount\": \"262\",\r\n                  \"sysBytes\": \"39413\",\r\n                  \"sysCount\": \"55\"\r\n                },\r\n                \"txnSpanGcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"frozen\": 2\r\n              },\r\n              \"lastIndex\": \"470943\",\r\n              \"numPending\": \"0\",\r\n              \"numDropped\": \"0\",\r\n              \"raftLogSize\": \"11677207\"\r\n            }\r\n          }\r\n        },\r\n        {\r\n          \"nodeId\": 4,\r\n          \"range\": {\r\n            \"span\": {\r\n              \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n              \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n            },\r\n            \"raftState\": {\r\n              \"replicaId\": \"17\",\r\n              \"hardState\": {\r\n                \"term\": \"13\",\r\n                \"vote\": \"20\",\r\n                \"commit\": \"470943\"\r\n              },\r\n              \"lead\": \"20\",\r\n              \"state\": \"StateFollower\",\r\n              \"applied\": \"470943\"\r\n            },\r\n            \"state\": {\r\n              \"state\": {\r\n                \"raftAppliedIndex\": \"470943\",\r\n                \"leaseAppliedIndex\": \"470591\",\r\n                \"desc\": {\r\n                  \"rangeId\": \"44\",\r\n                  \"startKey\": \"u4n9FBE9TNZcX54SZWE4YzFjMjctNTI0MC00NzY0LTk0MjgtMjI1NjFhNmIxMGJmAAH4AYmP\",\r\n                  \"endKey\": \"u4n9FhUipypk8rYSNTZiODBjOTktNDJkMi00MzRiLThhOGYtYjYxMjI3MzUxN2RiAAH4AeJb\",\r\n                  \"replicas\": [\r\n                    {\r\n                      \"nodeId\": 10,\r\n                      \"storeId\": 20,\r\n                      \"replicaId\": 18\r\n                    },\r\n                    {\r\n                      \"nodeId\": 7,\r\n                      \"storeId\": 13,\r\n                      \"replicaId\": 20\r\n                    },\r\n                    {\r\n                      \"nodeId\": 4,\r\n                      \"storeId\": 8,\r\n                      \"replicaId\": 17\r\n                    }\r\n                  ],\r\n                  \"nextReplicaId\": 22\r\n                },\r\n                \"lease\": {\r\n                  \"start\": {\r\n                    \"wallTime\": \"1479311378880480247\",\r\n                    \"logical\": 1\r\n                  },\r\n                  \"startStasis\": {\r\n                    \"wallTime\": \"1479311389924558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"expiration\": {\r\n                    \"wallTime\": \"1479311390174558799\",\r\n                    \"logical\": 0\r\n                  },\r\n                  \"replica\": {\r\n                    \"nodeId\": 7,\r\n                    \"storeId\": 13,\r\n                    \"replicaId\": 20\r\n                  },\r\n                  \"proposedTs\": {\r\n                    \"wallTime\": \"1479311380924570592\",\r\n                    \"logical\": 0\r\n                  }\r\n                },\r\n                \"truncatedState\": {\r\n                  \"index\": \"470938\",\r\n                  \"term\": \"13\"\r\n                },\r\n                \"gcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"stats\": {\r\n                  \"containsEstimates\": false,\r\n                  \"lastUpdateNanos\": \"1479311381761426084\",\r\n                  \"intentAge\": \"343837\",\r\n                  \"gcBytesAge\": \"0\",\r\n                  \"liveBytes\": \"36150276\",\r\n                  \"liveCount\": \"254388\",\r\n                  \"keyBytes\": \"17170701\",\r\n                  \"keyCount\": \"254388\",\r\n                  \"valBytes\": \"18979575\",\r\n                  \"valCount\": \"254388\",\r\n                  \"intentBytes\": \"22419\",\r\n                  \"intentCount\": \"262\",\r\n                  \"sysBytes\": \"39413\",\r\n                  \"sysCount\": \"55\"\r\n                },\r\n                \"txnSpanGcThreshold\": {\r\n                  \"wallTime\": \"0\",\r\n                  \"logical\": 0\r\n                },\r\n                \"frozen\": 2\r\n              },\r\n              \"lastIndex\": \"470943\",\r\n              \"numPending\": \"0\",\r\n              \"numDropped\": \"0\",\r\n              \"raftLogSize\": \"59378075\"\r\n            }\r\n          }\r\n        }\r\n      ]\r\n    },\r\n```\r\n\r\nSome odd details include:\r\n```\r\n                      \"nodeId\": 3,\r\n\"nodeId\": 9,\r\n    \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n    \"endKey\": \"/Table/51/1/1735854988129937498/\\\"12b68893-1568-425f-8d2f-3e3e441ec6ce\\\"/30880\"\r\n    \"state\": \"StateLeader\",\r\n\"nodeId\": 10,\r\n    \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n    \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n    \"state\": \"StateFollower\",\r\n\"nodeId\": 7,\r\n    \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n    \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n    \"state\": \"StateLeader\",\r\n\"nodeId\": 4,\r\n    \"startKey\": \"/Table/51/1/1446004355585826718/\\\"ea8c1c27-5240-4764-9428-22561a6b10bf\\\"/100751\",\r\n    \"endKey\": \"/Table/51/1/1591216144711479990/\\\"56b80c99-42d2-434b-8a8f-b612273517db\\\"/123483\"\r\n    \"state\": \"StateFollower\",\r\n```\r\n\r\nThis could very well just be me being confused about what this should really be showing, but it certainly looks like two nodes think they're leaders of this range."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/261277846","html_url":"https://github.com/cockroachdb/cockroach/issues/10733#issuecomment-261277846","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10733","id":261277846,"user":{"login":"tamird","id":1535036,"avatar_url":"https://avatars.githubusercontent.com/u/1535036?v=3","gravatar_id":"","url":"https://api.github.com/users/tamird","html_url":"https://github.com/tamird","followers_url":"https://api.github.com/users/tamird/followers","following_url":"https://api.github.com/users/tamird/following{/other_user}","gists_url":"https://api.github.com/users/tamird/gists{/gist_id}","starred_url":"https://api.github.com/users/tamird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tamird/subscriptions","organizations_url":"https://api.github.com/users/tamird/orgs","repos_url":"https://api.github.com/users/tamird/repos","events_url":"https://api.github.com/users/tamird/events{/privacy}","received_events_url":"https://api.github.com/users/tamird/received_events","type":"User","site_admin":false},"created_at":"2016-11-17T15:30:38Z","updated_at":"2016-11-17T15:30:38Z","body":"Meta comment: is it time to run a cluster with the `deadlock` tag?"}}
{"actor":{"display_login":"tamird"}}