{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3732","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3732/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3732/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3732/events","html_url":"https://github.com/cockroachdb/cockroach/issues/3732","id":125911185,"number":3732,"title":"Failed tests (10955): TestStoreRangeDownReplicate: infinite loop on empty replica set","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21"},{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/TestStoreRangeDownReplicate","name":"TestStoreRangeDownReplicate","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"tamird","id":1535036,"avatar_url":"https://avatars.githubusercontent.com/u/1535036?v=3","gravatar_id":"","url":"https://api.github.com/users/tamird","html_url":"https://github.com/tamird","followers_url":"https://api.github.com/users/tamird/followers","following_url":"https://api.github.com/users/tamird/following{/other_user}","gists_url":"https://api.github.com/users/tamird/gists{/gist_id}","starred_url":"https://api.github.com/users/tamird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tamird/subscriptions","organizations_url":"https://api.github.com/users/tamird/orgs","repos_url":"https://api.github.com/users/tamird/repos","events_url":"https://api.github.com/users/tamird/events{/privacy}","received_events_url":"https://api.github.com/users/tamird/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-01-11T10:07:29Z","updated_at":"2016-01-11T16:18:53Z","closed_at":null,"body":"The following test appears to have failed:\n\n[#10955](https://circleci.com/gh/cockroachdb/cockroach/10955):\n\n```\nI0111 10:06:40.165730 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\nI0111 10:06:40.165863 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\nI0111 10:06:40.165950 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\nI0111 10:06:40.166047 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\nI0111 10:06:40.166184 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- FAIL: TestStoreRangeDownReplicate (10.23s)\n\tclient_raft_test.go:691: Failed to achieve proper replication within 10 seconds\n=== RUN   TestChangeReplicasDescriptorInvariant\nE0111 10:06:40.167366 960 gossip/gossip.go:185  different node IDs were set for the same gossip instance (2147483647, 1)\nI0111 10:06:40.167525 960 raft/raft.go:446  [group 1] 1 became follower at term 5\nI0111 10:06:40.167572 960 raft/raft.go:234  [group 1] newRaft 1 [peers: [1], term: 5, commit: 10, applied: 10, lastindex: 10, lastterm: 5]\nI0111 10:06:40.167590 960 raft/raft.go:526  [group 1] 1 is starting a new election at term 5\nI0111 10:06:40.167610 960 raft/raft.go:459  [group 1] 1 became candidate at term 6\nI0111 10:06:40.167625 960 raft/raft.go:508  [group 1] 1 received vote from 1 at term 6\nI0111 10:06:40.167646 960 raft/raft.go:487  [group 1] 1 became leader at term 6\nI0111 10:06:40.167870 960 storage/replica.go:1600  gossiping cluster id  from store 1, range 1\n--\nI0111 10:06:51.409945 960 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestLeaderAfterSplit (0.21s)\n\tleaktest.go:110: prior leak detected, leaktest disabled\n=== RUN   Example_rebalancing\n--- PASS: Example_rebalancing (0.13s)\nFAIL\nFAIL\tgithub.com/cockroachdb/cockroach/storage\t25.413s\n=== RUN   TestBatchBasics\nI0111 10:06:57.594844 1029 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchBasics (0.00s)\n=== RUN   TestBatchGet\nI0111 10:06:57.595915 1029 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchGet (0.00s)\n=== RUN   TestBatchMerge\nI0111 10:06:57.599077 1029 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchMerge (0.00s)\n=== RUN   TestBatchProto\n\n```\nPlease assign, take a look and update the issue accordingly."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/170602118","html_url":"https://github.com/cockroachdb/cockroach/issues/3732#issuecomment-170602118","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3732","id":170602118,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-01-11T16:18:53Z","updated_at":"2016-01-11T16:18:53Z","body":"@tamird I **think** this is something slightly different but I'd expect you to run into this one as you deal with the other error. The infinite retry here never sends RPCs, I think the following diff is helpful to get better trace output than that which currently occurs.\r\n\r\n```diff\r\ndiff --git a/kv/dist_sender.go b/kv/dist_sender.go\r\nindex f5eddf9..970d368 100644\r\n--- a/kv/dist_sender.go\r\n+++ b/kv/dist_sender.go\r\n@@ -315,6 +315,7 @@ func (ds *DistSender) getNodeDescriptor() *roachpb.NodeDescriptor {\r\n func (ds *DistSender) sendRPC(trace *tracer.Trace, rangeID roachpb.RangeID, replicas replicaSlice, order rpc.OrderingPolicy,\r\n        ba roachpb.BatchRequest) (*roachpb.BatchResponse, error) {\r\n        if len(replicas) == 0 {\r\n+               trace.Event(\"no node addresses found\")\r\n                return nil, noNodeAddrsAvailError{}\r\n        }\r\n\r\n@@ -567,6 +568,7 @@ func (ds *DistSender) sendChunk(ctx context.Context, ba roachpb.BatchRequest) (*\r\n                        // getDescriptors may fail retryably if the first range isn't\r\n                        // available via Gossip.\r\n                        if pErr != nil {\r\n+                               trace.Event(fmt.Sprintf(\"error looking up descriptor: %T\", pErr))\r\n                                if pErr.Retryable {\r\n                                        if log.V(1) {\r\n                                                log.Warning(pErr)\r\n        ba roachpb.BatchRequest) (*roachpb.BatchResponse, error) {\r\n        if len(replicas) == 0 {\r\n+               trace.Event(\"no node addresses found\")\r\n                return nil, noNodeAddrsAvailError{}\r\n        }\r\n```"}}