{"action":"closed","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10806","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10806/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10806/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10806/events","html_url":"https://github.com/cockroachdb/cockroach/issues/10806","id":190482791,"number":10806,"title":"github.com/cockroachdb/cockroach/pkg/storage: TestGossipHandlesReplacedNode failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"avatar_url":"https://avatars.githubusercontent.com/u/20559710?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":335217860,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7","default":false},{"id":191587409,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21","default":false}],"state":"closed","locked":false,"assignee":{"login":"spencerkimball","id":6147149,"avatar_url":"https://avatars.githubusercontent.com/u/6147149?v=3","gravatar_id":"","url":"https://api.github.com/users/spencerkimball","html_url":"https://github.com/spencerkimball","followers_url":"https://api.github.com/users/spencerkimball/followers","following_url":"https://api.github.com/users/spencerkimball/following{/other_user}","gists_url":"https://api.github.com/users/spencerkimball/gists{/gist_id}","starred_url":"https://api.github.com/users/spencerkimball/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spencerkimball/subscriptions","organizations_url":"https://api.github.com/users/spencerkimball/orgs","repos_url":"https://api.github.com/users/spencerkimball/repos","events_url":"https://api.github.com/users/spencerkimball/events{/privacy}","received_events_url":"https://api.github.com/users/spencerkimball/received_events","type":"User","site_admin":false},"assignees":[{"login":"spencerkimball","id":6147149,"avatar_url":"https://avatars.githubusercontent.com/u/6147149?v=3","gravatar_id":"","url":"https://api.github.com/users/spencerkimball","html_url":"https://github.com/spencerkimball","followers_url":"https://api.github.com/users/spencerkimball/followers","following_url":"https://api.github.com/users/spencerkimball/following{/other_user}","gists_url":"https://api.github.com/users/spencerkimball/gists{/gist_id}","starred_url":"https://api.github.com/users/spencerkimball/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spencerkimball/subscriptions","organizations_url":"https://api.github.com/users/spencerkimball/orgs","repos_url":"https://api.github.com/users/spencerkimball/repos","events_url":"https://api.github.com/users/spencerkimball/events{/privacy}","received_events_url":"https://api.github.com/users/spencerkimball/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2016-11-19T07:40:35Z","updated_at":"2016-11-19T10:24:14Z","closed_at":"2016-11-19T10:24:14Z","body":"SHA: https://github.com/cockroachdb/cockroach/commits/509e36d94b447f0e11b69ba32a7e5f095b8b2057\n\nStress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=52249\n\n```\nI161119 07:39:14.510807 28831 gossip/gossip.go:244  [n?] initial resolvers: []\nW161119 07:39:14.510845 28831 gossip/gossip.go:1120  [n?] no resolvers found; use --join to specify a connected node\nW161119 07:39:14.512925 28831 server/status/runtime.go:116  Could not parse build timestamp: parsing time \"\" as \"2006/01/02 15:04:05\": cannot parse \"\" as \"2006\"\nI161119 07:39:14.513122 28831 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161119 07:39:14.513733 28831 server/config.go:443  1 storage engine initialized\nI161119 07:39:14.514046 28831 server/node.go:419  [n?] store [n0,s0] not bootstrapped\nI161119 07:39:14.515716 28940 storage/replica_proposal.go:349  [s1,r1/1:/M{in-ax}] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 410983h39m23.515127897s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=2016-11-19 07:39:14.51568339 +0000 UTC]\nI161119 07:39:14.516648 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161119 07:39:14.516748 28831 server/node.go:348  [n?] **** cluster 3b23228d-7f01-47a9-85eb-1cf5272f617d has been created\nI161119 07:39:14.516764 28831 server/node.go:349  [n?] **** add additional nodes by specifying --join=127.0.0.1:35857\nI161119 07:39:14.517068 28831 base/node_id.go:62  [n1] NodeID set to 1\nI161119 07:39:14.517492 28831 storage/store.go:1188  [n1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\nI161119 07:39:14.517516 28831 server/node.go:432  [n1] initialized store [n1,s1]: {Capacity:536870912 Available:536870912 RangeCount:0 LeaseCount:0}\nI161119 07:39:14.517540 28831 server/node.go:317  [n1] node ID 1 initialized\nI161119 07:39:14.517574 28831 gossip/gossip.go:286  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"tcp\" address_field:\"127.0.0.1:35857\" > attrs:<> locality:<> \nI161119 07:39:14.517765 28831 storage/stores.go:296  [n1] read 0 node addresses from persistent storage\nI161119 07:39:14.517885 28831 server/node.go:562  [n1] connecting to gossip network to verify cluster ID...\nI161119 07:39:14.517909 28831 server/node.go:582  [n1] node connected via gossip and verified as part of cluster \"3b23228d-7f01-47a9-85eb-1cf5272f617d\"\nI161119 07:39:14.517931 28831 server/node.go:367  [n1] node=1: started with [[]=] engine(s) and attributes []\nI161119 07:39:14.517951 28831 server/server.go:630  [n1] starting https server at 127.0.0.1:58960\nI161119 07:39:14.517964 28831 server/server.go:631  [n1] starting grpc/postgres server at 127.0.0.1:35857\nI161119 07:39:14.517976 28831 server/server.go:632  [n1] advertising CockroachDB node at 127.0.0.1:35857\nI161119 07:39:14.518624 28782 storage/split_queue.go:103  [n1,split,s1,r1/1:/M{in-ax}] splitting at keys [/Table/11/0 /Table/12/0 /Table/13/0 /Table/14/0]\nI161119 07:39:14.519463 28782 storage/replica_command.go:2361  [n1,split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /Table/11 [r2]\nE161119 07:39:14.527493 28783 storage/queue.go:586  [n1,replicate,s1,r1/1:/{Min-Table/11}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nE161119 07:39:14.527622 28782 storage/queue.go:575  [n1,split,s1,r1/1:/{Min-Table/11}] unable to split [n1,s1,r1/1:/{Min-Table/11}] at key \"/Table/12/0\": key range /Table/12/0-/Table/12/0 outside of bounds of range /Min-/Max\nE161119 07:39:14.527645 28877 storage/queue.go:586  [n1,replicate,s1,r1/1:/{Min-Table/11}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nI161119 07:39:14.527778 28782 storage/split_queue.go:103  [n1,split,s1,r2/1:/{Table/11-Max}] splitting at keys [/Table/12/0 /Table/13/0 /Table/14/0]\nI161119 07:39:14.527830 28782 storage/replica_command.go:2361  [n1,split,s1,r2/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r3]\nE161119 07:39:14.532707 28783 storage/queue.go:586  [n1,replicate,s1,r2/1:/Table/1{1-2}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nE161119 07:39:14.533805 28782 storage/queue.go:575  [n1,split,s1,r2/1:/Table/1{1-2}] unable to split [n1,s1,r2/1:/Table/1{1-2}] at key \"/Table/13/0\": key range /Table/13/0-/Table/13/0 outside of bounds of range /Table/11-/Max\nI161119 07:39:14.534273 28782 storage/split_queue.go:103  [n1,split,s1,r3/1:/{Table/12-Max}] splitting at keys [/Table/13/0 /Table/14/0]\nI161119 07:39:14.534310 28782 storage/replica_command.go:2361  [n1,split,s1,r3/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r4]\nE161119 07:39:14.543028 28783 storage/queue.go:586  [n1,replicate,s1,r3/1:/Table/1{2-3}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nE161119 07:39:14.543063 28782 storage/queue.go:575  [n1,split,s1,r3/1:/Table/1{2-3}] unable to split [n1,s1,r3/1:/Table/1{2-3}] at key \"/Table/14/0\": key range /Table/14/0-/Table/14/0 outside of bounds of range /Table/12-/Max\nI161119 07:39:14.543167 28782 storage/split_queue.go:103  [n1,split,s1,r4/1:/{Table/13-Max}] splitting at keys [/Table/14/0]\nI161119 07:39:14.543210 28782 storage/replica_command.go:2361  [n1,split,s1,r4/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r5]\nE161119 07:39:14.551843 28783 storage/queue.go:586  [n1,replicate,s1,r4/1:/Table/1{3-4}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nE161119 07:39:14.551945 28783 storage/queue.go:586  [n1,replicate,s1,r5/1:/{Table/14-Max}] purgatory: 0 of 1 store with an attribute matching []; likely not enough nodes in cluster\nI161119 07:39:14.566882 28831 gossip/gossip.go:244  [n?] initial resolvers: [127.0.0.1:35857]\nW161119 07:39:14.566904 28831 gossip/gossip.go:1122  [n?] no incoming or outgoing connections\nW161119 07:39:14.569228 28831 server/status/runtime.go:116  Could not parse build timestamp: parsing time \"\" as \"2006/01/02 15:04:05\": cannot parse \"\" as \"2006\"\nI161119 07:39:14.569417 28831 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161119 07:39:14.574598 28831 server/config.go:443  1 storage engine initialized\nI161119 07:39:14.574887 28831 server/node.go:419  [n?] store [n0,s0] not bootstrapped\nI161119 07:39:14.574909 28831 storage/stores.go:296  [n?] read 0 node addresses from persistent storage\nI161119 07:39:14.574931 28831 server/node.go:562  [n?] connecting to gossip network to verify cluster ID...\nI161119 07:39:14.575654 28999 sql/event_log.go:95  [n1] Event: \"node_join\", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:35857} Attrs: Locality:} ClusterID:3b23228d-7f01-47a9-85eb-1cf5272f617d StartedAt:1479541154517917269}\nI161119 07:39:14.582950 29043 gossip/client.go:125  [n?] started gossip client to 127.0.0.1:35857\nI161119 07:39:14.583292 28984 gossip/server.go:285  [n1] received gossip from unknown node\nI161119 07:39:14.583497 29015 storage/stores.go:312  [n?] wrote 1 node addresses to persistent storage\nI161119 07:39:14.583549 28831 server/node.go:582  [n?] node connected via gossip and verified as part of cluster \"3b23228d-7f01-47a9-85eb-1cf5272f617d\"\nI161119 07:39:14.585530 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:14.586202 28831 server/node.go:310  [n?] new node allocated ID 2\nI161119 07:39:14.586221 28831 base/node_id.go:62  [n2] NodeID set to 2\nI161119 07:39:14.586249 28831 gossip/gossip.go:286  [n2] NodeDescriptor set to node_id:2 address:<network_field:\"tcp\" address_field:\"127.0.0.1:59719\" > attrs:<> locality:<> \nI161119 07:39:14.586305 28831 server/node.go:367  [n2] node=2: started with [[]=] engine(s) and attributes []\nI161119 07:39:14.586329 28831 server/server.go:630  [n2] starting https server at 127.0.0.1:39759\nI161119 07:39:14.586340 28831 server/server.go:631  [n2] starting grpc/postgres server at 127.0.0.1:59719\nI161119 07:39:14.586351 28831 server/server.go:632  [n2] advertising CockroachDB node at 127.0.0.1:59719\nI161119 07:39:14.586574 28985 storage/stores.go:312  [n1] wrote 1 node addresses to persistent storage\nI161119 07:39:14.593463 28904 server/node.go:543  [n2] bootstrapped store [n2,s2]\nI161119 07:39:14.594227 28831 gossip/gossip.go:244  [n?] initial resolvers: [127.0.0.1:35857]\nW161119 07:39:14.594252 28831 gossip/gossip.go:1122  [n?] no incoming or outgoing connections\nI161119 07:39:14.594887 28877 storage/replica_raftstorage.go:453  [n1,replicate,s1,r1/1:/{Min-Table/11}] generated snapshot a44accae at index 34\nI161119 07:39:14.595981 28907 sql/event_log.go:95  [n2] Event: \"node_join\", target: 2, info: {Descriptor:{NodeID:2 Address:{NetworkField:tcp AddressField:127.0.0.1:59719} Attrs: Locality:} ClusterID:3b23228d-7f01-47a9-85eb-1cf5272f617d StartedAt:1479541154586292856}\nW161119 07:39:14.596484 28831 server/status/runtime.go:116  Could not parse build timestamp: parsing time \"\" as \"2006/01/02 15:04:05\": cannot parse \"\" as \"2006\"\nI161119 07:39:14.596717 28831 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161119 07:39:14.597505 28831 server/config.go:443  1 storage engine initialized\nI161119 07:39:14.597844 28831 server/node.go:419  [n?] store [n0,s0] not bootstrapped\nI161119 07:39:14.597875 28831 storage/stores.go:296  [n?] read 0 node addresses from persistent storage\nI161119 07:39:14.597896 28831 server/node.go:562  [n?] connecting to gossip network to verify cluster ID...\nI161119 07:39:14.607057 28991 gossip/client.go:125  [n?] started gossip client to 127.0.0.1:35857\nI161119 07:39:14.607403 29218 gossip/server.go:285  [n1] received gossip from unknown node\nI161119 07:39:14.608561 28831 server/node.go:582  [n?] node connected via gossip and verified as part of cluster \"3b23228d-7f01-47a9-85eb-1cf5272f617d\"\nI161119 07:39:14.608679 28877 storage/store.go:3134  [n1,replicate,s1,r1/1:/{Min-Table/11}] streamed snapshot: kv pairs: 397, log entries: 24\nI161119 07:39:14.608731 29086 storage/stores.go:312  [n?] wrote 1 node addresses to persistent storage\nI161119 07:39:14.608802 29086 storage/stores.go:312  [n?] wrote 2 node addresses to persistent storage\nI161119 07:39:14.609024 29192 storage/replica_raftstorage.go:612  [n2,s2,r1/?:{-}] applying preemptive snapshot at index 34 (id=a44accae, encoded size=188616, 1 rocksdb batches, 24 log entries)\nI161119 07:39:14.609570 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:14.610318 28831 server/node.go:310  [n?] new node allocated ID 3\nI161119 07:39:14.610339 28831 base/node_id.go:62  [n3] NodeID set to 3\nI161119 07:39:14.610369 28831 gossip/gossip.go:286  [n3] NodeDescriptor set to node_id:3 address:<network_field:\"tcp\" address_field:\"127.0.0.1:39458\" > attrs:<> locality:<> \nI161119 07:39:14.610434 28831 server/node.go:367  [n3] node=3: started with [[]=] engine(s) and attributes []\nI161119 07:39:14.610457 28831 server/server.go:630  [n3] starting https server at 127.0.0.1:43315\nI161119 07:39:14.610468 28831 server/server.go:631  [n3] starting grpc/postgres server at 127.0.0.1:39458\nI161119 07:39:14.610479 28831 server/server.go:632  [n3] advertising CockroachDB node at 127.0.0.1:39458\nI161119 07:39:14.610553 29192 storage/replica_raftstorage.go:620  [n2,s2,r1/?:/{Min-Table/11}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=1ms]\nI161119 07:39:14.610661 29184 storage/stores.go:312  [n1] wrote 2 node addresses to persistent storage\nI161119 07:39:14.610853 29072 storage/stores.go:312  [n2] wrote 2 node addresses to persistent storage\nI161119 07:39:14.611490 28877 storage/replica_command.go:3261  [n1,replicate,s1,r1/1:/{Min-Table/11}] change replicas: read existing descriptor range_id:1 start_key:\"\" end_key:\"\\223\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.611841 29113 server/node.go:543  [n3] bootstrapped store [n3,s3]\nI161119 07:39:14.617672 28877 storage/replica_command.go:3261  [n1,replicate,s1,r1/1:/{Min-Table/11}] change replicas: read existing descriptor range_id:1 start_key:\"\" end_key:\"\\223\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.620237 29116 sql/event_log.go:95  [n3] Event: \"node_join\", target: 3, info: {Descriptor:{NodeID:3 Address:{NetworkField:tcp AddressField:127.0.0.1:39458} Attrs: Locality:} ClusterID:3b23228d-7f01-47a9-85eb-1cf5272f617d StartedAt:1479541154610421845}\nI161119 07:39:14.620668 28877 storage/replica.go:2088  [n1,s1,r1/1:/{Min-Table/11}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]\nI161119 07:39:14.621423 28877 storage/replica_raftstorage.go:453  [n1,replicate,s1,r2/1:/Table/1{1-2}] generated snapshot a38873c8 at index 16\nI161119 07:39:14.622933 29249 storage/raft_transport.go:437  [n2] raft transport stream to node 1 established\nI161119 07:39:14.631212 28877 storage/store.go:3134  [n1,replicate,s1,r2/1:/Table/1{1-2}] streamed snapshot: kv pairs: 10, log entries: 6\nI161119 07:39:14.631442 29349 storage/replica_raftstorage.go:612  [n3,s3,r2/?:{-}] applying preemptive snapshot at index 16 (id=a38873c8, encoded size=7293, 1 rocksdb batches, 6 log entries)\nI161119 07:39:14.631769 29349 storage/replica_raftstorage.go:620  [n3,s3,r2/?:/Table/1{1-2}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.632272 28877 storage/replica_command.go:3261  [n1,replicate,s1,r2/1:/Table/1{1-2}] change replicas: read existing descriptor range_id:2 start_key:\"\\223\" end_key:\"\\224\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.636901 28877 storage/replica.go:2088  [n1,s1,r2/1:/Table/1{1-2}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:2}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2}]\nI161119 07:39:14.637548 28877 storage/replica_raftstorage.go:453  [n1,replicate,s1,r3/1:/Table/1{2-3}] generated snapshot ab275af3 at index 22\nI161119 07:39:14.639724 28877 storage/store.go:3134  [n1,replicate,s1,r3/1:/Table/1{2-3}] streamed snapshot: kv pairs: 25, log entries: 12\nI161119 07:39:14.640203 29353 storage/replica_raftstorage.go:612  [n2,s2,r3/?:{-}] applying preemptive snapshot at index 22 (id=ab275af3, encoded size=19221, 1 rocksdb batches, 12 log entries)\nI161119 07:39:14.641294 29353 storage/replica_raftstorage.go:620  [n2,s2,r3/?:/Table/1{2-3}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.641750 28877 storage/replica_command.go:3261  [n1,replicate,s1,r3/1:/Table/1{2-3}] change replicas: read existing descriptor range_id:3 start_key:\"\\224\" end_key:\"\\225\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.642759 29372 storage/raft_transport.go:437  [n3] raft transport stream to node 1 established\nI161119 07:39:14.644994 28877 storage/replica.go:2088  [n1,s1,r3/1:/Table/1{2-3}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]\nI161119 07:39:14.645621 28877 storage/replica_raftstorage.go:453  [n1,replicate,s1,r4/1:/Table/1{3-4}] generated snapshot e27d170d at index 22\nI161119 07:39:14.646365 28877 storage/store.go:3134  [n1,replicate,s1,r4/1:/Table/1{3-4}] streamed snapshot: kv pairs: 54, log entries: 12\nI161119 07:39:14.646690 29376 storage/replica_raftstorage.go:612  [n3,s3,r4/?:{-}] applying preemptive snapshot at index 22 (id=e27d170d, encoded size=22148, 1 rocksdb batches, 12 log entries)\nI161119 07:39:14.647122 29376 storage/replica_raftstorage.go:620  [n3,s3,r4/?:/Table/1{3-4}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.647511 28877 storage/replica_command.go:3261  [n1,replicate,s1,r4/1:/Table/1{3-4}] change replicas: read existing descriptor range_id:4 start_key:\"\\225\" end_key:\"\\226\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.651333 28877 storage/replica.go:2088  [n1,s1,r4/1:/Table/1{3-4}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:2}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2}]\nI161119 07:39:14.651978 28877 storage/replica_raftstorage.go:453  [n1,replicate,s1,r5/1:/{Table/14-Max}] generated snapshot 21fe5a18 at index 11\nI161119 07:39:14.652394 28877 storage/store.go:3134  [n1,replicate,s1,r5/1:/{Table/14-Max}] streamed snapshot: kv pairs: 9, log entries: 1\nI161119 07:39:14.652628 29381 storage/replica_raftstorage.go:612  [n3,s3,r5/?:{-}] applying preemptive snapshot at index 11 (id=21fe5a18, encoded size=504, 1 rocksdb batches, 1 log entries)\nI161119 07:39:14.652890 29381 storage/replica_raftstorage.go:620  [n3,s3,r5/?:/{Table/14-Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.654640 28877 storage/replica_command.go:3261  [n1,replicate,s1,r5/1:/{Table/14-Max}] change replicas: read existing descriptor range_id:5 start_key:\"\\226\" end_key:\"\\377\\377\" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 \nI161119 07:39:14.659809 28877 storage/replica.go:2088  [n1,s1,r5/1:/{Table/14-Max}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:2}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2}]\nI161119 07:39:14.660358 28877 storage/queue.go:638  [n1,replicate] purgatory is now empty\nI161119 07:39:14.660573 28783 storage/replica_raftstorage.go:453  [n1,replicate,s1,r1/1:/{Min-Table/11}] generated snapshot 1b4b8995 at index 54\nI161119 07:39:14.665917 28783 storage/store.go:3134  [n1,replicate,s1,r1/1:/{Min-Table/11}] streamed snapshot: kv pairs: 787, log entries: 44\nI161119 07:39:14.666283 29357 storage/replica_raftstorage.go:612  [n3,s3,r1/?:{-}] applying preemptive snapshot at index 54 (id=1b4b8995, encoded size=403058, 1 rocksdb batches, 44 log entries)\nI161119 07:39:14.669240 29357 storage/replica_raftstorage.go:620  [n3,s3,r1/?:/{Min-Table/11}] applied preemptive snapshot in 3ms [clear=0ms batch=0ms entries=1ms commit=2ms]\nI161119 07:39:14.669762 28783 storage/replica_command.go:3261  [n1,replicate,s1,r1/1:/{Min-Table/11}] change replicas: read existing descriptor range_id:1 start_key:\"\" end_key:\"\\223\" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > next_replica_id:3 \nI161119 07:39:14.673468 28783 storage/replica.go:2088  [n1,s1,r1/1:/{Min-Table/11}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:3}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3}]\nI161119 07:39:14.674719 28783 storage/replica_raftstorage.go:453  [n1,replicate,s1,r2/1:/Table/1{1-2}] generated snapshot de225ca9 at index 19\nI161119 07:39:14.675210 28783 storage/store.go:3134  [n1,replicate,s1,r2/1:/Table/1{1-2}] streamed snapshot: kv pairs: 11, log entries: 9\nI161119 07:39:14.675545 29416 storage/replica_raftstorage.go:612  [n2,s2,r2/?:{-}] applying preemptive snapshot at index 19 (id=de225ca9, encoded size=10080, 1 rocksdb batches, 9 log entries)\nI161119 07:39:14.676264 29416 storage/replica_raftstorage.go:620  [n2,s2,r2/?:/Table/1{1-2}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.677216 28783 storage/replica_command.go:3261  [n1,replicate,s1,r2/1:/Table/1{1-2}] change replicas: read existing descriptor range_id:2 start_key:\"\\223\" end_key:\"\\224\" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:3 store_id:3 replica_id:2 > next_replica_id:3 \nI161119 07:39:14.681834 28783 storage/replica.go:2088  [n1,s1,r2/1:/Table/1{1-2}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:3}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2} {NodeID:2 StoreID:2 ReplicaID:3}]\nI161119 07:39:14.683017 28783 storage/replica_raftstorage.go:453  [n1,replicate,s1,r5/1:/{Table/14-Max}] generated snapshot 0133bc87 at index 14\nI161119 07:39:14.683441 28783 storage/store.go:3134  [n1,replicate,s1,r5/1:/{Table/14-Max}] streamed snapshot: kv pairs: 10, log entries: 4\nI161119 07:39:14.683744 29324 storage/replica_raftstorage.go:612  [n2,s2,r5/?:{-}] applying preemptive snapshot at index 14 (id=0133bc87, encoded size=3298, 1 rocksdb batches, 4 log entries)\nI161119 07:39:14.684072 29324 storage/replica_raftstorage.go:620  [n2,s2,r5/?:/{Table/14-Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.685293 28783 storage/replica_command.go:3261  [n1,replicate,s1,r5/1:/{Table/14-Max}] change replicas: read existing descriptor range_id:5 start_key:\"\\226\" end_key:\"\\377\\377\" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:3 store_id:3 replica_id:2 > next_replica_id:3 \nI161119 07:39:14.691227 28783 storage/replica.go:2088  [n1,s1,r5/1:/{Table/14-Max}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:3}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2} {NodeID:2 StoreID:2 ReplicaID:3}]\nI161119 07:39:14.692408 28783 storage/replica_raftstorage.go:453  [n1,replicate,s1,r4/1:/Table/1{3-4}] generated snapshot 6801ae67 at index 34\nI161119 07:39:14.693216 28783 storage/store.go:3134  [n1,replicate,s1,r4/1:/Table/1{3-4}] streamed snapshot: kv pairs: 80, log entries: 24\nI161119 07:39:14.693469 29168 storage/replica_raftstorage.go:612  [n2,s2,r4/?:{-}] applying preemptive snapshot at index 34 (id=6801ae67, encoded size=41707, 1 rocksdb batches, 24 log entries)\nI161119 07:39:14.694183 29168 storage/replica_raftstorage.go:620  [n2,s2,r4/?:/Table/1{3-4}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.694606 28783 storage/replica_command.go:3261  [n1,replicate,s1,r4/1:/Table/1{3-4}] change replicas: read existing descriptor range_id:4 start_key:\"\\225\" end_key:\"\\226\" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:3 store_id:3 replica_id:2 > next_replica_id:3 \nI161119 07:39:14.699020 28783 storage/replica.go:2088  [n1,s1,r4/1:/Table/1{3-4}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:3}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:3 StoreID:3 ReplicaID:2} {NodeID:2 StoreID:2 ReplicaID:3}]\nI161119 07:39:14.700088 28783 storage/replica_raftstorage.go:453  [n1,replicate,s1,r3/1:/Table/1{2-3}] generated snapshot 3efd4e5e at index 25\nI161119 07:39:14.700648 28783 storage/store.go:3134  [n1,replicate,s1,r3/1:/Table/1{2-3}] streamed snapshot: kv pairs: 26, log entries: 15\nI161119 07:39:14.701161 29337 storage/replica_raftstorage.go:612  [n3,s3,r3/?:{-}] applying preemptive snapshot at index 25 (id=3efd4e5e, encoded size=22008, 1 rocksdb batches, 15 log entries)\nI161119 07:39:14.701671 29337 storage/replica_raftstorage.go:620  [n3,s3,r3/?:/Table/1{2-3}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]\nI161119 07:39:14.702084 28783 storage/replica_command.go:3261  [n1,replicate,s1,r3/1:/Table/1{2-3}] change replicas: read existing descriptor range_id:3 start_key:\"\\224\" end_key:\"\\225\" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > next_replica_id:3 \nI161119 07:39:14.706325 28783 storage/replica.go:2088  [n1,s1,r3/1:/Table/1{2-3}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:3}: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3}]\nI161119 07:39:14.867417 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161119 07:39:14.867445 28831 util/stop/stopper.go:468  quiescing; tasks left:\n1      storage/queue.go:477\nE161119 07:39:14.867547 29010 storage/queue.go:575  [n1,timeSeriesMaintenance,s1,r1/1:/{Min-Table/11}] node unavailable; try another peer\nW161119 07:39:14.867579 29369 storage/raft_transport.go:443  [n1] raft transport stream to node 3 failed: rpc error: code = 13 desc = transport is closing\nW161119 07:39:14.867646 29319 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:3 StoreID:3 ReplicaID:3}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:14.867758 29372 storage/raft_transport.go:443  [n3] raft transport stream to node 1 failed: EOF\nI161119 07:39:14.867776 29208 http2_client.go:1053  transport: http2Client.notifyError got notified that the client transport was broken EOF.\nI161119 07:39:14.867786 29051 http2_client.go:1053  transport: http2Client.notifyError got notified that the client transport was broken EOF.\nI161119 07:39:14.867818 29049 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:35857->127.0.0.1:59413: use of closed network connection\nI161119 07:39:14.867829 29206 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:35857->127.0.0.1:59417: use of closed network connection\nW161119 07:39:14.867851 29249 storage/raft_transport.go:443  [n2] raft transport stream to node 1 failed: EOF\nI161119 07:39:14.867999 28993 /go/src/google.golang.org/grpc/clientconn.go:667  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = \"transport: dial tcp 127.0.0.1:35857: getsockopt: connection refused\"; Reconnecting to {\"127.0.0.1:35857\" <nil>}\nI161119 07:39:14.868147 29045 /go/src/google.golang.org/grpc/clientconn.go:667  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = \"transport: dial tcp 127.0.0.1:35857: getsockopt: connection refused\"; Reconnecting to {\"127.0.0.1:35857\" <nil>}\nI161119 07:39:14.869661 28831 gossip/gossip.go:244  [n?] initial resolvers: [127.0.0.1:59719]\nW161119 07:39:14.869680 28831 gossip/gossip.go:1122  [n?] no incoming or outgoing connections\nW161119 07:39:14.873782 28831 server/status/runtime.go:116  Could not parse build timestamp: parsing time \"\" as \"2006/01/02 15:04:05\": cannot parse \"\" as \"2006\"\nI161119 07:39:14.874045 28831 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161119 07:39:14.875634 28831 server/config.go:443  1 storage engine initialized\nI161119 07:39:14.875943 28831 server/node.go:419  [n?] store [n0,s0] not bootstrapped\nI161119 07:39:14.875965 28831 storage/stores.go:296  [n?] read 0 node addresses from persistent storage\nI161119 07:39:14.875987 28831 server/node.go:562  [n?] connecting to gossip network to verify cluster ID...\nI161119 07:39:14.884793 29339 gossip/client.go:125  [n?] started gossip client to 127.0.0.1:59719\nI161119 07:39:14.884955 29438 gossip/server.go:285  [n2] received gossip from unknown node\nI161119 07:39:14.885254 29460 storage/stores.go:312  [n?] wrote 1 node addresses to persistent storage\nI161119 07:39:14.885318 29460 storage/stores.go:312  [n?] wrote 2 node addresses to persistent storage\nI161119 07:39:14.885384 28831 server/node.go:582  [n?] node connected via gossip and verified as part of cluster \"3b23228d-7f01-47a9-85eb-1cf5272f617d\"\nI161119 07:39:14.886517 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:14.951578 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:15.041729 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:15.257570 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nW161119 07:39:15.569900 29524 gossip/client.go:119  [n2] failed to start gossip client to 127.0.0.1:35857: rpc error: code = 14 desc = grpc: the connection is unavailable\nI161119 07:39:15.608791 29556 gossip/client.go:125  [n3] started gossip client to 127.0.0.1:59719\nI161119 07:39:15.717201 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:16.551971 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:16.570259 29592 gossip/client.go:125  [n2] started gossip client to 127.0.0.1:39458\nI161119 07:39:18.339450 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:19.587297 29556 gossip/client.go:130  [n3] closing client to node 2 (127.0.0.1:59719): stopping outgoing client to node 2 (127.0.0.1:59719); already have incoming\nI161119 07:39:19.587342 29189 gossip/gossip.go:1136  [n3] node has connected to cluster via gossip\nI161119 07:39:19.587488 29189 storage/stores.go:312  [n3] wrote 2 node addresses to persistent storage\nI161119 07:39:20.589088 29648 storage/raft_transport.go:437  [n2] raft transport stream to node 3 established\nI161119 07:39:20.589538 29667 storage/raft_transport.go:437  [n3] raft transport stream to node 2 established\nI161119 07:39:21.088048 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:24.227203 29123 storage/replica_proposal.go:398  [n2,s2,r1/2:/{Min-Table/11}] range [n2,s2,r1/2:/{Min-Table/11}]: transferring raft leadership to replica ID 3\nI161119 07:39:24.227541 29277 storage/replica_proposal.go:349  [n3,s3,r1/3:/{Min-Table/11}] new range lease replica {3 3 3} 2016-11-19 07:39:23.767560408 +0000 UTC 9.708524682s following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 410983h39m23.767560408s [physicalTime=2016-11-19 07:39:24.227506434 +0000 UTC]\nI161119 07:39:25.887961 28831 kv/dist_sender.go:331  [n?] unable to determine this node's attributes for replica selection; node is most likely bootstrapping\nI161119 07:39:25.889676 28831 server/node.go:310  [n?] new node allocated ID 4\nI161119 07:39:25.889695 28831 base/node_id.go:62  [n4] NodeID set to 4\nI161119 07:39:25.889726 28831 gossip/gossip.go:286  [n4] NodeDescriptor set to node_id:4 address:<network_field:\"tcp\" address_field:\"127.0.0.1:35857\" > attrs:<> locality:<> \nI161119 07:39:25.889779 28831 server/node.go:367  [n4] node=4: started with [[]=] engine(s) and attributes []\nI161119 07:39:25.889799 28831 server/server.go:630  [n4] starting https server at 127.0.0.1:37146\nI161119 07:39:25.889807 28831 server/server.go:631  [n4] starting grpc/postgres server at 127.0.0.1:35857\nI161119 07:39:25.889814 28831 server/server.go:632  [n4] advertising CockroachDB node at 127.0.0.1:35857\nI161119 07:39:25.890121 29501 gossip/gossip.go:639  [n2] removing node 1 which was at same address ({tcp 127.0.0.1:35857}) as new node {4 {tcp 127.0.0.1:35857}  }\nI161119 07:39:25.890221 29502 gossip/gossip.go:599  [n2] removed node 1 from gossip\nI161119 07:39:25.890318 29504 gossip/gossip.go:599  [n3] removed node 1 from gossip\nI161119 07:39:25.890371 29504 gossip/gossip.go:639  [n3] removing node 1 which was at same address ({tcp 127.0.0.1:35857}) as new node {4 {tcp 127.0.0.1:35857}  }\nI161119 07:39:25.890398 29691 gossip/gossip.go:599  [n4] removed node 1 from gossip\nI161119 07:39:25.890437 29505 gossip/gossip.go:599  [n3] removed node 1 from gossip\nI161119 07:39:25.890832 29730 gossip/gossip.go:599  [n2] removed node 1 from gossip\nI161119 07:39:25.890943 29450 gossip/gossip.go:599  [n4] removed node 1 from gossip\nI161119 07:39:25.900225 29719 server/node.go:543  [n4] bootstrapped store [n4,s4]\nI161119 07:39:25.906682 29647 storage/raft_transport.go:437  [n2] raft transport stream to node 1 established\nW161119 07:39:25.906859 29703 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:2 StoreID:2 ReplicaID:2}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.906916 29703 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:2 StoreID:2 ReplicaID:2}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.906955 29703 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:2 StoreID:2 ReplicaID:2}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.906979 29702 storage/store.go:2990  [n2,s2] raft error: node 1 claims to not contain store 1 for replica {1 1 1}: store 1 was not found\nW161119 07:39:25.907000 29703 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:2 StoreID:2 ReplicaID:2}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.907057 29703 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:2 StoreID:2 ReplicaID:2}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nI161119 07:39:25.910076 29687 storage/raft_transport.go:437  [n3] raft transport stream to node 1 established\nW161119 07:39:25.910294 29738 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:3 StoreID:3 ReplicaID:3}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.910345 29738 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:3 StoreID:3 ReplicaID:3}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.910412 29738 storage/raft_transport.go:258  unable to accept Raft message from {NodeID:3 StoreID:3 ReplicaID:3}: no handler registered for {NodeID:1 StoreID:1 ReplicaID:1}\nW161119 07:39:25.910471 29806 storage/store.go:2990  [n3,s3] raft error: node 1 claims to not contain store 1 for replica {1 1 1}: store 1 was not found\nI161119 07:39:25.927764 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161119 07:39:25.927804 29709 util/stop/stopper.go:468  quiescing; tasks left:\n1      server/node.go:706\n1      server/node.go:481\nI161119 07:39:25.927933 29707 util/stop/stopper.go:468  quiescing; tasks left:\n1      storage/replica_range_lease.go:159\n1      server/node.go:830\nW161119 07:39:25.927951 29705 storage/replica.go:1809  [n3,s3,r1/3:/{Min-Table/11}] context cancellation of command Put [/System/StatusNode/4,/Min)\nW161119 07:39:25.928023 29648 storage/raft_transport.go:443  [n2] raft transport stream to node 3 failed: rpc error: code = 13 desc = transport is closing\nI161119 07:39:25.928029 29707 util/stop/stopper.go:468  quiescing; tasks left:\n1      storage/replica_range_lease.go:159\nI161119 07:39:25.928055 29708 util/stop/stopper.go:468  quiescing; tasks left:\n1      storage/queue.go:477\n1      server/node.go:830\nW161119 07:39:25.928124 29667 storage/raft_transport.go:443  [n3] raft transport stream to node 2 failed: rpc error: code = 13 desc = transport is closing\nW161119 07:39:25.928199 29827 storage/replica.go:1825  [n2,s2,r3/2:/Table/1{2-3}] shutdown cancellation of command RequestLease [/Table/12,/Min)\nE161119 07:39:25.928241 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: failed to send RPC: sending to all 3 replicas failed; last error: store 1 was not found\nE161119 07:39:25.928323 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: failed to send RPC: sending to all 3 replicas failed; last error: store 1 was not found\nE161119 07:39:25.928349 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:25.928397 29722 server/node.go:743  [n4] node=4: unable to log node_join event: failed to send RPC: sending to all 2 replicas failed; last error: rpc error: code = 9 desc = grpc: the client connection is closing\nW161119 07:39:25.928419 29719 server/node.go:553  [n4] error writing node summary after store bootstrap: failed to send RPC: sending to all 2 replicas failed; last error: rpc error: code = 9 desc = grpc: the client connection is closing\nE161119 07:39:25.928445 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:25.985715 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nW161119 07:39:26.097710 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nW161119 07:39:26.316655 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nW161119 07:39:26.766081 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:27.186666 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:27.210792 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:27.580766 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:27.690136 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:28.986573 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:29.010797 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:29.040039 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:29.490130 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:29.586850 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:29.610958 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nW161119 07:39:30.212222 29267 raft/raft.go:794  [n3,s3,r1/3:/{Min-Table/11}] 3 stepped down to follower since quorum is not active\nE161119 07:39:30.794082 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:30.810724 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:30.890115 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:31.290137 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:31.870903 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:32.586621 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:32.610807 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:33.090187 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:34.386632 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:34.410753 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:34.587403 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:34.611075 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:39:34.890228 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:35.890086 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:36.186568 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:36.212067 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:36.690129 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:36.715797 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:37.986587 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:38.010749 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:38.490173 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:39.586861 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:39.610941 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:39:39.786628 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:39.810768 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:40.290135 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:40.890068 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:41.586627 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:41.610772 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:42.090165 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:42.191537 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:43.386697 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:43.410770 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:43.890261 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:44.586920 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:44.610951 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:39:45.186593 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:45.210765 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:45.690103 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:45.890180 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:46.986591 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:47.006802 29189 gossip/gossip.go:1122  [n3] no incoming or outgoing connections\nE161119 07:39:47.010800 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:47.342760 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:47.490112 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:48.786675 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:48.810793 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:49.290173 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:49.586916 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:49.611429 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:39:50.586603 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:50.610758 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:50.890091 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:51.090174 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:52.386639 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:52.410753 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:52.548564 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:52.890145 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:54.186602 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:54.210785 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:54.586888 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:39:54.611006 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:39:54.690201 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:55.890150 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:39:55.986655 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:56.010767 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:56.490210 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:57.786633 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:57.810791 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:58.246238 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:39:58.290201 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:39:59.586794 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:59.586961 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nE161119 07:39:59.610828 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:39:59.610972 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:40:00.090177 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:00.890189 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:40:01.386645 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:01.410780 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:01.890163 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:02.787634 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:40:03.186578 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:03.211228 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:03.690166 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:04.586855 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:04.611025 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:40:04.986626 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:05.010759 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:05.490125 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:05.890083 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:40:06.786677 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:06.810771 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:07.290233 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:08.306945 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:40:08.586610 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:08.617229 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:09.090208 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:09.586846 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:09.610937 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:40:10.386657 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:10.410773 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:10.890052 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:40:10.890134 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:12.187333 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:12.210792 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:12.690143 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:13.510266 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:40:13.986572 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:14.010813 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:14.490144 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nI161119 07:40:14.586857 28906 gossip/gossip.go:431  [n2] gossip status (ok, 3 nodes)\ngossip client (0/3 cur/max conns)\ngossip server (0/3 cur/max conns, infos 143/46 sent/received, bytes 46586B/15166B sent/received)\nW161119 07:40:14.586960 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:14.589573 29145 storage/store.go:1246  [n2,s2] error gossiping system config: node unavailable; try another peer\nW161119 07:40:14.589623 29144 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:14.589650 29144 storage/store.go:1246  [n2,s2] error gossiping first range descriptor: node unavailable; try another peer\nI161119 07:40:14.610939 29115 gossip/gossip.go:431  [n3] gossip status (ok, 3 nodes)\ngossip client (0/3 cur/max conns)\ngossip server (0/3 cur/max conns, infos 24/51 sent/received, bytes 9666B/15171B sent/received)\nW161119 07:40:14.611036 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nW161119 07:40:14.612216 29289 storage/store.go:1246  [n3,s3] error gossiping system config: node unavailable; try another peer\nW161119 07:40:14.612270 29288 gossip/infostore.go:303  [n3] node unavailable; try another peer\nW161119 07:40:14.612283 29288 storage/store.go:1246  [n3,s3] error gossiping first range descriptor: node unavailable; try another peer\nE161119 07:40:15.786638 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:15.810712 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:15.890190 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:40:16.290103 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:17.586594 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:17.610997 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:18.090120 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:18.577200 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:40:19.386594 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:19.410740 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:19.587105 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:19.610923 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:40:19.890190 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:20.890078 29721 gossip/infostore.go:303  [n4] node unavailable; try another peer\nE161119 07:40:21.186617 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:21.211466 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:21.690286 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:22.891606 29722 server/node.go:743  [n4] node=4: unable to log node_join event: node unavailable; try another peer\nE161119 07:40:22.986999 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:23.010732 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:23.490133 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:24.586893 28906 gossip/infostore.go:303  [n2] node unavailable; try another peer\nW161119 07:40:24.610937 29115 gossip/infostore.go:303  [n3] node unavailable; try another peer\nE161119 07:40:24.786616 28908 storage/node_liveness.go:141  [n2,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:24.810807 29117 storage/node_liveness.go:141  [n3,hb] failed liveness heartbeat: node unavailable; try another peer\nE161119 07:40:25.290191 29723 storage/node_liveness.go:141  [n4,hb] failed liveness heartbeat: node unavailable; try another peer\nW161119 07:40:25.828255 29298 storage/replica.go:1362  [n3,timeSeriesMaintenance,s3,r1/3:/{Min-Table/11}] context deadline exceeded while in command queue: DeleteRange [/System/tsd/insufficient bytes to decode uvarint value,/System/tsd/cr.node.sql.mon.conns.max-p75//10s/2016-10-20T07:00:00Z)\nE161119 07:40:25.828429 29298 storage/queue.go:575  [n3,timeSeriesMaintenance,s3,r1/3:/{Min-Table/11}] context deadline exceeded\nI161119 07:40:25.828453 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161119 07:40:25.829098 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161119 07:40:25.829747 28831 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\n\tleaktest.go:93: Leaked goroutine: goroutine 29814 [chan receive]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).beginCmds.func1(0xc420cbfe20, 0xc420539c00, 0xc42098e160, 0xc42098e168)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1370 +0x64\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/storage.(*Replica).beginCmds\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1376 +0xfde\n```"}}
{"actor":{"display_login":"tamird"}}