{"action":"opened","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9685","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9685/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9685/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9685/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9685","id":180721520,"number":9685,"title":"storage: TestStoreRangeDownReplicate fails quickly under stress","user":{"login":"jordanlewis","id":43821,"avatar_url":"https://avatars.githubusercontent.com/u/43821?v=3","gravatar_id":"","url":"https://api.github.com/users/jordanlewis","html_url":"https://github.com/jordanlewis","followers_url":"https://api.github.com/users/jordanlewis/followers","following_url":"https://api.github.com/users/jordanlewis/following{/other_user}","gists_url":"https://api.github.com/users/jordanlewis/gists{/gist_id}","starred_url":"https://api.github.com/users/jordanlewis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordanlewis/subscriptions","organizations_url":"https://api.github.com/users/jordanlewis/orgs","repos_url":"https://api.github.com/users/jordanlewis/repos","events_url":"https://api.github.com/users/jordanlewis/events{/privacy}","received_events_url":"https://api.github.com/users/jordanlewis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-03T19:13:17Z","updated_at":"2016-10-03T19:13:17Z","closed_at":null,"body":"```\r\nmake stress PKG=./storage TESTS=TestStoreRangeDownReplicate\r\n\r\ngo test -v  -tags '' -i -c ./storage -o ./storage/stress.test\r\ngithub.com/cockroachdb/cockroach/storage\r\ngithub.com/cockroachdb/cockroach/storage_test\r\ntestmain\r\ncd ./storage && stress -stderr -maxfails 1 ./stress.test -test.run \"TestStoreRangeDownReplicate\" -test.timeout 2m\r\n255 runs so far, 0 failures, over 5s\r\n\r\nI161003 19:08:06.777797 1 rand.go:76  Random seed: -5835408053286116671\r\nI161003 19:08:06.791885 22 storage/engine/rocksdb.go:359  opening in memory rocksdb instance\r\nI161003 19:08:06.854477 22 gossip/gossip.go:234  initial resolvers: []\r\nW161003 19:08:06.854508 22 gossip/gossip.go:1045  no resolvers found; use --join to specify a connected node\r\nI161003 19:08:06.854597 22 gossip/gossip.go:268  NodeID set to 1\r\nI161003 19:08:06.856785 22 gossip/gossip.go:268  NodeID set to 1\r\nI161003 19:08:06.857464 22 storage/store.go:1024  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\r\nI161003 19:08:06.857584 22 gossip/gossip.go:282  NodeDescriptor set to node_id:1 address:<network_field:\"tcp\" address_field:\"127.0.0.1:39579\" > attrs:<>\r\nI161003 19:08:06.861267 77 storage/replica_trigger.go:254  [s1,r1:/M{in-ax}] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=1970-01-01 00:00:00 +0000 UTC]\r\nI161003 19:08:06.862970 22 storage/engine/rocksdb.go:359  opening in memory rocksdb instance\r\nI161003 19:08:06.864268 22 gossip/gossip.go:234  initial resolvers: [127.0.0.1:39579]\r\nW161003 19:08:06.864340 22 gossip/gossip.go:1047  no incoming or outgoing connections\r\nI161003 19:08:06.864411 22 gossip/gossip.go:268  NodeID set to 2\r\nI161003 19:08:06.876703 22 gossip/gossip.go:268  NodeID set to 2\r\nI161003 19:08:06.876889 22 storage/store.go:1024  [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available\r\nI161003 19:08:06.876929 22 gossip/gossip.go:282  NodeDescriptor set to node_id:2 address:<network_field:\"tcp\" address_field:\"127.0.0.1:42977\" > attrs:<>\r\nI161003 19:08:06.878634 22 storage/engine/rocksdb.go:359  opening in memory rocksdb instance\r\nI161003 19:08:06.879459 22 gossip/gossip.go:234  initial resolvers: [127.0.0.1:39579]\r\nW161003 19:08:06.879476 22 gossip/gossip.go:1047  no incoming or outgoing connections\r\nI161003 19:08:06.879516 22 gossip/gossip.go:268  NodeID set to 3\r\nI161003 19:08:06.880896 22 gossip/gossip.go:268  NodeID set to 3\r\nI161003 19:08:06.881060 22 storage/store.go:1024  [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available\r\nI161003 19:08:06.881107 22 gossip/gossip.go:282  NodeDescriptor set to node_id:3 address:<network_field:\"tcp\" address_field:\"127.0.0.1:44662\" > attrs:<>\r\nI161003 19:08:06.882673 259 gossip/client.go:125  node 2: started gossip client to 127.0.0.1:39579\r\nI161003 19:08:06.883823 22 storage/engine/rocksdb.go:359  opening in memory rocksdb instance\r\nI161003 19:08:06.884724 22 gossip/gossip.go:234  initial resolvers: [127.0.0.1:39579]\r\nW161003 19:08:06.884741 22 gossip/gossip.go:1047  no incoming or outgoing connections\r\nI161003 19:08:06.884781 22 gossip/gossip.go:268  NodeID set to 4\r\nI161003 19:08:06.898785 22 gossip/gossip.go:268  NodeID set to 4\r\nI161003 19:08:06.901616 22 storage/store.go:1024  [n4,s4]: failed initial metrics computation: [n4,s4]: system config not yet available\r\nI161003 19:08:06.901700 22 gossip/gossip.go:282  NodeDescriptor set to node_id:4 address:<network_field:\"tcp\" address_field:\"127.0.0.1:34504\" > attrs:<>\r\nI161003 19:08:06.901824 199 gossip/client.go:125  node 3: started gossip client to 127.0.0.1:39579\r\nI161003 19:08:06.905830 22 storage/engine/rocksdb.go:359  opening in memory rocksdb instance\r\nI161003 19:08:06.906666 22 gossip/gossip.go:234  initial resolvers: [127.0.0.1:39579]\r\nW161003 19:08:06.906684 22 gossip/gossip.go:1047  no incoming or outgoing connections\r\nI161003 19:08:06.906725 22 gossip/gossip.go:268  NodeID set to 5\r\nI161003 19:08:06.908113 22 gossip/gossip.go:268  NodeID set to 5\r\nI161003 19:08:06.908299 22 storage/store.go:1024  [n5,s5]: failed initial metrics computation: [n5,s5]: system config not yet available\r\nI161003 19:08:06.908340 22 gossip/gossip.go:282  NodeDescriptor set to node_id:5 address:<network_field:\"tcp\" address_field:\"127.0.0.1:37671\" > attrs:<>\r\nI161003 19:08:06.910550 482 gossip/client.go:125  node 4: started gossip client to 127.0.0.1:39579\r\nI161003 19:08:06.910950 360 gossip/server.go:252  node 1: refusing gossip from node 4 (max 3 conns); forwarding to 5 ({tcp 127.0.0.1:37671})\r\nI161003 19:08:06.911259 482 gossip/client.go:130  node 4: closing client to node 1 (127.0.0.1:39579): received forward from node 1 to 5 (127.0.0.1:37671)\r\nI161003 19:08:06.911457 368 gossip/client.go:125  node 4: started gossip client to 127.0.0.1:37671\r\nI161003 19:08:06.918196 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.918600 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.918791 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.918984 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.919151 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.919305 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.919670 508 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:37671->127.0.0.1:39808: use of closed network connection\r\nI161003 19:08:06.919738 352 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:34504->127.0.0.1:49024: use of closed network connection\r\nI161003 19:08:06.919795 280 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:44662->127.0.0.1:39660: use of closed network connection\r\nI161003 19:08:06.919853 197 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:42977->127.0.0.1:45464: use of closed network connection\r\nI161003 19:08:06.919902 14 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:39579->127.0.0.1:54840: use of closed network connection\r\nI161003 19:08:06.920366 483 gossip/client.go:125  node 5: started gossip client to 127.0.0.1:39579\r\nW161003 19:08:06.920414 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920438 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920454 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920485 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920503 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920520 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920536 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920552 483 gossip/infostore.go:295  node unavailable; try another peer\r\nW161003 19:08:06.920590 483 gossip/infostore.go:295  node unavailable; try another peer\r\nI161003 19:08:06.920665 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.921444 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.921569 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.921670 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\nI161003 19:08:06.921770 610 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks\r\n--- FAIL: TestStoreRangeDownReplicate (0.14s)\r\npanic: [n5,s5]: not connected to gossip [recovered]\r\n\tpanic: [n5,s5]: not connected to gossip [recovered]\r\n\tpanic: [n5,s5]: not connected to gossip\r\n\r\ngoroutine 22 [running]:\r\npanic(0x15ecec0, 0xc420abfbf0)\r\n\t/home/jordan/go/src/runtime/panic.go:500 +0x1a1\r\ntesting.tRunner.func1(0xc420184300)\r\n\t/home/jordan/go/src/testing/testing.go:579 +0x25d\r\npanic(0x15ecec0, 0xc420abfbf0)\r\n\t/home/jordan/go/src/runtime/panic.go:458 +0x243\r\ngithub.com/cockroachdb/cockroach/util/leaktest.AfterTest.func1()\r\n\t/home/jordan/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:73 +0x45c\r\npanic(0x15ecec0, 0xc420abfbf0)\r\n\t/home/jordan/go/src/runtime/panic.go:458 +0x243\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).GossipStore(0xc420694600, 0x7f300aafb108, 0xc420012518, 0x7f300aafb108, 0xc420012518)\r\n\t/home/jordan/src/github.com/cockroachdb/cockroach/storage/store.go:1151 +0x456\r\ngithub.com/cockroachdb/cockroach/storage_test.(*multiTestContext).gossipStores(0xc4204da340)\r\n\t/home/jordan/src/github.com/cockroachdb/cockroach/storage/client_test.go:366 +0x13e\r\ngithub.com/cockroachdb/cockroach/storage_test.(*multiTestContext).initGossipNetwork(0xc4204da340)\r\n\t/home/jordan/src/github.com/cockroachdb/cockroach/storage/client_test.go:395 +0x2f\r\ngithub.com/cockroachdb/cockroach/storage_test.TestStoreRangeDownReplicate(0xc420184300)\r\n\t/home/jordan/src/github.com/cockroachdb/cockroach/storage/client_raft_test.go:969 +0xed\r\ntesting.tRunner(0xc420184300, 0x19575d0)\r\n\t/home/jordan/go/src/testing/testing.go:610 +0x81\r\ncreated by testing.(*T).Run\r\n\t/home/jordan/go/src/testing/testing.go:646 +0x2ec\r\n\r\n\r\nERROR: exit status 2\r\n\r\n558 runs completed, 1 failures, over 10s\r\nFAIL\r\nMakefile:146: recipe for target 'stress' failed\r\nmake: *** [stress] Error 1\r\n```"}}
{"actor":{"display_login":"jordanlewis"}}