{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3906","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3906/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3906/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3906/events","html_url":"https://github.com/cockroachdb/cockroach/issues/3906","id":127016671,"number":3906,"title":"Failed tests (11557): TestReplicateAddAndRemove","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21"},{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/TestReplicateAddAndRemove","name":"TestReplicateAddAndRemove","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-16T11:02:37Z","updated_at":"2016-01-19T17:33:48Z","closed_at":null,"body":"The following test appears to have failed:\n\n[#11557](https://circleci.com/gh/cockroachdb/cockroach/11557):\n\n```\nI0116 11:01:35.720773 952 storage/replica.go:1633  gossiping sentinel from store 1, range 1\nI0116 11:01:35.722002 952 storage/replica.go:1639  gossiping first range from store 1, range 1\nI0116 11:01:35.722693 952 storage/split_queue.go:97  splitting range=1 [/Min-/Max) at keys [/Table/11/0]\nI0116 11:01:35.723076 952 kv/range_cache.go:128  lookup range descriptor: key=/Table/11/0\nI0116 11:01:35.723217 952 kv/range_cache.go:128  lookup range descriptor: key=/Meta2/\"\\x93\\x88\"\npanic: first Range is not present on any store in the multiTestContext. [recovered]\n\tpanic: first Range is not present on any store in the multiTestContext.\n\ngoroutine 6008 [running]:\ngithub.com/cockroachdb/cockroach/util/tracer.(*Trace).Finalize(0xc820053260)\n\t/go/src/github.com/cockroachdb/cockroach/util/tracer/tracer.go:148 +0x227\ngithub.com/cockroachdb/cockroach/storage_test.(*multiTestContext).FirstRange(0xc82016b680, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:407 +0x3b5\ngithub.com/cockroachdb/cockroach/kv.(*rangeDescriptorCache).LookupRangeDescriptor.func1(0xc8204054d0, 0xc8200ab0d0, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/kv/range_cache.go:152 +0x30c\ngithub.com/cockroachdb/cockroach/kv.(*rangeDescriptorCache).LookupRangeDescriptor(0xc8204054d0, 0xc8200ab0d0, 0x3, 0x3, 0x0, 0x1, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/kv/range_cache.go:164 +0x2cf\ngithub.com/cockroachdb/cockroach/kv.(*rangeDescriptorCache).LookupRangeDescriptor.func1(0xc8204054d0, 0xc8200aaab0, 0x2, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/kv/range_cache.go:158 +0x475\n--\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5da0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 1 [chan receive]:\ntesting.RunTests(0x1ccd2d8, 0x23fb200, 0xe0, 0xe0, 0x1)\n\t/tmp/workdir/go/src/testing/testing.go:562 +0xafa\ntesting.(*M).Run(0xc820503f00, 0x1)\n\t/tmp/workdir/go/src/testing/testing.go:494 +0xe5\ngithub.com/cockroachdb/cockroach/util/leaktest.TestMainWithLeakCheck(0xc820503f00)\n\t/go/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:37 +0x2f\ngithub.com/cockroachdb/cockroach/storage_test.TestMain(0xc820503f00)\n\t/go/src/github.com/cockroachdb/cockroach/storage/main_test.go:36 +0x34\nmain.main()\n\tgithub.com/cockroachdb/cockroach/storage/_test/_testmain.go:508 +0x20a\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/tmp/workdir/go/src/runtime/asm_amd64.s:1721 +0x1\n\ngoroutine 5 [chan receive]:\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x27431c0)\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1029 +0x76\ncreated by github.com/cockroachdb/cockroach/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:610 +0x117\n\ngoroutine 6009 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:317 +0x427\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5e00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6014 [runnable]:\ngithub.com/cockroachdb/cockroach/gossip.(*infoStore).processCallbacks.func2(0xc8201f0b00)\n\t/go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:257\ncreated by github.com/cockroachdb/cockroach/gossip.(*infoStore).processCallbacks\n\t/go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:272 +0x56d\n\ngoroutine 5996 [chan receive]:\ngithub.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc8204839a0)\n\t/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:123 +0x5b\ncreated by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open\n\t/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:124 +0x581\n\ngoroutine 5982 [chan receive]:\ngithub.com/cockroachdb/cockroach/util.ListenAndServe.func2()\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:88 +0x7b\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bbea0, 0xc8203f5620)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6012 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:317 +0x427\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc82000e560)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5985 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:626 +0x3ee\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5680)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5994 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:213 +0x208\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bbea0, 0xc82033cd20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5983 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:1588 +0xb48\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc82044f9f0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5981 [IO wait]:\nnet.runtime_pollWait(0x7f0570fb1b98, 0x72, 0xc820037bd0)\n\t/tmp/workdir/go/src/runtime/netpoll.go:157 +0x63\nnet.(*pollDesc).Wait(0xc8200526f0, 0x72, 0x0, 0x0)\n\t/tmp/workdir/go/src/net/fd_poll_runtime.go:73 +0x56\nnet.(*pollDesc).WaitRead(0xc8200526f0, 0x0, 0x0)\n\t/tmp/workdir/go/src/net/fd_poll_runtime.go:78 +0x44\nnet.(*netFD).accept(0xc820052690, 0x0, 0x7f057384f328, 0xc8203f5760)\n\t/tmp/workdir/go/src/net/fd_unix.go:408 +0x2f6\nnet.(*TCPListener).AcceptTCP(0xc820026830, 0x61a030, 0x0, 0x0)\n\t/tmp/workdir/go/src/net/tcpsock_posix.go:254 +0x77\n--\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bbea0, 0xc82041fc50)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6007 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:317 +0x427\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5d80)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5984 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).Start.func3()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:566 +0x15d\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5660)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6011 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:317 +0x427\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5e40)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5993 [runnable]:\nsync.runtime_Semacquire(0xc82033f3bc)\n\t/tmp/workdir/go/src/runtime/sema.go:43 +0x26\nsync.(*WaitGroup).Wait(0xc82033f3b0)\n\t/tmp/workdir/go/src/sync/waitgroup.go:126 +0x118\ngithub.com/cockroachdb/cockroach/storage.(*Store).WaitForInit(0xc82033f200)\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:606 +0x3e\ngithub.com/cockroachdb/cockroach/storage_test.(*multiTestContext).addStore(0xc82016b680)\n\t/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:487 +0xde9\ngithub.com/cockroachdb/cockroach/storage_test.(*multiTestContext).Start(0xc82016b680, 0xc8205c6240, 0x4)\n\t/go/src/github.com/cockroachdb/cockroach/storage/client_test.go:235 +0x1468\n--\ntesting.tRunner(0xc8205c6240, 0x23fc3b8)\n\t/tmp/workdir/go/src/testing/testing.go:456 +0xdd\ncreated by testing.RunTests\n\t/tmp/workdir/go/src/testing/testing.go:561 +0xaa4\n\ngoroutine 6002 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:645 +0x3ee\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f56a0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 5995 [select]:\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8209e3900)\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:214 +0xf1a\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:201 +0x2e\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bbea0, 0xc8203f8b30)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6010 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:317 +0x427\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc8203f5e20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\n\ngoroutine 6015 [runnable]:\ngithub.com/cockroachdb/cockroach/gossip.(*infoStore).processCallbacks.func2(0xc8201f0b00)\n\t/go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:257\ncreated by github.com/cockroachdb/cockroach/gossip.(*infoStore).processCallbacks\n\t/go/src/github.com/cockroachdb/cockroach/gossip/infostore.go:272 +0x56d\n\ngoroutine 6013 [runnable]:\nsync.runtime_Semacquire(0x2743200)\n\t/tmp/workdir/go/src/runtime/sema.go:43 +0x26\nsync.(*Mutex).Lock(0x27431fc)\n\t/tmp/workdir/go/src/sync/mutex.go:82 +0x1ee\ngithub.com/cockroachdb/cockroach/util/log.VDepth(0x6, 0x1, 0xc8203eb100)\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1152 +0x9d\ngithub.com/cockroachdb/cockroach/util/log.V(0x8b00000006, 0xc8202e78c0)\n\t/go/src/github.com/cockroachdb/cockroach/util/log/log.go:184 +0x36\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820245620, 0xece2c1a0f, 0x2b03cc82, 0x2742c40, 0xc8204fdef0, 0xc8200bb500, 0xc8202f9a00, 0x62787c)\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:172 +0xeb\n--\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:216 +0x2fe\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8200bb500, 0xc82000e580)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x60\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x70\nFAIL\tgithub.com/cockroachdb/cockroach/storage\t8.393s\n=== RUN   TestBatchBasics\nI0116 11:01:46.909914 1014 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchBasics (0.01s)\n=== RUN   TestBatchGet\nI0116 11:01:46.915120 1014 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchGet (0.01s)\n=== RUN   TestBatchMerge\nI0116 11:01:46.921665 1014 storage/engine/rocksdb.go:137  closing in-memory rocksdb instance\n--- PASS: TestBatchMerge (0.00s)\n=== RUN   TestBatchProto\n\n```\nPlease assign, take a look and update the issue accordingly."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/172927592","html_url":"https://github.com/cockroachdb/cockroach/issues/3906#issuecomment-172927592","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/3906","id":172927592,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-01-19T17:33:48Z","updated_at":"2016-01-19T17:33:48Z","body":"same error as #3903, probably not specific to that test."}}