{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9749","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9749/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9749/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9749/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9749","id":181183995,"number":9749,"title":"stability: `(*Store).mu` permanently unavailable on a node","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/stability","name":"stability","color":"b60205"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-10-05T15:14:36Z","updated_at":"2016-10-05T16:08:48Z","closed_at":null,"body":"```\r\ngoroutine 25 [semacquire, 14 minutes]:\r\nsync.runtime_Semacquire(0xc42013feac)\r\n        /usr/local/go/src/runtime/sema.go:47 +0x30\r\nsync.(*Mutex).Lock(0xc42013fea8)\r\n        /usr/local/go/src/sync/mutex.go:85 +0xd0\r\ngithub.com/cockroachdb/cockroach/util/syncutil.(*TimedMutex).Lock(0xc42031b150)\r\n        /go/src/github.com/cockroachdb/cockroach/util/syncutil/timedmutex.go:63 +0x30\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).tryGetOrCreateReplica(0xc42031af00, 0x259e, 0xc400000001, 0xc42d142c88, 0x8, 0xc42002cc00, 0xc42033c000, 0x1960248)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2903 +0x54\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).getOrCreateReplica(0xc42031af00, 0x259e, 0x1, 0xc42d142c88, 0xc421180380, 0x0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2880 +0x51\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaftRequest(0xc42031af00, 0x7f8acde4a0b8, 0xc42b4ae030, 0xc42d142c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2346 +0x9a\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRequestQueue(0xc42031af00, 0x259e)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2763 +0x180\r\ngithub.com/cockroachdb/cockroach/storage.(*raftScheduler).worker(0xc4204072d0, 0xc420456090)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/scheduler.go:204 +0x308\r\ngithub.com/cockroachdb/cockroach/storage.(*raftScheduler).Start.func1()\r\n        /go/src/github.com/cockroachdb/cockroach/storage/scheduler.go:160 +0x33\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420456090, 0xc420652e80)\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x7d\r\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:192 +0x66\r\n```\r\n\r\nThis is of course accompanied by thousands of other goroutines which have the same problem.\r\n\r\nWill investigate further.\r\n\r\ncc @BramGruneir @andreimatei "},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/251720832","html_url":"https://github.com/cockroachdb/cockroach/issues/9749#issuecomment-251720832","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9749","id":251720832,"user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2016-10-05T16:08:48Z","updated_at":"2016-10-05T16:08:48Z","body":"Eh? I thought the `newStoreRangeSet(s).Visit` pattern was created to avoid holding `Store.mu` while iterating over replicas. The stack you posted seems fine. We're holding `Store.mu` while grabbing `Replica.mu`. That's kosher, right?"}}
{"actor":{"display_login":"petermattis"}}