{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarmkit/issues/998","repository_url":"https://api.github.com/repos/docker/swarmkit","labels_url":"https://api.github.com/repos/docker/swarmkit/issues/998/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarmkit/issues/998/comments","events_url":"https://api.github.com/repos/docker/swarmkit/issues/998/events","html_url":"https://github.com/docker/swarmkit/issues/998","id":160569479,"number":998,"title":"Started channel not recreated on Stop","user":{"login":"diogomonica","id":55332,"avatar_url":"https://avatars.githubusercontent.com/u/55332?v=3","gravatar_id":"","url":"https://api.github.com/users/diogomonica","html_url":"https://github.com/diogomonica","followers_url":"https://api.github.com/users/diogomonica/followers","following_url":"https://api.github.com/users/diogomonica/following{/other_user}","gists_url":"https://api.github.com/users/diogomonica/gists{/gist_id}","starred_url":"https://api.github.com/users/diogomonica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diogomonica/subscriptions","organizations_url":"https://api.github.com/users/diogomonica/orgs","repos_url":"https://api.github.com/users/diogomonica/repos","events_url":"https://api.github.com/users/diogomonica/events{/privacy}","received_events_url":"https://api.github.com/users/diogomonica/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-16T03:21:09Z","updated_at":"2016-06-16T03:35:43Z","closed_at":null,"body":"Someone reported this panic:\r\n```\r\nJun 16 10:01:27 i7 docker[12901]: time=\"2016-06-16T10:01:27.620816158+10:00\" level=info msg=\"27a3a6da90ef87ce [quorum:2] has received 2 votes and 0 vote rejections\"\r\nJun 16 10:01:27 i7 docker[12901]: time=\"2016-06-16T10:01:27.620850173+10:00\" level=info msg=\"27a3a6da90ef87ce became leader at term 9678\"\r\nJun 16 10:01:27 i7 docker[12901]: time=\"2016-06-16T10:01:27.620870977+10:00\" level=info msg=\"raft.node: 27a3a6da90ef87ce elected leader 27a3a6da90ef87ce at term 9678\"\r\nJun 16 10:01:27 i7 docker[12901]: panic: close of closed channel\r\nJun 16 10:01:27 i7 docker[12901]: goroutine 70413 [running]:\r\nJun 16 10:01:27 i7 docker[12901]: panic(0x16768e0, 0xc821c82080)\r\nJun 16 10:01:27 i7 docker[12901]: /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\nJun 16 10:01:27 i7 docker[12901]: github.com/docker/swarmkit/ca.(*Server).Run(0xc8210e10e0, 0x7fcedd239760, 0xc821724060, 0x0, 0x0)\r\nJun 16 10:01:27 i7 docker[12901]: /usr/src/docker/vendor/src/github.com/docker/swarmkit/ca/server.go:361 +0x499\r\nJun 16 10:01:27 i7 docker[12901]: github.com/docker/swarmkit/manager.(*Manager).Run.func2.4(0x7fcedd2397d0, 0xc8219ad040, 0xc8210e10e0)\r\nJun 16 10:01:27 i7 docker[12901]: /usr/src/docker/vendor/src/github.com/docker/swarmkit/manager/manager.go:354 +0x4a\r\nJun 16 10:01:27 i7 docker[12901]: created by github.com/docker/swarmkit/manager.(*Manager).Run.func2\r\nJun 16 10:01:27 i7 docker[12901]: /usr/src/docker/vendor/src/github.com/docker/swarmkit/manager/manager.go:357 +0x1384\r\nJun 16 10:01:27 i7 systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nJun 16 10:02:57 i7 systemd[1]: docker.service stop-sigterm timed out. Killing.\r\nJun 16 10:02:57 i7 systemd[1]: Unit docker.service entered failed state.\r\n```\r\n\r\nIt seems to be because the `CAServer` got Stopped and restarted (probably because of promotion or leader election). This caused the `Run()` method to call `close(a.Started)` in `ca/server.go` and panic since it was already closed.\r\n\r\nLooking at the rest of the code, we seem to have the same pattern in `agent` and in `node`, where stop doesn't recreate the channels.\r\n\r\nI think I'm more concerned why a `CAServer` is getting reused and `Run` is being called on it twice though.\r\n\r\n/cc @aaronlehmann @tonistiigi "},"comment":{"url":"https://api.github.com/repos/docker/swarmkit/issues/comments/226380228","html_url":"https://github.com/docker/swarmkit/issues/998#issuecomment-226380228","issue_url":"https://api.github.com/repos/docker/swarmkit/issues/998","id":226380228,"user":{"login":"diogomonica","id":55332,"avatar_url":"https://avatars.githubusercontent.com/u/55332?v=3","gravatar_id":"","url":"https://api.github.com/users/diogomonica","html_url":"https://github.com/diogomonica","followers_url":"https://api.github.com/users/diogomonica/followers","following_url":"https://api.github.com/users/diogomonica/following{/other_user}","gists_url":"https://api.github.com/users/diogomonica/gists{/gist_id}","starred_url":"https://api.github.com/users/diogomonica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diogomonica/subscriptions","organizations_url":"https://api.github.com/users/diogomonica/orgs","repos_url":"https://api.github.com/users/diogomonica/repos","events_url":"https://api.github.com/users/diogomonica/events{/privacy}","received_events_url":"https://api.github.com/users/diogomonica/received_events","type":"User","site_admin":false},"created_at":"2016-06-16T03:35:43Z","updated_at":"2016-06-16T03:35:43Z","body":"@aaronlehmann not sure, that is a great question."}}
{"actor":{"display_login":"diogomonica"}}