{"action":"created","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/2684","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/2684/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/2684/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/2684/events","html_url":"https://github.com/flynn/flynn/issues/2684","id":147865380,"number":2684,"title":"Router fd exhaustion","user":{"login":"titanous","id":13026,"avatar_url":"https://avatars.githubusercontent.com/u/13026?v=3","gravatar_id":"","url":"https://api.github.com/users/titanous","html_url":"https://github.com/titanous","followers_url":"https://api.github.com/users/titanous/followers","following_url":"https://api.github.com/users/titanous/following{/other_user}","gists_url":"https://api.github.com/users/titanous/gists{/gist_id}","starred_url":"https://api.github.com/users/titanous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/titanous/subscriptions","organizations_url":"https://api.github.com/users/titanous/orgs","repos_url":"https://api.github.com/users/titanous/repos","events_url":"https://api.github.com/users/titanous/events{/privacy}","received_events_url":"https://api.github.com/users/titanous/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/flynn/flynn/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/flynn/flynn/labels/router","name":"router","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-04-12T20:33:34Z","updated_at":"2016-06-29T02:34:13Z","closed_at":null,"body":"It looks like the router keeps client connections alive for way too long:\r\n\r\n```text\r\ngoroutine 68281 [IO wait, 9569 minutes]:\r\nnet.(*pollDesc).Wait(0xc208914f40, 0x72, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc208914f40, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc208914ee0, 0xc209dd2000, 0x1000, 0x1000, 0x0, 0x7f07ba754bb0, 0xc20877eff8)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc20a0734b8, 0xc209dd2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/net.go:121 +0xdc\r\nnet/http.(*liveSwitchReader).Read(0xc209a0ba88, 0xc209dd2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/http/server.go:214 +0xab\r\nio.(*LimitedReader).Read(0xc20942efc0, 0xc209dd2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/io/io.go:408 +0xce\r\nbufio.(*Reader).fill(0xc209d3e000)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadSlice(0xc209d3e000, 0xc20803210a, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/bufio/bufio.go:295 +0x257\r\nbufio.(*Reader).ReadLine(0xc209d3e000, 0x0, 0x0, 0x0, 0xc207ea2600, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/bufio/bufio.go:324 +0x62\r\nnet/textproto.(*Reader).readLineSlice(0xc2095d9f20, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/textproto/reader.go:55 +0x9e\r\nnet/textproto.(*Reader).ReadLine(0xc2095d9f20, 0x0, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/textproto/reader.go:36 +0x4f\r\nnet/http.ReadRequest(0xc209d3e000, 0xc208c7f6c0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/http/request.go:598 +0xcb\r\nnet/http.(*conn).readRequest(0xc209a0ba40, 0x0, 0x0, 0x0)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/http/server.go:586 +0x26f\r\nnet/http.(*conn).serve(0xc209a0ba40)\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/http/server.go:1162 +0x69e\r\ncreated by net/http.(*Server).Serve\r\n\t/tmp/tmp.AaXXtTCrZT/src/github.com/flynn/flynn/util/_toolchain/go/src/net/http/server.go:1751 +0x35e\r\n```\r\n\r\n@quentez reports that a cluster that they were running ran out of fds after receiving a lot of traffic.\r\n\r\n\r\n[Goroutine trace](https://github.com/flynn/flynn/files/215884/flynn-router.log.zip)"},"comment":{"url":"https://api.github.com/repos/flynn/flynn/issues/comments/229240703","html_url":"https://github.com/flynn/flynn/issues/2684#issuecomment-229240703","issue_url":"https://api.github.com/repos/flynn/flynn/issues/2684","id":229240703,"user":{"login":"swaits","id":6324,"avatar_url":"https://avatars.githubusercontent.com/u/6324?v=3","gravatar_id":"","url":"https://api.github.com/users/swaits","html_url":"https://github.com/swaits","followers_url":"https://api.github.com/users/swaits/followers","following_url":"https://api.github.com/users/swaits/following{/other_user}","gists_url":"https://api.github.com/users/swaits/gists{/gist_id}","starred_url":"https://api.github.com/users/swaits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/swaits/subscriptions","organizations_url":"https://api.github.com/users/swaits/orgs","repos_url":"https://api.github.com/users/swaits/repos","events_url":"https://api.github.com/users/swaits/events{/privacy}","received_events_url":"https://api.github.com/users/swaits/received_events","type":"User","site_admin":false},"created_at":"2016-06-29T02:34:13Z","updated_at":"2016-06-29T02:34:13Z","body":"@titanous FYI I tried this today. Installer hung on \"reloading\" just after it had started all the various system apps. LMK if you want me to grab a debug info dump from any (or all?) of the hosts, please."}}
{"actor":{"display_login":"swaits"}}