{"action":"created","issue":{"url":"https://api.github.com/repos/tazjin/journaldriver/issues/8","repository_url":"https://api.github.com/repos/tazjin/journaldriver","labels_url":"https://api.github.com/repos/tazjin/journaldriver/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/tazjin/journaldriver/issues/8/comments","events_url":"https://api.github.com/repos/tazjin/journaldriver/issues/8/events","html_url":"https://github.com/tazjin/journaldriver/issues/8","id":377144282,"node_id":"MDU6SXNzdWUzNzcxNDQyODI=","number":8,"title":"Occasional panic and crash from AddrNotAvailable","user":{"login":"ttencate","id":90930,"node_id":"MDQ6VXNlcjkwOTMw","avatar_url":"https://avatars0.githubusercontent.com/u/90930?v=4","gravatar_id":"","url":"https://api.github.com/users/ttencate","html_url":"https://github.com/ttencate","followers_url":"https://api.github.com/users/ttencate/followers","following_url":"https://api.github.com/users/ttencate/following{/other_user}","gists_url":"https://api.github.com/users/ttencate/gists{/gist_id}","starred_url":"https://api.github.com/users/ttencate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ttencate/subscriptions","organizations_url":"https://api.github.com/users/ttencate/orgs","repos_url":"https://api.github.com/users/ttencate/repos","events_url":"https://api.github.com/users/ttencate/events{/privacy}","received_events_url":"https://api.github.com/users/ttencate/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-04T11:40:34Z","updated_at":"2018-11-08T08:07:26Z","closed_at":null,"author_association":"NONE","body":"I'm seeing this about once a week (but at unpredictable moments) running journaldriver on a GCE instance:\r\n\r\n```\r\nthread 'main' panicked at 'log receiver encountered an unexpected error: Os { code: 99, kind: AddrNotAvailable, message: \"Cannot assign requested address\" }\r\nstack backtrace:\r\n   0:     0x5638719d00ec - <no info>\r\n   1:     0x5638719cf752 - <no info>\r\n   2:     0x5638719cf30f - <no info>\r\n   3:     0x5638719cede1 - <no info>\r\n   4:     0x5638719cee38 - <no info>\r\n   5:     0x563871841da0 - <no info>\r\n   6:     0x563871836cd2 - <no info>\r\n   7:     0x5638719ee632 - <no info>\r\n   8:     0x5638719fa339 - <no info>\r\n   9:     0x5638719eef4a - <no info>\r\n  10:     0x563871843213 - <no info>\r\n  11:     0x7f1494e762b0 - <no info>\r\n  12:     0x563871820c5d - <no info>\r\n  13:                0x0 - <no info>', libcore/result.rs:945:5\r\nstack backtrace:\r\n   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace\r\n   1: std::sys_common::backtrace::_print\r\n   2: std::panicking::default_hook::{{closure}}\r\n   3: std::panicking::default_hook\r\n   4: std::panicking::rust_panic_with_hook\r\n   5: std::panicking::continue_panic_fmt\r\n   6: rust_begin_unwind\r\n   7: core::panicking::panic_fmt\r\n   8: core::result::unwrap_failed\r\n   9: journaldriver::main\r\n  10: std::rt::lang_start::{{closure}}\r\n  11: std::panicking::try::do_call\r\n  12: __rust_maybe_catch_panic\r\n  13: std::rt::lang_start_internal\r\n  14: main\r\n  15: __libc_start_main\r\n  16: _start\r\njournaldriver.service: Main process exited, code=exited, status=101/n/a\r\n```\r\n\r\nI used `RUST_BACKTRACE=1` in production to capture this trace, but I don't have a reliable way to reproduce the crash on demand.\r\n\r\nAccording to the docs, `AddrNotAvailable` means \"A nonexistent interface was requested or the requested address was not local.\"; not very descriptive. I suppose it corresponds to POSIX's `EADDRNOTAVAIL`, about which we can find a lot more information, e.g. from `connect(2)`:\r\n\r\n> `EADDRNOTAVAIL`\r\n>\r\n> (Internet  domain  sockets)  The  socket  referred  to by sockfd had not previously been bound to an\r\n> address and, upon attempting to bind it to an ephemeral port, it was determined that all  port  num‐\r\n> bers   in   the   ephemeral   port   range   are   currently   in   use.    See  the  discussion  of\r\n> /proc/sys/net/ipv4/ip_local_port_range in ip(7).\r\n\r\nThe crash sometimes happens during a nightly cron job that causes some heavy network traffic; it mirrors the contents of an HTTP server onto Google Cloud Storage. But journaldriver often survives this job, and it sometimes crashes at other moments of the day as well.\r\n\r\nThe cron job isn't parallel, but it does make a lot of HTTP requests in short succession. (The Python `requests` package _should_ [use keep-alive](http://docs.python-requests.org/en/master/) but I'm not sure whether the Google Cloud Storage client library also does that. Would be silly if it didn't!) If each of those connections results in a socket lingering in `TIME_WAIT` state, I can imagine running out of ephemeral ports. If that hypothesis is correct, [this ServerFault answer](https://serverfault.com/a/212127/14335) has some things I could try, in particular setting `net.ipv4.tcp_tw_reuse=1`.\r\n\r\nBut whatever the root cause is, I think journaldriver should not crash, but rather just log the error (probably at \"notice\" or \"warning\" level) and retry later."},"comment":{"url":"https://api.github.com/repos/tazjin/journaldriver/issues/comments/436907290","html_url":"https://github.com/tazjin/journaldriver/issues/8#issuecomment-436907290","issue_url":"https://api.github.com/repos/tazjin/journaldriver/issues/8","id":436907290,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjkwNzI5MA==","user":{"login":"ttencate","id":90930,"node_id":"MDQ6VXNlcjkwOTMw","avatar_url":"https://avatars0.githubusercontent.com/u/90930?v=4","gravatar_id":"","url":"https://api.github.com/users/ttencate","html_url":"https://github.com/ttencate","followers_url":"https://api.github.com/users/ttencate/followers","following_url":"https://api.github.com/users/ttencate/following{/other_user}","gists_url":"https://api.github.com/users/ttencate/gists{/gist_id}","starred_url":"https://api.github.com/users/ttencate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ttencate/subscriptions","organizations_url":"https://api.github.com/users/ttencate/orgs","repos_url":"https://api.github.com/users/ttencate/repos","events_url":"https://api.github.com/users/ttencate/events{/privacy}","received_events_url":"https://api.github.com/users/ttencate/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T08:07:26Z","updated_at":"2018-11-08T08:07:26Z","author_association":"NONE","body":"Going to deploy a debug build now, so we'll hopefully get a proper backtrace for the actual panic as well."}}
{"actor":{"display_login":"ttencate"}}