{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/10","repository_url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess","labels_url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/10/comments","events_url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/10/events","html_url":"https://github.com/fluent/fluent-plugin-multiprocess/issues/10","id":168775976,"number":10,"title":"We're using the fluent-plugin-multiprocess plugin and sometimes td-agent binds to 0.0.0.0 instead of 127.0.0.1","user":{"login":"jonathanhle","id":3851599,"avatar_url":"https://avatars.githubusercontent.com/u/3851599?v=3","gravatar_id":"","url":"https://api.github.com/users/jonathanhle","html_url":"https://github.com/jonathanhle","followers_url":"https://api.github.com/users/jonathanhle/followers","following_url":"https://api.github.com/users/jonathanhle/following{/other_user}","gists_url":"https://api.github.com/users/jonathanhle/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathanhle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathanhle/subscriptions","organizations_url":"https://api.github.com/users/jonathanhle/orgs","repos_url":"https://api.github.com/users/jonathanhle/repos","events_url":"https://api.github.com/users/jonathanhle/events{/privacy}","received_events_url":"https://api.github.com/users/jonathanhle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-02T00:52:38Z","updated_at":"2016-08-06T00:55:52Z","closed_at":null,"body":"Hi- \r\n\r\nWe're using the fluent-plugin-multiprocess plugin and sometimes td-agent binds to 0.0.0.0 instead of 127.0.0.1, when we have the following specified in our configs:\r\n\r\n> ```\r\n    ad-events:\r\n      # Defines the content of the source section\r\n      source: {\r\n        type: 'forward',\r\n        port: 24221,\r\n        bind: '127.0.0.1',\r\n        tag: 'gg.adevent'\r\n      }\r\n      # Defines the content of the match section\r\n      match:\r\n        general: {\r\n          type: forward,\r\n          buffer_type: memory,\r\n          buffer_queue_limit: 128,\r\n          send_timeout: 10s,\r\n          recover_wait: 10s,\r\n          heartbeat_interval: 2s,\r\n          phi_threshold: 5,\r\n          hard_timeout: 10s,\r\n          flush_interval: 5s,\r\n          retry_limit: 2\r\n        }\r\n        subsections:\r\n          server: [\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[0] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[0] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            },\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[1] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[1] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            },\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[2] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[2] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            },\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[3] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[3] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            },\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[4] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[4] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            },\r\n            {\r\n              name: \"aggregator-{{ fluentd_aggregator_ips[5] }}\",\r\n              host: \"{{ fluentd_aggregator_ips[5] }}\",\r\n              weight: 33,\r\n              port: 24221\r\n            }\r\n          ]\r\n          # Secondary section will forward back to a local backup socket\r\n          # Which will trigger the standalone mode (Collect and S3 upload)\r\n          # On the client\r\n          secondary: [\r\n            {\r\n              type: 'forward',\r\n              host: '127.0.0.1',\r\n              port: 34221\r\n            }\r\n          ]\r\n```\r\n\r\nThis prevents things from actually shipping logs for us on that specific port, in those times td-agent binds to 0.0.0.0 instead of localhost 127.0.0.1.  The issue recurs often, but is not consistently reproducible for us and the logs just say:\r\n![image](https://cloud.githubusercontent.com/assets/3851599/17313389/082c6728-5810-11e6-8597-c61bf7ab0f9c.png)\r\n> `2016-07-31 12:47:07 -0700 [error]: unexpected error error_class=Errno::EADDRINUSE error=#<Errno::EADDRINUSE: Address already in use - bind(2) for \"127.0.0.1\" port 24226>`\r\n\r\nThe \"fix\"/remediation for the issue has been to just stop/start td-agent, which then allows td-agent to bind to localhost instead of 0.0.0.0.  \r\n\r\nThe versions from the startup of the agent are:\r\n\r\n> 2016-07-31 12:46:52 -0700 [info]: reading config file path=\"/etc/td-agent/td-agent.conf\"\r\n2016-07-31 12:46:52 -0700 [info]: starting fluentd-0.12.26\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-datacounter' version '0.4.5'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-mongo' version '0.7.13'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-multiprocess' version '0.2.1'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-s3' version '0.6.8'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-scribe' version '0.10.14'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-td' version '0.10.28'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'\r\n2016-07-31 12:46:53 -0700 [info]: gem 'fluentd' version '0.12.26'\r\n2016-07-31 12:46:53 -0700 [info]: adding source type=\"multiprocess\"\r\n2016-07-31 12:46:53 -0700 [info]: using configuration file: <ROOT>\r\n\r\nIf you have and advice on what the issue might be, it'd be much appreciated.\r\nBest regards!!!!\r\nJonathan\r\n"},"comment":{"url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/comments/237996608","html_url":"https://github.com/fluent/fluent-plugin-multiprocess/issues/10#issuecomment-237996608","issue_url":"https://api.github.com/repos/fluent/fluent-plugin-multiprocess/issues/10","id":237996608,"user":{"login":"jonathanhle","id":3851599,"avatar_url":"https://avatars.githubusercontent.com/u/3851599?v=3","gravatar_id":"","url":"https://api.github.com/users/jonathanhle","html_url":"https://github.com/jonathanhle","followers_url":"https://api.github.com/users/jonathanhle/followers","following_url":"https://api.github.com/users/jonathanhle/following{/other_user}","gists_url":"https://api.github.com/users/jonathanhle/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathanhle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathanhle/subscriptions","organizations_url":"https://api.github.com/users/jonathanhle/orgs","repos_url":"https://api.github.com/users/jonathanhle/repos","events_url":"https://api.github.com/users/jonathanhle/events{/privacy}","received_events_url":"https://api.github.com/users/jonathanhle/received_events","type":"User","site_admin":false},"created_at":"2016-08-06T00:55:52Z","updated_at":"2016-08-06T00:55:52Z","body":"Hi -\r\nAll of our configuration have td-agent bind to something like this in the config:\r\n`bind: '127.0.0.1'`\r\n\r\nThe times we see it bind to 0.0.0.0 (which we don't want and don't have errors for) is when we get an EADDRINUSE for \"127.0.0.1\".\r\n\r\nERROR EXAMPLE - td-agent didn't come up on both TCP and UDP on localhost. UDP came up on 0.0.0.0 only.  We can see an EADDRINUSE in the logs for 127.0.0.1.  \r\n  Doing an lsof, we'll get something like this:\r\n```\r\non 0.0.0.0 on the port only.\r\n# lsof -i udp:34228\r\nCOMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME\r\nruby    2081 td-agent   11u  IPv4   9505      0t0  UDP *:34228\r\n```\r\n\r\nWORKING EXAMPLE - td-agent comes up on both localhost for both UDP and TCP\r\n```\r\nCOMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME\r\nruby    5520 td-agent   10u  IPv4  19228      0t0  TCP localhost:34228 (LISTEN)\r\nruby    5520 td-agent   11u  IPv4  19229      0t0  UDP localhost:34228\r\n```\r\n\r\nWhen we see td-agent bind to 0.0.0.0 we see the EADDRINUSE for 127.0.0.1.  Binding to 0.0.0.0 is an error for us since we have it configured to bind to 127.0.0.1 in our configurations."}}
{"actor":{"display_login":"jonathanhle"}}