{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/36797","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/36797/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/36797/comments","events_url":"https://api.github.com/repos/moby/moby/issues/36797/events","html_url":"https://github.com/moby/moby/issues/36797","id":311686239,"number":36797,"title":"Random no route to host in docker swarm","user":{"login":"alexpirine","id":1610035,"avatar_url":"https://avatars1.githubusercontent.com/u/1610035?v=4","gravatar_id":"","url":"https://api.github.com/users/alexpirine","html_url":"https://github.com/alexpirine","followers_url":"https://api.github.com/users/alexpirine/followers","following_url":"https://api.github.com/users/alexpirine/following{/other_user}","gists_url":"https://api.github.com/users/alexpirine/gists{/gist_id}","starred_url":"https://api.github.com/users/alexpirine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexpirine/subscriptions","organizations_url":"https://api.github.com/users/alexpirine/orgs","repos_url":"https://api.github.com/users/alexpirine/repos","events_url":"https://api.github.com/users/alexpirine/events{/privacy}","received_events_url":"https://api.github.com/users/alexpirine/received_events","type":"User","site_admin":false},"labels":[{"id":237472034,"url":"https://api.github.com/repos/moby/moby/labels/area/networking","name":"area/networking","color":"fbca04","default":false},{"id":392678049,"url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-04-05T16:41:11Z","updated_at":"2018-05-03T12:44:02Z","closed_at":null,"author_association":"NONE","body":"**Description**\r\n\r\nDocker swarm ingress network seems to not be properly working.\r\n\r\nI have a simple nginx web server exposing ports 443 & 80.\r\n\r\nHalf of the requests time out.\r\n\r\nIf I connect to the worker node:\r\n\r\n```\r\n$ for i in {1..500000}; do curl --connect-timeout 0.5 172.18.0.2 && echo; done\r\ncurl: (7) Failed to connect to 172.18.0.2 port 80: No route to host\r\nok\r\ncurl: (28) Connection timed out after 500 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 501 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 500 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 502 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 500 milliseconds\r\nok\r\ncurl: (7) Failed to connect to 172.18.0.2 port 80: No route to host\r\nok\r\ncurl: (28) Connection timed out after 501 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 502 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 500 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 502 milliseconds\r\nok\r\ncurl: (28) Connection timed out after 501 milliseconds\r\nok\r\n```\r\n\r\nNotice how half of the connections succeed (nginx responds with _ok_) and the other ones time out (if I wait long enough I get _no route to host_)\r\n\r\nThe interesting thing is that if I run this loop in the background, now most of my external requests succeed. As if the loop was keeping all the \"bad\" connection busy… How is it possible?\r\n\r\nI tried to reduce number of replicas to 1 and number of worker nodes to 1, same results.\r\n\r\n**Steps to reproduce the issue:**\r\n\r\nI do have a bunch of other services services so I can not share a reproductible config right now, but will try to.\r\n\r\n**Describe the results you received:**\r\n\r\ntime out / no route to host\r\n\r\n**Describe the results you expected:**\r\n\r\nnginx \"ok\" response\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\nI am able to replicate the issue by creating a fresh manager node, with a fresh install, and fresh worker nodes.\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\n# docker --version\r\nDocker version 18.03.0-ce, build 0520e24\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\n# docker info\r\nContainers: 27\r\n Running: 25\r\n Paused: 0\r\n Stopped: 2\r\nImages: 10\r\nServer Version: 18.03.0-ce\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 125\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: active\r\n NodeID: jincab3p1i1qia66yre5zwoxr\r\n Is Manager: false\r\n Node Address: 10.135.90.45\r\n Manager Addresses:\r\n  10.135.54.123:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c\r\nrunc version: 4fc53a81fb7c994640722ac585fa9ca548971871\r\ninit version: 949e6fa\r\nSecurity Options:\r\n apparmor\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.4.0-119-generic\r\nOperating System: Ubuntu 16.04.4 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 3.859GiB\r\nName: node10\r\nID: IZAT:E3YD:7HQP:M5UO:4LCO:NQX2:X3QB:KUNK:RCW5:LVFK:CNHT:4Y2Y\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\n provider=digitalocean\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: No swap limit support\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nHosted on Digital Ocean."},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/386282701","html_url":"https://github.com/moby/moby/issues/36797#issuecomment-386282701","issue_url":"https://api.github.com/repos/moby/moby/issues/36797","id":386282701,"user":{"login":"pstrzelczak","id":7737102,"avatar_url":"https://avatars1.githubusercontent.com/u/7737102?v=4","gravatar_id":"","url":"https://api.github.com/users/pstrzelczak","html_url":"https://github.com/pstrzelczak","followers_url":"https://api.github.com/users/pstrzelczak/followers","following_url":"https://api.github.com/users/pstrzelczak/following{/other_user}","gists_url":"https://api.github.com/users/pstrzelczak/gists{/gist_id}","starred_url":"https://api.github.com/users/pstrzelczak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pstrzelczak/subscriptions","organizations_url":"https://api.github.com/users/pstrzelczak/orgs","repos_url":"https://api.github.com/users/pstrzelczak/repos","events_url":"https://api.github.com/users/pstrzelczak/events{/privacy}","received_events_url":"https://api.github.com/users/pstrzelczak/received_events","type":"User","site_admin":false},"created_at":"2018-05-03T12:44:02Z","updated_at":"2018-05-03T12:44:02Z","author_association":"NONE","body":"It seems that I am hitting this problem in my work. I started to experience this after I tried to move to 18.03.0-ce from 17.06.2-ce. This blocks me to switch to newest \"stable\" version of Swarm. I will give 17.12.1-ce a try.\r\n\r\nSimilarly as reporter of this bug, I don't have an isolated test case which is able to reproduce this issue but in my use-case I have a haproxy Swarm service with an http port exposed and attempt to use frequently ends up with 'No route to host' error.\r\nI managed to catch the error with curl and record its strace but found nothing special there - it seems curl uses non-blocking sockets and fails to connect to a remote site:\r\n\r\n```\r\nsocket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4\r\nsetsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0\r\nsetsockopt(4, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0\r\nsetsockopt(4, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0\r\nfcntl(4, F_GETFL)                       = 0x2 (flags O_RDWR)\r\nfcntl(4, F_SETFL, O_RDWR|O_NONBLOCK)    = 0\r\nconnect(4, {sa_family=AF_INET, sin_port=htons(5601), sin_addr=inet_addr(\"10.9.4.120\")}, 16) = -1 EINPROGRESS (Operation now in progress)\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT}], 1, 1)    = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT}], 1, 198)  = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)\r\npoll([{fd=4, events=POLLOUT}], 1, 1000) = 1 ([{fd=4, revents=POLLOUT|POLLERR|POLLHUP}])\r\npoll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 1 ([{fd=4, revents=POLLOUT|POLLWRNORM|POLLERR|POLLHUP}])\r\ngetsockopt(4, SOL_SOCKET, SO_ERROR, [113], [4]) = 0\r\nclose(4)                                = 0\r\n```\r\n\r\n"}}
{"actor":{"display_login":"pstrzelczak"}}