{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/28493","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/28493/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/28493/comments","events_url":"https://api.github.com/repos/docker/docker/issues/28493/events","html_url":"https://github.com/docker/docker/issues/28493","id":189825294,"number":28493,"title":"Routing mesh problems in mixed version swarm","user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"labels":[{"id":237472034,"url":"https://api.github.com/repos/docker/docker/labels/area/networking","name":"area/networking","color":"fbca04","default":false},{"id":392678049,"url":"https://api.github.com/repos/docker/docker/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false},{"id":190258329,"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false},{"id":283253884,"url":"https://api.github.com/repos/docker/docker/labels/priority/P0","name":"priority/P0","color":"e11d21","default":false}],"state":"closed","locked":false,"assignee":{"login":"sanimej","id":10071849,"avatar_url":"https://avatars.githubusercontent.com/u/10071849?v=3","gravatar_id":"","url":"https://api.github.com/users/sanimej","html_url":"https://github.com/sanimej","followers_url":"https://api.github.com/users/sanimej/followers","following_url":"https://api.github.com/users/sanimej/following{/other_user}","gists_url":"https://api.github.com/users/sanimej/gists{/gist_id}","starred_url":"https://api.github.com/users/sanimej/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanimej/subscriptions","organizations_url":"https://api.github.com/users/sanimej/orgs","repos_url":"https://api.github.com/users/sanimej/repos","events_url":"https://api.github.com/users/sanimej/events{/privacy}","received_events_url":"https://api.github.com/users/sanimej/received_events","type":"User","site_admin":false},"assignees":[{"login":"sanimej","id":10071849,"avatar_url":"https://avatars.githubusercontent.com/u/10071849?v=3","gravatar_id":"","url":"https://api.github.com/users/sanimej","html_url":"https://github.com/sanimej","followers_url":"https://api.github.com/users/sanimej/followers","following_url":"https://api.github.com/users/sanimej/following{/other_user}","gists_url":"https://api.github.com/users/sanimej/gists{/gist_id}","starred_url":"https://api.github.com/users/sanimej/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanimej/subscriptions","organizations_url":"https://api.github.com/users/sanimej/orgs","repos_url":"https://api.github.com/users/sanimej/repos","events_url":"https://api.github.com/users/sanimej/events{/privacy}","received_events_url":"https://api.github.com/users/sanimej/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/56","html_url":"https://github.com/docker/docker/milestones/1.13.0","labels_url":"https://api.github.com/repos/docker/docker/milestones/56/labels","id":1839404,"number":56,"title":"1.13.0","description":"","creator":{"login":"thaJeztah","id":1804568,"avatar_url":"https://avatars.githubusercontent.com/u/1804568?v=3","gravatar_id":"","url":"https://api.github.com/users/thaJeztah","html_url":"https://github.com/thaJeztah","followers_url":"https://api.github.com/users/thaJeztah/followers","following_url":"https://api.github.com/users/thaJeztah/following{/other_user}","gists_url":"https://api.github.com/users/thaJeztah/gists{/gist_id}","starred_url":"https://api.github.com/users/thaJeztah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thaJeztah/subscriptions","organizations_url":"https://api.github.com/users/thaJeztah/orgs","repos_url":"https://api.github.com/users/thaJeztah/repos","events_url":"https://api.github.com/users/thaJeztah/events{/privacy}","received_events_url":"https://api.github.com/users/thaJeztah/received_events","type":"User","site_admin":false},"open_issues":111,"closed_issues":1134,"state":"open","created_at":"2016-06-20T22:51:25Z","updated_at":"2016-12-05T19:14:45Z","due_on":null,"closed_at":null},"comments":9,"created_at":"2016-11-16T19:57:28Z","updated_at":"2016-12-05T19:15:15Z","closed_at":"2016-12-05T19:14:45Z","body":"I created a swarm running 1.12.3 on two manager hosts. I created a redis service with two instances as follows:\r\n\r\n```\r\ndocker service create --replicas 2 --name redis -p 6379:6379 redis\r\n```\r\n\r\nOne replica ran on each host. I could connect through the routing mesh and confirm that the round robin was working:\r\n\r\n```\r\n$ telnet 52.23.174.225 6379\r\nTrying 52.23.174.225...\r\nConnected to ec2-52-23-174-225.compute-1.amazonaws.com.\r\nEscape character is '^]'.\r\nPING\r\n+PONG\r\nSET key value\r\n+OK\r\nEXISTS key\r\n:1\r\n^]\r\ntelnet> Connection closed.\r\n$ telnet 52.23.174.225 6379\r\nTrying 52.23.174.225...\r\nConnected to ec2-52-23-174-225.compute-1.amazonaws.com.\r\nEscape character is '^]'.\r\nEXISTS key\r\n:0\r\n^]\r\ntelnet> Connection closed.\r\n$ telnet 52.23.174.225 6379\r\nTrying 52.23.174.225...\r\nConnected to ec2-52-23-174-225.compute-1.amazonaws.com.\r\nEscape character is '^]'.\r\nEXISTS key\r\n:1\r\n^]\r\ntelnet> Connection closed.\r\n$ telnet 52.23.174.225 6379\r\nTrying 52.23.174.225...\r\nConnected to ec2-52-23-174-225.compute-1.amazonaws.com.\r\nEscape character is '^]'.\r\nEXISTS key\r\n:0\r\n^]\r\ntelnet> Connection closed.\r\n```\r\n\r\nThen I upgraded one of the nodes to 1.13.1-rc1. After upgrading, I continued to see the routing mesh work correctly when connecting to `.225` (which was still running 1.12.3, and was the Raft leader at the time). I did not try connecting to the other node at this point.\r\n\r\nI restarted Docker on `.225` to trigger a leader failover. The other node (`.184`, running 1.13.1-rc1) became the leader.\r\n\r\nNow the situation is:\r\n\r\n- Both redis containers are running on `.184`.\r\n- Connections to `.225` never succeed. They always get \"Connection refused\"\r\n- Every other connection to `.184` succeeds. The others fail with \"Connection refused\"\r\n\r\n`.225` has this in its log (garbling is real and not a mispaste):\r\n\r\n```\r\ntime=\"2016-11-16T19:41:29.285637014Z\" level=info msg=\"API listen on /var/run/docker.sock\"\r\npeerdbupdate in sandbox failed for ip 10.255.0.4 and mac 02:42:0a:ff:00:04: subnet sandbox join failed for \"10.255.0.0/16\": overlay subnet 10.255.0.0/16 has conflicts in the host while running in host modetime=\"2016-11-16T19:41:29.965589879Z\" level=error msg=\"Failed joining ingress sandbox to ingress endpoint: subnet sandbox join failed for \\\"10.255.0.0/16\\\": overlay subnet 10.255.0.0/16 has conflicts in the host while running in host mode\"\r\n```\r\n\r\n`.184` does not have anything out of the ordinary in its log except this repeated every 20 seconds:\r\n\r\n```\r\ntime=\"2016-11-16T19:50:11.192677365Z\" level=warning msg=\"failed to retrieve docker-init version: unknown output format\"\r\n```\r\n\r\nI'll file a separate issue about that log message."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/264947647","html_url":"https://github.com/docker/docker/issues/28493#issuecomment-264947647","issue_url":"https://api.github.com/repos/docker/docker/issues/28493","id":264947647,"user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T19:15:14Z","updated_at":"2016-12-05T19:15:14Z","body":"(and https://github.com/docker/docker/pull/29049 for 1.13)"}}
{"actor":{"display_login":"aaronlehmann"}}