{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/3430","labels_url":"https://api.github.com/repos/rancher/rancher/issues/3430/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/3430/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/3430/events","html_url":"https://github.com/rancher/rancher/issues/3430","id":129225758,"number":3430,"title":"Upgrade to 0.56.0 caused several network errors","user":{"login":"jraede","id":4572130,"avatar_url":"https://avatars.githubusercontent.com/u/4572130?v=3","gravatar_id":"","url":"https://api.github.com/users/jraede","html_url":"https://github.com/jraede","followers_url":"https://api.github.com/users/jraede/followers","following_url":"https://api.github.com/users/jraede/following{/other_user}","gists_url":"https://api.github.com/users/jraede/gists{/gist_id}","starred_url":"https://api.github.com/users/jraede/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jraede/subscriptions","organizations_url":"https://api.github.com/users/jraede/orgs","repos_url":"https://api.github.com/users/jraede/repos","events_url":"https://api.github.com/users/jraede/events{/privacy}","received_events_url":"https://api.github.com/users/jraede/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-01-27T18:15:26Z","updated_at":"2016-01-27T19:30:36Z","closed_at":null,"body":"We're currently running:\r\n```\r\nRancher\tv0.51.0\r\nCattle\tv0.130.0\r\nUser Interface\tv0.78.0\r\nRancher Compose\tv0.7.0\r\n```\r\n\r\nI upgraded to 0.56.0 earlier, and it caused the following problems:\r\n\r\n* service links across physical servers stopped working (ending up with request timeouts). Restarting the network agent, restarting the services, and restarting the physical server did not fix the issue. Services linked to services on the *same* physical server, however, did not have problems\r\n* Probably related - Load Balancer services went from \"active\" (green) to \"initializing\" (yellow) and remained there through restarts, etc.\r\n* After trying several different things, we ended up fixing the problem by reverting to rancher-server 0.51.0 and that immediately fixed all networking problems.\r\n\r\nWere there significant changes to the service discovery/networking that may have caused this in 0.56? Any other ideas for how this could have been fixed?\r\n\r\nThanks\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/175812103","html_url":"https://github.com/rancher/rancher/issues/3430#issuecomment-175812103","issue_url":"https://api.github.com/repos/rancher/rancher/issues/3430","id":175812103,"user":{"login":"jraede","id":4572130,"avatar_url":"https://avatars.githubusercontent.com/u/4572130?v=3","gravatar_id":"","url":"https://api.github.com/users/jraede","html_url":"https://github.com/jraede","followers_url":"https://api.github.com/users/jraede/followers","following_url":"https://api.github.com/users/jraede/following{/other_user}","gists_url":"https://api.github.com/users/jraede/gists{/gist_id}","starred_url":"https://api.github.com/users/jraede/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jraede/subscriptions","organizations_url":"https://api.github.com/users/jraede/orgs","repos_url":"https://api.github.com/users/jraede/repos","events_url":"https://api.github.com/users/jraede/events{/privacy}","received_events_url":"https://api.github.com/users/jraede/received_events","type":"User","site_admin":false},"created_at":"2016-01-27T19:30:36Z","updated_at":"2016-01-27T19:30:36Z","body":"Correct. We did the following in order:\r\n\r\n* Restart the services (both sides of the link)\r\n* Restart the network agent on all servers\r\n* Restarting the physical ubuntu server via the EC2 dashboard (although not *all* servers, since we had some services that were unaffected, so we shouldn't necessarily rule this out as a solution)"}}