{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/4089","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/4089/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/4089/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/4089/events","html_url":"https://github.com/rancher/rancher/issues/4089","id":142823713,"number":4089,"title":"Instance stuck in \"networking\" state forever. ","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"},{"url":"https://api.github.com/repos/rancher/rancher/labels/status/resolved","name":"status/resolved","color":"0052cc"},{"url":"https://api.github.com/repos/rancher/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66"}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/4","html_url":"https://github.com/rancher/rancher/milestones/Release%201.0","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/4/labels","id":947265,"number":4,"title":"Release 1.0","description":"1.0 GA","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":39,"closed_issues":620,"state":"open","created_at":"2015-01-22T18:32:24Z","updated_at":"2016-03-28T18:55:03Z","due_on":"2016-03-31T07:00:00Z","closed_at":null},"comments":2,"created_at":"2016-03-23T01:02:49Z","updated_at":"2016-03-28T18:55:03Z","closed_at":"2016-03-28T18:55:02Z","body":"Fresh install of v0.69.1\r\n\r\nCreated and activated few services with health check enabled and global flag on.\r\nThere were also 2 Lb services , 1 global and another with scale 1.\r\n\r\nUpgrade to v1.0.0-rc1\r\n\r\nAfter upgrade , 1 of the instance of LB from both the Lb services got removed and recreated.\r\n\r\nStopped the global lb service and started it again.\r\n\r\nWhen LB service was started , 1 out of the 3 instances got stuck in \"starting\" state ( stuck in \"networking\" state). \r\nTrying to start any other instance in this host also results in being stuck in \"starting\" state.\r\n\r\nFollowing exception is seen repeatedly in logs:\r\n\r\n\r\n```\r\n2016-03-23 00:57:46,094 INFO  [:] [] [] [] [tp639313753-230] [i.c.p.c.s.impl.ConfigItemServerImpl ] Processing [hosts] for client [agent:6] \r\n2016-03-23 00:57:46,502 ERROR [:] [] [] [] [ecutorService-3] [.p.c.v.i.ConfigItemStatusManagerImpl] Failed null, exit code [7] output [INFO: Updating hosts\r\nINFO: Downloading http://146.148.109.137:8080/v1//configcontent//hosts current=hosts-337-46bf3f2cd5e10e348cea0a9b2c2f20e1ff10eda99e38ca8db85820b45973a1d8\r\nINFO: Running /var/lib/cattle/download/hosts/hosts-370-df032449a7f08eeb26687a2e6ec2649d7fa7da0a45f897a92f55dc1b1d2451c1/apply.sh\r\nINFO: HOME -> ./\r\nINFO: HOME -> ./etc/\r\nINFO: HOME -> ./etc/cattle/\r\nINFO: HOME -> ./etc/cattle/metadata/\r\nINFO: HOME -> ./etc/cattle/metadata/answers.yml\r\nINFO: HOME -> ./etc/cattle/dns/\r\nINFO: HOME -> ./etc/cattle/dns/answers.json\r\n[[scripts.sh:184] script_env\r\n[[scripts.sh:109] '[' -n http://146.148.109.137:8080/v1 ']'\r\n[[scripts.sh:110] return\r\n[[scripts.sh:186] export CATTLE_HOME=/var/lib/cattle\r\n[[scripts.sh:186] CATTLE_HOME=/var/lib/cattle\r\n[[scripts.sh:187] export CATTLE_CONFIG_URL=http://146.148.109.137:8080/v1\r\n[[scripts.sh:187] CATTLE_CONFIG_URL=http://146.148.109.137:8080/v1\r\n[[scripts.sh:188] export CATTLE_STORAGE_URL=http://146.148.109.137:8080/v1\r\n[[scripts.sh:188] CATTLE_STORAGE_URL=http://146.148.109.137:8080/v1\r\n[apply.sh:23] DNS=\r\n[[apply.sh:25] awk '{print $2}'\r\n[[apply.sh:25] grep '^nameserver'\r\n[[apply.sh:25] sed 's/^# nameserver/nameserver/'\r\n[[apply.sh:25] cat /etc/resolv.conf\r\n[apply.sh:25] for address in '$(cat /etc/resolv.conf  | sed '\\''s/^# nameserver/nameserver/'\\'' | grep ^nameserver | awk '\\''{print $2}'\\'')'\r\n[apply.sh:26] '[' 169.254.169.254 = 169.254.169.250 ']'\r\n[apply.sh:30] '[' -z '' ']'\r\n[apply.sh:31] DNS='\"169.254.169.254\"'\r\n[apply.sh:37] sed -i 's/\"PARENT_DNS\"/\"169.254.169.254\"/g' content-home/etc/cattle/dns/answers.json\r\n[apply.sh:39] stage_files\r\n[scripts.sh:77] local monit=false\r\n[scripts.sh:79] '[' -d content-home ']'\r\n[scripts.sh:80] find content-home -name '*.sh' -exec chmod +x '{}' ';'\r\n[scripts.sh:81] sed 's/^/INFO: HOME -> /'\r\n[scripts.sh:81] tar xvf - --no-overwrite-dir -C /var/lib/cattle\r\n[scripts.sh:81] tar cf - -C content-home .\r\nINFO: HOME -> ./\r\nINFO: HOME -> ./etc/\r\nINFO: HOME -> ./etc/cattle/\r\nINFO: HOME -> ./etc/cattle/metadata/\r\nINFO: HOME -> ./etc/cattle/metadata/answers.yml\r\nINFO: HOME -> ./etc/cattle/dns/\r\nINFO: HOME -> ./etc/cattle/dns/answers.json\r\n[scripts.sh:84] '[' -d content ']'\r\n[scripts.sh:102] '[' false = true ']'\r\n[apply.sh:41] ip addr add 169.254.169.250/32 dev eth0\r\n[apply.sh:41] true\r\n[apply.sh:42] grep -q 169.254.169.254\r\n[apply.sh:42] ip route show\r\n[apply.sh:46] reload_service dns 8113\r\n[apply.sh:7] local service=dns\r\n[apply.sh:8] local port=8113\r\n[apply.sh:9] '[' '!' -e /etc/init.d/rancher-dns ']'\r\n[[apply.sh:14] pidof rancher-dns\r\n[apply.sh:14] PID=594\r\n[apply.sh:16] '[' -z 594 ']'\r\n[apply.sh:19] curl -sf -X POST http://localhost:8113/v1/reload\r\n```"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/202528214","html_url":"https://github.com/rancher/rancher/issues/4089#issuecomment-202528214","issue_url":"https://api.github.com/repos/rancher/rancher/issues/4089","id":202528214,"user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"created_at":"2016-03-28T18:55:02Z","updated_at":"2016-03-28T18:55:02Z","body":"This issue was not seen when testing upgrades from v0.69.1 to v1.0.0-rc2/v1.0.0-rc4."}}