{"action":"closed","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/5796","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/5796/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/5796/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/5796/events","html_url":"https://github.com/rancher/rancher/issues/5796","id":173011263,"number":5796,"title":"Upgrades stuck on \"Starting\" while containers are actually running on hosts","user":{"login":"schuylr","id":12628715,"avatar_url":"https://avatars.githubusercontent.com/u/12628715?v=3","gravatar_id":"","url":"https://api.github.com/users/schuylr","html_url":"https://github.com/schuylr","followers_url":"https://api.github.com/users/schuylr/followers","following_url":"https://api.github.com/users/schuylr/following{/other_user}","gists_url":"https://api.github.com/users/schuylr/gists{/gist_id}","starred_url":"https://api.github.com/users/schuylr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schuylr/subscriptions","organizations_url":"https://api.github.com/users/schuylr/orgs","repos_url":"https://api.github.com/users/schuylr/repos","events_url":"https://api.github.com/users/schuylr/events{/privacy}","received_events_url":"https://api.github.com/users/schuylr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-08-24T17:24:44Z","updated_at":"2016-09-15T23:31:17Z","closed_at":"2016-09-15T23:31:16Z","body":"**Rancher Version:** v1.1.2\r\n\r\n**Docker Version:** 1.10.3\r\n\r\n**OS and where are the hosts located? (cloud, bare metal, etc):** RancherOS 0.4.5, AWS EC2\r\n\r\n**Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)** Single Node behind AWS ELB, external DB\r\n\r\n**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)** Cattle\r\n\r\n**Steps to Reproduce:**\r\n\r\nOn any of my services, run an upgrade with a batch interval of 120-600s (to account for initializing on my containers).\r\n\r\n**Results:** \r\n\r\nContainers are stuck on \"Starting (Starting)\" with no extra messages:\r\n\r\n![screen shot 2016-08-24 at 1 15 11 pm](https://cloud.githubusercontent.com/assets/12628715/17940394/dd3a783a-69fc-11e6-8d88-f0014afec83d.png)\r\n\r\n![screen shot 2016-08-24 at 1 15 49 pm](https://cloud.githubusercontent.com/assets/12628715/17940398/e4b56d2c-69fc-11e6-8b09-1157520897af.png)\r\n\r\nWhen I ssh into the host OS, I can see the running process but with a UUID name, whereas the container in the UI doesn't have a `io.rancher.container.uuid` label to identify the container in `docker ps`:\r\n\r\n```\r\n[rancher@ip-10-2-2-227 ~]$ docker ps\r\nCONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                        PORTS                                                              NAMES\r\ne2493bceb6d0        quay.io/jewlr/jewlr4:stable      \"/sbin/my_init\"          9 minutes ago       Up 9 minutes                  80-81/tcp, 443/tcp, 0.0.0.0:30001->30001/tcp                       a2f3b836-fa42-4b3a-b44c-f87e0d7d7655\r\nc55e58f44b12        rancher/agent:v1.0.2             \"/run.sh run\"            27 hours ago        Up 27 hours                                                                                      rancher-agent\r\nd6840b302dbe        rancher/etcd:v2.3.7-6            \"/run.sh node\"           5 days ago          Up 5 days                                                                                        r-etcd-ha_etcd_1\r\n6ec77e66c635        quay.io/jewlr/haproxy:latest     \"/bin/sh -c /sbin/my_\"   5 days ago          Up 5 days                     0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8080->8080/tcp   r-jewlr4-de_loadbalancer-de_1\r\nbd3428d771c6        meltwater/docker-cleanup:1.6.0   \"/run.sh\"                5 days ago          Up 5 days                                                                                        r-janitor_cleanup_17\r\ne36409417335        rancher/agent-instance:v0.8.3    \"/etc/init.d/agent-in\"   5 days ago          Up 5 days                     0.0.0.0:500->500/udp, 0.0.0.0:4500->4500/udp                       7a1a95d9-d134-42af-8f90-9235eaad1603\r\nca00074117e3        rancher/agent:v1.0.2             \"/run.sh https://ranc\"   5 days ago          Restarting (0) 27 hours ago                                                                      register\r\n```\r\n\r\nThe second host doesn't start the container because the first container is still not running, it seems. The rancher server logs don't seem to indicate any problems.\r\n\r\nIf I stop the UUID container, the scheduler will restart it and begin the second container:\r\n\r\n![screen shot 2016-08-24 at 1 19 51 pm](https://cloud.githubusercontent.com/assets/12628715/17940505/7b19e5c2-69fd-11e6-9ec3-de71a1c35fe5.png)\r\n\r\nExcept this time the first container says \"Starting (Networking)\" and there's no UUID or actual containers in my `docker ps`. The second container then says \"Starting (Starting)\" and has the same issue that the first host originally had.\r\n\r\nThen, if I stop the first container a second time, and the second container a first time, the first container finally starts and goes into its correct \"Initializing\" state. Once it's \"Running\", the second container starts again and goes into its \"Initializing\" state, and then \"Running\". Then I can finally complete my upgrade.\r\n\r\n**Expected:**\r\n\r\nZero-downtime jank-free upgrades :("}}
{"actor":{"display_login":"schuylr"}}