{"action":"closed","issue":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/335","repository_url":"https://api.github.com/repos/aws/amazon-ecs-agent","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/335/labels{/name}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/335/comments","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/335/events","html_url":"https://github.com/aws/amazon-ecs-agent/issues/335","id":140048616,"number":335,"title":"docker kill -s HUP <container> blows up a task","user":{"login":"natefox","id":513476,"avatar_url":"https://avatars.githubusercontent.com/u/513476?v=3","gravatar_id":"","url":"https://api.github.com/users/natefox","html_url":"https://github.com/natefox","followers_url":"https://api.github.com/users/natefox/followers","following_url":"https://api.github.com/users/natefox/following{/other_user}","gists_url":"https://api.github.com/users/natefox/gists{/gist_id}","starred_url":"https://api.github.com/users/natefox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/natefox/subscriptions","organizations_url":"https://api.github.com/users/natefox/orgs","repos_url":"https://api.github.com/users/natefox/repos","events_url":"https://api.github.com/users/natefox/events{/privacy}","received_events_url":"https://api.github.com/users/natefox/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels/pending%20release","name":"pending release","color":"0052cc"}],"state":"closed","locked":false,"assignee":{"login":"samuelkarp","id":737750,"avatar_url":"https://avatars.githubusercontent.com/u/737750?v=3","gravatar_id":"","url":"https://api.github.com/users/samuelkarp","html_url":"https://github.com/samuelkarp","followers_url":"https://api.github.com/users/samuelkarp/followers","following_url":"https://api.github.com/users/samuelkarp/following{/other_user}","gists_url":"https://api.github.com/users/samuelkarp/gists{/gist_id}","starred_url":"https://api.github.com/users/samuelkarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samuelkarp/subscriptions","organizations_url":"https://api.github.com/users/samuelkarp/orgs","repos_url":"https://api.github.com/users/samuelkarp/repos","events_url":"https://api.github.com/users/samuelkarp/events{/privacy}","received_events_url":"https://api.github.com/users/samuelkarp/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-03-11T00:25:29Z","updated_at":"2016-04-05T21:56:39Z","closed_at":"2016-04-05T21:56:39Z","body":"I'm running nginx:latest with consul-template. Each time consul-template updates the nginx config I need to HUP the container/nginx process so nginx will run the latest config (per the official docs under [Controlling NGINX](https://blog.docker.com/2015/04/tips-for-deploying-nginx-official-image-with-docker/)).\r\n\r\nWhen I run `docker kill -s HUP <container id>` nginx reloads the config and all is well. However, ECS hates this. ECS-agent thinks the container has stopped - but it actually keeps running just fine. Then the task/container is reported as stopped and nginx is attempted to be relaunched (which it cant cause the existing container is holding the host port open).\r\n\r\nHere's the ecs-agent log from when I run the `kill -s HUP` (with personal info removed):\r\n```\r\n2016-03-10T23:55:55Z [INFO] Adding event module=\"eventhandler\" change=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 nginx -> STOPPED, Ports [{80 80 0.0.0.0 0}], Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Stopping container module=\"TaskEngine\" task=\"mytask:5 arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22, Status: (RUNNING->STOPPED) Containers: [nginx (STOPPED->STOPPED),consul-template (RUNNING->STOPPED),]\" container=\"consul-template(myregistry/consul-template) (RUNNING->STOPPED)\"\r\n2016-03-10T23:55:55Z [INFO] Sending container change module=\"eventhandler\" event=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 nginx -> STOPPED, Ports [{80 80 0.0.0.0 0}], Known Sent: RUNNING\" change=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 nginx -> STOPPED, Ports [{80 80 0.0.0.0 0}], Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Task change event module=\"TaskEngine\" event=\"{TaskArn:arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 Status:STOPPED Reason: SentStatus:RUNNING}\"\r\n2016-03-10T23:55:55Z [INFO] Adding event module=\"eventhandler\" change=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 consul-template -> STOPPED, Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Saving state! module=\"statemanager\"\r\n2016-03-10T23:55:55Z [INFO] Adding event module=\"eventhandler\" change=\"TaskChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 -> STOPPED, Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Sending container change module=\"eventhandler\" event=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 consul-template -> STOPPED, Known Sent: RUNNING\" change=\"ContainerChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 consul-template -> STOPPED, Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Sending task change module=\"eventhandler\" event=\"TaskChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 -> STOPPED, Known Sent: RUNNING\" change=\"TaskChange: arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22 -> STOPPED, Known Sent: RUNNING\"\r\n2016-03-10T23:55:55Z [INFO] Redundant container state change for task mytask:5 arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22, Status: (STOPPED->STOPPED) Containers: [nginx (STOPPED->STOPPED),consul-template (STOPPED->STOPPED),]: consul-template(myregistry/consul-template) (STOPPED->STOPPED) to STOPPED, but already STOPPED\r\n2016-03-10T23:55:55Z [INFO] Redundant container state change for task mytask:5 arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22, Status: (STOPPED->STOPPED) Containers: [nginx (STOPPED->STOPPED),consul-template (STOPPED->STOPPED),]: consul-template(myregistry/consul-template) (STOPPED->STOPPED) to STOPPED, but already STOPPED\r\n2016-03-10T23:55:55Z [INFO] Redundant container state change for task mytask:5 arn:aws:ecs:us-west-2:xxxxxxxxxxxx:task/8c119869-9ee3-467d-a467-c9a3c8193a22, Status: (STOPPED->STOPPED) Containers: [nginx (STOPPED->STOPPED),consul-template (STOPPED->STOPPED),]: consul-template(myregistry/consul-template) (STOPPED->STOPPED) to STOPPED, but already STOPPED\r\n2016-03-10T23:56:05Z [INFO] Saving state! module=\"statemanager\"\r\n```\r\n\r\nThe biggest problem I have with this is if you have 2 containers in a task definition (nginx being one of them and marked essential), the second container is killed because it thinks nginx was killed. Due to this we're unable to run consul-template as a container along with nginx. We have to run 2 processes in one container and background nginx so it can be HUP'd properly.\r\n\r\n"}}