title: Shutting down Docker containers via supervisor
body: <p>I'm unable to shut down Docker containers that were launched by supervisor through <code>supervisorctl stop all</code>. Even through <code>supervisorctl status</code> shows the containers are down, <code>docker ps</code> and <code>ps</code> indicate that they are in fact still running.</p>
  
  <p>Consulting the supervisor documentation on the action for <code>supervisorctl stop &lt;name&gt;</code> reveals that <code>SIGTERM</code> is sent to processes followed by <code>SIGKILL</code> if still running after some grace period. I tried to do this manually and found that</p>
  
  <ul>
  <li><code>SIGTERM</code> sent to a <code>docker run</code> process doesn't do anything</li>
  <li><code>SIGKILL</code> does kill the process, but doesn't actually update docker. <code>docker ps</code> shows this container is still running</li>
  <li>Supervisor's <code>SIGKILL</code> doesn't shut down the container</li>
  </ul>
  
  <p>The questions is: How do I properly shut down a Docker container by supervisor?</p>
  
  <hr>
  
  <p>Here's the result of my experiments simulating supervisor:</p>
  
  <p>Starting position: <code>foo-1</code> and <code>bar-1</code> are running (I left the GCE containers in, in case they make a difference). <code>ps aux</code> and <code>docker ps</code> are in sync.</p>
  
  <pre><code>me@devenv:~$ sudo docker ps
  CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
  5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   5 minutes ago       Up 5 minutes                                 foo-1
  e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   5 minutes ago       Up 5 minutes                                 bar-1
  fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    35 minutes ago      Up 35 minutes                                bbbb 
  db677eed47ef        kubernetes/pause:go     "/pause"               35 minutes ago      Up 35 minutes       0.0.0.0:4194-&gt;8080/tcp   aaaa
  
  me@devenv:~$ ps aux | grep "docker run"
  root     23358  0.0  0.1 124092 11856 pts/0    Sl   02:05   0:00 docker run --rm --name foo-1 ... -i me/app:foo
  root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar
  </code></pre>
  
  <p>Simulate <code>supervisorctl stop foo-1</code> by sending <code>SIGTERM</code> to the process. Result: process still active.</p>
  
  <pre><code>me@devenv:~$ sudo kill -SIGTERM 23358
  
  ... &lt;waiting&gt; ...
  
  me@devenv:~$ ps aux | grep "docker run"
  root     23358  0.0  0.1 124092 11856 pts/0    Sl   02:05   0:00 docker run --rm --name foo-1 ... -i me/app:foo
  root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar
  
  me@devenv:~$ sudo docker ps
  CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
  5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   6 minutes ago       Up 6 minutes                                 foo-1
  e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   6 minutes ago       Up 6 minutes                                 bar-1
  fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    36 minutes ago      Up 36 minutes                                bbbb 
  db677eed47ef        kubernetes/pause:go     "/pause"               36 minutes ago      Up 36 minutes       0.0.0.0:4194-&gt;8080/tcp   aaaa
  </code></pre>
  
  <p>Next thing supervisor would do is issuing <code>SIGKILL</code>. Result: Process is killed (<code>ps aux</code>), but still showing as running docker process (<code>docker ps</code>).</p>
  
  <pre><code>me@devenv:~$ sudo kill -SIGKILL 23358
  me@devenv:~$ ps aux | grep "docker run"
  root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar
  
  me@devenv:~$ sudo docker ps
  CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
  5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   19 minutes ago      Up 19 minutes                                foo-1
  e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   19 minutes ago      Up 19 minutes                                bar-1
  fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    49 minutes ago      Up 49 minutes                                bbbb 
  db677eed47ef        kubernetes/pause:go     "/pause"               49 minutes ago      Up 49 minutes       0.0.0.0:4194-&gt;8080/tcp   aaaa
  </code></pre>
  
  <p>Supervisor was shut down during the above experiment (to avoid its autostart behaviour to interfere). The result for explicitly sending <code>SIGKILL</code> to the process could not be achieved by supervisor; the process was still alive (even though the supervisor logs state otherwise).
  <code>docker stop &lt;container_id&gt;</code> did however stop the container.</p>
  
  <p><strong>Update</strong></p>
  
  <p>Inside the Docker containers is also a <code>supervisord</code> process running which manages certain processes. Perhaps the problem is that signals aren't propagated and thus it won't shut down...</p>
  
  <p><strong>Update 2</strong></p>
  
  <p>I narrowed down the problem. I was able to directly start a containers process from the <code>Dockerfile</code> instead of going through starting <code>supervisord</code> and it makes a difference. I'm able to control this container through supervisor (the one outside of the docker containers which controls the containers).</p>
  
  <p><strong>Update 3</strong></p>
  
  <p>Setting <code>stopasgroup=true</code> as suggested <a href="https://serverfault.com/questions/459456/supervisord-leaking-child-processes-launched-through-shell-script">here</a> does not change anything for me.</p>
  
  <p><strong>Update 4</strong></p>
  
  <p>I was able to solve one of the problems: <code>supervisorctl</code> not being able to shut down the process. The problem was that I was launching the docker containers in the supervisor configuration file with <code>command=sudo docker run...</code> which created a <code>sudo docker run...</code> and a <code>docker run...</code> process. <code>supervisorctl stop...</code> just terminated the <code>sudo docker run...</code> process while the actual docker process was still running. When I omit the <code>sudo</code> command, only 1 process per supervisor program is started and <code>supervisorctl stop</code> terminates the process.</p>
  
  <p>One problem remains which is that <code>docker ps</code> shows the container is still running while <code>ps aux</code> doesn't. Oddly enough the containers still seem to be active as they respond to requests. A quick look at the process list confirms that all processes spawned by the docker container are still active, yet the <code>docker run...</code> process is missing from the process list.</p>
  
  <p><strong>Update 5</strong></p>
  
  <p>Sending <code>SIGTERM</code>, <code>SIGHUP</code> or <code>SIGQUIT</code> to the <code>docker run</code> process doesn't seem to do anything to the process. Only <code>SIGKILL</code> terminates the docker process properly. Supervisor gets updated properly, but <code>docker ps</code> still shows the docker process running.</p>