title: Python app in Docker container doesn't stop/remove Docker container when app fails
body: <p>I've got a Python application that polls a queue for new data, and inserts it into a TimescaleDB database (TimescaleDB is an extension for PostgreSQL). <strong>This application <em>must</em> stay running at all times.</strong></p>
  <p>The problem is, the Python program may fail from time to time, and I expect Docker Swarm to restart the container. However, the containers keep running even after failure. <strong>Why aren't my containers failing and then being restarted by Docker Swarm?</strong></p>
  <p>The Python app looks something like this:</p>
  <pre class="lang-py prettyprint-override"><code>def main():
      try:
          conn = get_db_conn()
          insert_data(conn)
      except Exception:
          logger.exception(&quot;Error with main inserter.py function&quot;)
          send_email_if_error()
          raise
      finally:
          try:
              conn.close()
              del conn
          except Exception:
              pass
  
          return 0
  
  
  if __name__ == &quot;__main__&quot;:
      main()
  </code></pre>
  <p>The Dockerfile looks like this:</p>
  <pre><code>FROM python:3.8-slim-buster
  
  # Configure apt and install packages
  RUN apt-get update &amp;&amp; \
      apt-get -y --no-install-recommends install cron nano procps
  
  # Install Python requirements.
  RUN pip3 install --upgrade pip &amp;&amp; \
      pip3 install poetry==1.0.10
  
  COPY poetry.lock pyproject.toml /
  RUN poetry config virtualenvs.create false &amp;&amp; \
    poetry install --no-interaction --no-ansi
  
  # Copy everything to the / folder inside the container
  COPY . /
  
  # Make /var/log the default directory in the container
  WORKDIR /var/log
  
  # Start Python app on container startup
  CMD [&quot;python3&quot;, &quot;/inserter/inserter.py&quot;]
  </code></pre>
  <p>Docker-Compose file:</p>
  <pre><code>version: '3.7'
  services:
    inserter13:
      # Name and tag of image the Dockerfile creates
      image: mccarthysean/ijack:timescale
      depends_on: 
        - timescale13
      env_file: .env
      environment: 
        POSTGRES_HOST: timescale13
      networks:
        - traefik-public
      deploy:
        # Either global (exactly one container per physical node) or
        # replicated (a specified number of containers). The default is replicated
        mode: replicated
        # For stateless applications using &quot;replicated&quot; mode,
        # the total number of replicas to create
        replicas: 2
        restart_policy:
          on-failure # default is 'any'
  
    timescale13:
      image: timescale/timescaledb:2.3.0-pg13
      volumes: 
        - type: volume
          source: ijack-timescale-db-pg13
          target: /var/lib/postgresql/data # the location in the container where the data are stored
          read_only: false
        # Custom postgresql.conf file will be mounted (see command: as well)
        - type: bind
          source: ./postgresql_custom.conf
          target: /postgresql_custom.conf
          read_only: false
      env_file: .env
      command: [&quot;-c&quot;, &quot;config_file=/postgresql_custom.conf&quot;]
      ports:
        - 0.0.0.0:5432:5432
      networks:
        traefik-public:
      deploy:
        # Either global (exactly one container per physical node) or
        # replicated (a specified number of containers). The default is replicated
        mode: replicated
        # For stateless applications using &quot;replicated&quot; mode,
        # the total number of replicas to create
        replicas: 1
        placement:
          constraints:
            # Since this is for the stateful database,
            # only run it on the swarm manager, not on workers
            - &quot;node.role==manager&quot;
        restart_policy:
          condition: on-failure # default is 'any'
  
  
  # Use a named external volume to persist our data
  volumes:
    ijack-timescale-db-pg13:
      external: true
  
  networks:
    # Use the previously created public network &quot;traefik-public&quot;, shared with other
    # services that need to be publicly available via this Traefik
    traefik-public:
      external: true
  </code></pre>
  <p>The &quot;Docker-compose.build.yml&quot; file I use for building the &quot;inserter.py&quot; container's image:</p>
  <pre><code>version: '3.7'
  services:
    inserter:
      # Name and tag of image the Dockerfile creates
      image: mccarthysean/ijack:timescale
      build:
        # context: where should docker-compose look for the Dockerfile?
        # i.e. either a path to a directory containing a Dockerfile, or a url to a git repository
        context: .
        dockerfile: Dockerfile.inserter
      environment: 
        POSTGRES_HOST: timescale
  </code></pre>
  <p>Bash script I run, which builds, pushes, and deploys the database and the inserter containers with Docker Swarm:</p>
  <pre><code>#!/bin/bash
  
  # Build and tag image locally in one step. 
  # No need for docker tag &lt;image&gt; mccarthysean/ijack:&lt;tag&gt;
  echo &quot;&quot;
  echo &quot;Building the image locally...&quot;
  echo &quot;docker-compose -f docker-compose.build.yml build&quot;
  docker-compose -f docker-compose.build.yml build
  
  # Push to Docker Hub
  # docker login --username=mccarthysean
  echo &quot;&quot;
  echo &quot;Pushing the image to Docker Hub...&quot;
  echo &quot;docker push mccarthysean/ijack:timescale&quot;
  docker push mccarthysean/ijack:timescale
  
  # Deploy to the Docker swarm and send login credentials 
  # to other nodes in the swarm with &quot;--with-registry-auth&quot;
  echo &quot;&quot;
  echo &quot;Deploying to the Docker swarm...&quot;
  echo &quot;docker stack deploy --with-registry-auth -c docker-compose.prod13.yml timescale13&quot;
  docker stack deploy --with-registry-auth -c docker-compose.prod13.yml timescale13
  </code></pre>
  <p>When the Python inserter program fails (could be a database connection issue, or something else), it sends me an email alert and then raises the error and fails. At this point, I expect the Docker container to fail and be restarted with Docker Swarm's <code>restart_policy: on-failure</code>. However, after an error, when I type <code>docker service ls</code> I see the following <code>0/2 replicas</code>:</p>
  <pre><code>ID                  NAME                                        MODE                REPLICAS            IMAGE                                         PORTS
  u354h0uj4ug6        timescale13_inserter13                      replicated          0/2                 mccarthysean/ijack:timescale
  o0rbfx5n2z4h        timescale13_timescale13                     replicated          1/1                 timescale/timescaledb:2.3.0-pg13              *:5432-&gt;5432/tcp
  </code></pre>
  <p>When it's healthy (most of the time), it shows <code>2/2</code> replicas. Why aren't my containers failing and then being restarted by Docker Swarm?</p>