title: Running "docker-compose run" hangs when it's run in cron
body: <h1>Environments</h1>
  <p>▶ uname -a<br />
  Linux ubuntu 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux</p>
  <p>▶ docker -v<br />
  Docker version 20.10.4, build d3cb89e</p>
  <p>▶ docker-compose -v<br />
  docker-compose version 1.28.4, build cabd5cfb</p>
  <p>▶ lsb_release -a<br />
  No LSB modules are available.<br />
  Distributor ID: Ubuntu<br />
  Description:    Ubuntu 20.04.2 LTS<br />
  Release:        20.04<br />
  Codename:       focal</p>
  <h1>Problem</h1>
  <p>I had been using ubuntu version 18.04 and my docker and docker-compose were running just fine.
  I didn't periodically update the server, but just 2 days ago, I did <code>sudo apt-get update &amp;&amp; apt-get upgrade</code>.  Then it all started acting wierd.</p>
  <p>I used to call docker-compose services from the host's cron like this.</p>
  <pre><code>0 * * * * cd /project/folder &amp;&amp; docker-compose run --rm service_name
  </code></pre>
  <p>However, after the update and upgrade, these cron jobs started to hanging.</p>
  <ul>
  <li>I could see processes dangling when I checked <code>top</code> command</li>
  <li>I could see the container program (command) was not even starting</li>
  </ul>
  <p>So every time the cron jobs tried to call those services, processes were stacking up.</p>
  <h1>Things I did</h1>
  <p>I uninstalled docker, docker-compose, reboot and reinstall them again, but it didn't help.  I even upgraded the OS from 18.04 to 20.04, and it didn't help either.</p>
  <p><a href="https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository" rel="noreferrer">https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository</a><br />
  <a href="https://docs.docker.com/compose/install/" rel="noreferrer">https://docs.docker.com/compose/install/</a></p>
  <pre><code>do-release-upgrade # to upgrade from v18 to v20
  </code></pre>
  <p>To investigate even further, I added <code>--verbose</code> like this and logged it.</p>
  <pre><code>0 * * * * cd /project/folder &amp;&amp; docker-compose run --rm --verbose service_name &gt;&gt; /project/folder/cron.log 2&gt;&amp;1
  </code></pre>
  <p>It seems it's hanging at <code>/usr/bin/docker start --attach --interactive</code> and nothing really happens.</p>
  <p>When I run the same command <code>cd /project/folder &amp;&amp; docker-compose run --rm --verbose service_name &gt;&gt; /project/folder/cron.log 2&gt;&amp;1</code> just from the terminal, it runs as expected.</p>
  <p>I made everything simple like below, but running it by cron still hangs in the middle.</p>
  <p>Anybody has an idea what is causing this?</p>
  <h1>docker-compose.yml</h1>
  <pre class="lang-yaml prettyprint-override"><code>services:
  
    test:
      build:
        context: ./test
      command: ash -c &quot;date&quot;
  </code></pre>
  <h1>Dockerfile</h1>
  <pre><code>FROM alpine:latest
  </code></pre>
  <h1>Logs</h1>
  <pre><code>compose.config.config.find: Using configuration files: ./docker-compose.yml
  compose.cli.docker_client.get_client: docker-compose version 1.28.4, build cabd5cfb
  docker-py version: 4.4.3
  CPython version: 3.7.10
  OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
  compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost
  compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '20.10.4', 'Details': {'ApiVersion': '1.41', 'Arch': 'amd64', 'BuildTime': '2021-02-25T07:03:23.000000000+00:00', 'Experimental': 'false', 'GitCommit': '363e9a8', 'GoVersion': 'go1.13.15', 'KernelVersion': '5.4.0-66-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.4.3', 'Details': {'GitCommit': '269548fa27e0089a8b8278fc4fc781d7f65a939b'}}, {'Name': 'runc', 'Version': '1.0.0-rc92', 'Details': {'GitCommit': 'ff819c7e9184c13b7c2607fe6c30ae19403a7aff'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=20.10.4, ApiVersion=1.41, MinAPIVersion=1.12, GitCommit=363e9a8, GoVersion=go1.13.15, Os=linux, Arch=amd64, KernelVersion=5.4.0-66-generic, BuildTime=2021-02-25T07:03:23.000000000+00:00
  compose.cli.verbose_proxy.proxy_callable: docker inspect_network &lt;- ('aaa_default')
  compose.cli.verbose_proxy.proxy_callable: docker inspect_network -&gt; {'Attachable': True,
   'ConfigFrom': {'Network': ''},
   'ConfigOnly': False,
   'Containers': {},
   'Created': '2020-07-19T12:11:28.401034027+09:00',
   'Driver': 'bridge',
   'EnableIPv6': False,
   'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
            'Driver': 'default',
            'Options': None},
  ...
  compose.cli.verbose_proxy.proxy_callable: docker inspect_network &lt;- ('aaa_default')
  compose.cli.verbose_proxy.proxy_callable: docker inspect_network -&gt; {'Attachable': True,
   'ConfigFrom': {'Network': ''},
   'ConfigOnly': False,
   'Containers': {},
   'Created': '2020-07-19T12:11:28.401034027+09:00',
   'Driver': 'bridge',
   'EnableIPv6': False,
   'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
            'Driver': 'default',
            'Options': None},
  ...
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker inspect_image &lt;- ('aaa_test')
  compose.cli.verbose_proxy.proxy_callable: docker inspect_image -&gt; {'Architecture': 'amd64',
   'Author': '',
   'Comment': '',
   'Config': {'AttachStderr': False,
              'AttachStdin': False,
              'AttachStdout': False,
              'Cmd': ['/bin/sh'],
              'Domainname': '',
              'Entrypoint': None,
              'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
  ...
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.parallel.feed_queue: Pending: {&lt;Service: test&gt;}
  compose.parallel.feed_queue: Starting producer thread for &lt;Service: test&gt;
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  Creating aaa_test_run ... 
  compose.parallel.feed_queue: Pending: {ServiceName(project='aaa', service='test', number=1)}
  compose.parallel.feed_queue: Starting producer thread for ServiceName(project='aaa', service='test', number=1)
  compose.cli.verbose_proxy.proxy_callable: docker inspect_image &lt;- ('aaa_test')
  compose.cli.verbose_proxy.proxy_callable: docker inspect_image -&gt; {'Architecture': 'amd64',
   'Author': '',
   'Comment': '',
   'Config': {'AttachStderr': False,
              'AttachStdin': False,
              'AttachStdout': False,
              'Cmd': ['/bin/sh'],
              'Domainname': '',
              'Entrypoint': None,
              'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
  ...
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker containers &lt;- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
  compose.cli.verbose_proxy.proxy_callable: docker containers -&gt; (list with 0 items)
  compose.cli.verbose_proxy.proxy_callable: docker create_host_config &lt;- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='aaa_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None)
  compose.cli.verbose_proxy.proxy_callable: docker create_host_config -&gt; {'Binds': [],
   'Links': [],
   'LogConfig': {'Config': {}, 'Type': ''},
   'NetworkMode': 'aaa_default',
   'PortBindings': {},
   'VolumesFrom': []}
  compose.cli.verbose_proxy.proxy_callable: docker create_container &lt;- (command='ash -c &quot;date&quot;', volumes={}, tty=False, stdin_open=True, detach=False, ports=[], name='aaa_test_run_b01e0a8c11e3', environment=[], labels={'com.docker.compose.project': 'aaa', 'com.docker.compose.service': 'test', 'com.docker.compose.oneoff': 'True', 'com.docker.compose.project.working_dir': '/home/ac/projects/playground/aaa', 'com.docker.compose.project.config_files': 'docker-compose.yml', 'com.docker.compose.slug': 'b01e0a8c11e319685038ac27761c14b08d5fa025dee67abab9da546afadc28f', 'com.docker.compose.version': '1.28.4'}, image='aaa_test', host_config={'NetworkMode': 'aaa_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'aaa_default': {'Aliases': ['test'], 'IPAMConfig': {}}}})
  compose.parallel.feed_queue: Pending: set()
  compose.parallel.feed_queue: Pending: set()
  compose.cli.verbose_proxy.proxy_callable: docker create_container -&gt; {'Id': 'c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b',
   'Warnings': []}
  compose.cli.verbose_proxy.proxy_callable: docker inspect_container &lt;- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b')
  compose.cli.verbose_proxy.proxy_callable: docker inspect_container -&gt; {'AppArmorProfile': '',
   'Args': ['-c', 'date'],
   'Config': {'AttachStderr': True,
              'AttachStdin': True,
              'AttachStdout': True,
              'Cmd': ['ash', '-c', 'date'],
              'Domainname': '',
              'Entrypoint': None,
              'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
              'Hostname': 'c743a18406c5',
  ...
  compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='aaa', service='test', number=1)
  Creating aaa_test_run ... done
  compose.parallel.feed_queue: Pending: set()
  compose.parallel.parallel_execute_iter: Finished processing: &lt;Service: test&gt;
  compose.parallel.feed_queue: Pending: set()
  compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network &lt;- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default')
  compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -&gt; None
  compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network &lt;- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default', aliases=[], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None)
  compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -&gt; None
  compose.cli.main.call_docker: /usr/bin/docker start --attach --interactive c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b
  </code></pre>