title: Certain Celery Tasks starts but hangs and never executes
body: <p>I have an issue with Django and Celery where some registered tasks never get executed.</p>
  <p>I have three tasks in my tasks.py file, two of them; <code>schedule_notification()</code> and <code>schedule_archive()</code> work without issue. They are executed without issue at the predefined ETA.</p>
  <p>With the <code>schedule_monitoring()</code> function, I can see the job is started in Celery Flower but it never actually executes. It just sits there.</p>
  <p>I have confirmed I can run the command locally from the worker so I'm not sure where the issue could be.</p>
  <p><strong>tasks.py (failing function)</strong></p>
  <pre><code>@task
  def schedule_monitoring(job_id: str, action: str) -&gt; str:
      salt = OSApi() # This is a wrapper around a REST API.
      job = Job.objects.get(pk=job_id)
      target = ('compound', f&quot;G@hostname:{ job.network.gateway.host_name } and G@serial:{ job.network.gateway.serial_number }&quot;)
  
      policies = [
          'foo',
          'bar',
          'foobar',
          'barfoo'
      ]
  
      if action == 'start':
          salt.run(target, 'spectrum.add_to_collection', fun_args=['foo'])  
          for policy in policies:
              salt.run(target, 'spectrum.refresh_policy', fun_args=[policy])
  
          create_activity(&quot;Informational&quot;, &quot;MONITORING&quot;, &quot;Started proactive monitoring for job.&quot;, job)
      elif action == 'stop':
          salt.run(target, 'spectrum.remove_from_collection', fun_args=['bar'])
          for policy in policies:
              salt.run(target, 'spectrum.refresh_policy', fun_args=[policy])
  
          create_activity(&quot;Informational&quot;, &quot;MONITORING&quot;, &quot;Stopped proactive monitoring for job.&quot;, job)
      else:
          raise NotImplementedError
  
      return f&quot;Applying monitoring action: {action.upper()} to Job: {job.job_code}&quot;
  </code></pre>
  <p><a href="https://i.stack.imgur.com/E4VOe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/E4VOe.png" alt="Celery Flower Output" /></a></p>
  <p><strong>Celery Configuration</strong></p>
  <pre><code># Async
  CELERY_BROKER_URL = os.environ.get('BROKER_URL', 'redis://localhost:6379')
  CELERY_RESULT_BACKEND = os.environ.get('RESULT_BACKEND', 'redis://localhost:6379')
  CELERY_ACCEPT_CONTENT = ['application/json']
  CELERY_TASK_SERIALIZER = 'json'
  CELERY_RESULT_SERIALIZER = 'json'
  CELERY_TIMEZONE = 'UTC'
  CELERY_ENABLE_UTC = True
  </code></pre>
  <p>Below is the successful execution of the command on the worker that was supposed to execute it:</p>
  <pre><code>&gt;&gt;&gt; schedule_monitoring(job.pk, 'start')
  'Applying monitoring action: START to Job: Test 1'
  &gt;&gt;&gt; schedule_monitoring(job.pk, 'stop')
  'Applying monitoring action: STOP to Job: Test 1'
  &gt;&gt;&gt; exit()
  Waiting up to 5 seconds.
  Sent all pending logs.
  root@9d045ff7dfc1:/app#
  </code></pre>
  <p>From debugging the worker; all I see is the following when the job starts, but then nothing interesting;</p>
  <pre><code>[2021-01-06 17:08:00,001: DEBUG/MainProcess] TaskPool: Apply &lt;function _trace_task_ret at 0x7f6adbc29680&gt; (args:('Operations.tasks.schedule_monitoring', '407e8a87-b3bf-4e8f-8a17-776a33ae5fea', {'lang': 'py', 'task': 'Operations.tasks.schedule_monitoring', 'id': '407e8a87-b3bf-4e8f-8a17-776a33ae5fea', 'shadow': None, 'eta': '2021-01-06T17:08:00+00:00', 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '407e8a87-b3bf-4e8f-8a17-776a33ae5fea', 'parent_id': None, 'argsrepr': &quot;(UUID('11118a85-20f2-488d-9a12-b8d200ea7a74'), 'start')&quot;, 'kwargsrepr': '{}', 'origin': 'gen442@31a9de56d061', 'reply_to': '24a8dc4c-2e5c-32ce-aa3d-84392d7cbf41', 'correlation_id': '407e8a87-b3bf-4e8f-8a17-776a33ae5fea', 'hostname': 'celery@bc4bb7af894f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': ['11118a85-20f2-488d-9a12-b8d200ea7a74', 'start'], 'kwargs': {}}, b'[[&quot;11118a85-20f2-488d-9a12-b8d200ea7a74&quot;, &quot;start&quot;], {}, {&quot;callbacks&quot;: null, &quot;errbacks&quot;: null, &quot;chain&quot;: null, &quot;chord&quot;: null}]', 'application/json', 'utf-8') kwargs:{})
  [2021-01-06 17:08:00,303: DEBUG/MainProcess] basic.qos: prefetch_count-&gt;32
  [2021-01-06 17:08:00,305: DEBUG/MainProcess] Task accepted: Operations.tasks.schedule_monitoring[407e8a87-b3bf-4e8f-8a17-776a33ae5fea] pid:44
  [2021-01-06 17:08:00,311: DEBUG/ForkPoolWorker-3] Resetting dropped connection: storage.googleapis.com
  [2021-01-06 17:08:00,383: DEBUG/ForkPoolWorker-3] https://storage.googleapis.com:443 &quot;GET /download/storage/v1/b/foo/o/bar?alt=media HTTP/1.1&quot; 200 96
  [2021-01-06 17:08:01,228: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:06,228: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:11,227: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:16,228: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:21,227: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:26,229: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  [2021-01-06 17:08:31,231: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
  </code></pre>