title: How to retry Django Celery task on Exception when Internal Server Error is raised?
body: <p>I am trying to use Celery to send anywhere from 1 to about 25 consecutive requests to a third-party API. The measure of success is whether I get a URL back in the response payload: <code>response_json[&quot;data&quot;][&quot;url&quot;]</code>. Celery or not, sometimes I get the data I expect, and sometimes I don't.</p>
  <p>I decided to experiment with Celery to retry the API call while taking advantage of the built-in <a href="https://docs.celeryq.dev/en/stable/userguide/tasks.html#automatic-retry-for-known-exceptions" rel="nofollow noreferrer">exponential backoff</a>, which sounds perfectly suited to my needs, but I am struggling to implement.</p>
  <p>When the data I am expecting is not available in the response payload, I get a <code>TypeError: 'type' object is not iterable</code> (i.e., there is no such item in the response). But I am also getting an <code>Internal Server Error</code>, which I wonder whether I need to handle or whether I could use to trigger the retry.</p>
  <p>Here is a example of one of several similar approaches I have tried:</p>
  <pre><code>@shared_task(autoretry_for=(Exception), retry_backoff=True, retry_backoff_max=120)
  def third_party_api_request(payload, api_url, headers):
      response = requests.request(&quot;POST&quot;, api_url, headers=headers, data=payload)
      response_json = response.json()
      return response_json[&quot;data&quot;][&quot;url&quot;]
  
  # output:
  Internal Server Error:
  -- snip --
      autoretry_for = tuple(
  TypeError: 'type' object is not iterable
  
  </code></pre>
  <p>Another approach I have tried:</p>
  <pre><code>@shared_task(bind=True)
  def third_party_api_request(self, payload, api_url, headers):
      try:
          response = requests.request(&quot;POST&quot;, api_url, headers=headers, data=payload)
          response_json = response.json()
          return response_json[&quot;data&quot;][&quot;url&quot;]
      except TypeError as exc:
          logger.error(&quot;Error sending request to API: %s&quot;, exc)
          raise self.retry(exc=exc)
  
  # output:
  ERROR 2022-04-22 17:31:40,131 tasks 72780 123145528369152 Error sending request to API: 'NoneType' object is not subscriptable
  Internal Server Error:
  -- snip --
  TypeError: 'NoneType' object is not subscriptable
  -- snip --
      raise ret
  celery.exceptions.Retry: Retry in 180s: TypeError(&quot;'NoneType' object is not subscriptable&quot;)
  ERROR 2022-04-22 17:31:40,409 log 72780 123145528369152 Internal Server Error:
  -- snip --
  TypeError: 'NoneType' object is not subscriptable
  -- snip --
     raise ret
  celery.exceptions.Retry: Retry in 180s: TypeError(&quot;'NoneType' object is not subscriptable&quot;)
  
  </code></pre>
  <p>And yet another approach, with similar results:</p>
  <pre><code>@shared_task(autoretry_for=(TypeError), retry_backoff=True, retry_backoff_max=120)
  def send_http_request_to_proctoru_task(payload, api_url, headers):
      response = requests.request(&quot;POST&quot;, api_url, headers=headers, data=payload)
      response_json = response.json()
      try:
          return response_json[&quot;data&quot;][&quot;url&quot;]
      except TypeError:
          logger.error(&quot;API response: %s&quot;, response_json)
          raise
  </code></pre>