title: Django Celery ConnectionError: Too many heartbeats missed
body: <h1>Question</h1>
  
  <p>How can I solve the <code>ConnectionError: Too many heartbeats missed</code> from Celery?</p>
  
  <h2>Example Error</h2>
  
  <pre><code>[2013-02-11 15:15:38,513: ERROR/MainProcess] Error in timer: ConnectionError('Too many heartbeats missed', None, None, None, '')
  Traceback (most recent call last):
    File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 97, in apply_entry
      entry()
    File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 51, in __call__
      return self.fun(*self.args, **self.kwargs)
    File "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 153, in _reschedules
      return fun(*args, **kwargs)
    File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 265, in heartbeat_check
      return self.transport.heartbeat_check(self.connection, rate=rate)
    File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 134, in heartbeat_check
      return connection.heartbeat_tick(rate=rate)
    File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 837, in heartbeat_tick
      raise ConnectionError('Too many heartbeats missed')
  ConnectionError: Too many heartbeats missed
  </code></pre>
  
  <h2>App Overview</h2>
  
  <ul>
  <li>Django app using celery for periodic background tasks</li>
  <li>Hosted on Heroku</li>
  <li>Single task scheduled to run every 15 minutes via settings / celerybeat</li>
  <li>Messaging handled via CloudAMQP add-on</li>
  <li>Processes run by
  <ul>
  <li><code>web: newrelic-admin run-program gunicorn --workers=2 --worker-class=gevent someapp.wsgi:application</code></li>
  <li><code>scheduler: newrelic-admin run-program python manage.py celery worker -B -E --maxtasksperchild=1000 --loglevel=WARNING</code></li>
  </ul></li>
  </ul>
  
  <h2>Package Versions</h2>
  
  <p>Just what I think are relevant:</p>
  
  <pre><code>Django==1.4.3
  amqp==1.0.8
  billiard==2.7.3.20
  celery==3.0.14
  gevent==0.13.8
  greenlet==0.4.0
  kombu==2.5.6
  raven==3.1.10
  </code></pre>
  
  <h2>What I've Tried So Far</h2>
  
  <ul>
  <li>Correlating error with activities (doesn't seem to correlate with user's visiting app, background tasks being called, app idling)</li>
  <li>Googling / searching SO until my fingers were numb</li>
  <li>Upgrading packages to latest versions</li>
  <li>Various levels of logging</li>
  <li>Exception capturing with Sentry (doesn't appear in sentry)</li>
  <li>Cannot reproduce error locally under development environment, only in production on Heroku</li>
  </ul>
  
  <h2>Possible Relevant Info</h2>
  
  <ul>
  <li>I'm not sure exactly when this error first appeared (~ one month ago?)</li>
  <li>May be related in some way to the following changes (don't recall error before this, not 100% sure though)
  <ul>
  <li><code>celery==3.0.13</code> to <code>celery==3.0.14</code></li>
  <li><code>amqplib</code> -> <code>amqp</code></li>
  <li><code>kombu==2.4.8</code> to <code>kombu==2.5.4</code></li>
  </ul></li>
  <li>Error only appears in logs (doesn't get picked up by New Relic or getsentry.com)</li>
  </ul>