{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/5013","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/5013/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/5013/comments","events_url":"https://api.github.com/repos/celery/celery/issues/5013/events","html_url":"https://github.com/celery/celery/issues/5013","id":355245797,"node_id":"MDU6SXNzdWUzNTUyNDU3OTc=","number":5013,"title":"Celery worker closes unexpectedly","user":{"login":"bsolomon1124","id":25164676,"node_id":"MDQ6VXNlcjI1MTY0Njc2","avatar_url":"https://avatars1.githubusercontent.com/u/25164676?v=4","gravatar_id":"","url":"https://api.github.com/users/bsolomon1124","html_url":"https://github.com/bsolomon1124","followers_url":"https://api.github.com/users/bsolomon1124/followers","following_url":"https://api.github.com/users/bsolomon1124/following{/other_user}","gists_url":"https://api.github.com/users/bsolomon1124/gists{/gist_id}","starred_url":"https://api.github.com/users/bsolomon1124/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bsolomon1124/subscriptions","organizations_url":"https://api.github.com/users/bsolomon1124/orgs","repos_url":"https://api.github.com/users/bsolomon1124/repos","events_url":"https://api.github.com/users/bsolomon1124/events{/privacy}","received_events_url":"https://api.github.com/users/bsolomon1124/received_events","type":"User","site_admin":false},"labels":[{"id":39038,"node_id":"MDU6TGFiZWwzOTAzOA==","url":"https://api.github.com/repos/celery/celery/labels/Status:%20Feedback%20Needed%20%E2%9C%98","name":"Status: Feedback Needed ✘","color":"b59595","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-29T16:46:37Z","updated_at":"2018-09-01T12:11:01Z","closed_at":null,"author_association":"NONE","body":"## Checklist\r\n\r\n- [X] I have included the output of ``celery -A proj report`` in the issue. (bottom section)\r\n- [X] I have verified that the issue exists against the `master` branch of Celery.\r\n\r\n## Actual Behavior\r\n\r\nWe are using Celery, Supervisor, and Redis as (message broker) for task scheduling & async calls in a Django project.  (All version info given below.)\r\n\r\n### Summary\r\n\r\nUsing `celery worker` causes the worker to terminate suddenly (without CTRL-C) and without any meaningful log msgs.\r\n\r\n```bash\r\n$ celery -A project worker -l INFO -Q master -n master%i\r\n/path/to/project_name/py36/lib64/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\r\n  return f(*args, **kwds)\r\n/path/to/project_name/py36/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is\r\nabsolutely not recommended!\r\n\r\nPlease specify a different user using the --uid option.\r\n\r\nUser information: uid=0 euid=0 gid=0 egid=0\r\n\r\n  uid=uid, euid=euid, gid=gid, egid=egid,\r\n \r\n -------------- celery@master0 v4.2.1 (windowlicker)\r\n---- **** ----- \r\n--- * ***  * -- Linux-XX-XX-XX-XX-XX-with-redhat-7.1-Maipo 2018-08-29 18:12:57\r\n-- * - **** --- \r\n- ** ---------- [config]\r\n- ** ---------- .> app:         app:0x7fade7f909e8\r\n- ** ---------- .> transport:   redis://<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com:6379/0\r\n- ** ---------- .> results:     disabled://\r\n- *** --- * --- .> concurrency: 8 (prefork)\r\n-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)\r\n--- ***** ----- \r\n -------------- [queues]\r\n                .> master           exchange=master(direct) key=master\r\n                \r\n\r\n[tasks]\r\n  . dashboard.refresh_pickups\r\n  . dashboard.refresh_view\r\n  . fb.get_fb_ins\r\n  . google_analytics.events_all\r\n  . google_analytics.events_day\r\n  . google_analytics.metrics_all\r\n  . google_analytics.metrics_day\r\n  . master.insights\r\n  . master.insights_acct\r\n  . master.update\r\n  . master.update_acct\r\n  . singleton_write\r\n  . url.multiprocess_links\r\n  . urlanalytics.update_analytics\r\n\r\n\r\nINFO:celery.worker.consumer.connection [connection.py start:24]\r\n    asctime: 29/Aug/2018 18:12:57\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/connection.py\r\nConnected to redis://py3redisappserver.napxos.ng.0001.use1.cache.amazonaws.com:6379/0\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:43]\r\n    asctime: 29/Aug/2018 18:12:58\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: searching for neighbors\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:52]\r\n    asctime: 29/Aug/2018 18:12:59\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: all alone\r\n\r\n\r\nINFO:celery.apps.worker [worker.py on_consumer_ready:160]\r\n    asctime: 29/Aug/2018 18:12:59\r\n    pathname: /path/to/project_namepy36/lib/python3.6/site-packages/celery/apps/worker.py\r\ncelery@master0 ready.\r\n\r\n\r\nWARNING:celery.redirected [log.py write:235]\r\n    asctime: 29/Aug/2018 18:13:00\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/utils/log.py\r\nRestoring 1 unacknowledged message(s)\r\n```\r\n\r\n### Details\r\n\r\nWe are currently starting supervisord with just:\r\n\r\n```bash\r\nsupervisord -c ./supervisord.conf\r\n```\r\n\r\nIt looks like what's happening is that programs are spawning/starting for a few seconds, and then exiting with a nonzero code:\r\n\r\n```bash\r\n$ tail -n 25 /tmp/supervisor.log\r\n2018-08-28 16:20:52,057 INFO exited: celery_default (exit status 1; not expected)\r\n2018-08-28 16:20:52,216 INFO spawned: 'celery_default' with pid 48XX\r\n2018-08-28 16:20:52,883 INFO success: celery_google entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n2018-08-28 16:20:52,946 INFO exited: celery_dashboard (exit status 1; not expected)\r\n2018-08-28 16:20:53,253 INFO success: celery_default entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n2018-08-28 16:20:53,255 INFO spawned: 'celery_dashboard' with pid 48XX\r\n2018-08-28 16:20:54,760 INFO success: celery_dashboard entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n2018-08-28 16:20:55,746 INFO exited: celery_master (exit status 1; not expected)\r\n2018-08-28 16:20:55,796 INFO spawned: 'celery_master' with pid 48XX\r\n2018-08-28 16:20:56,052 INFO exited: celery_google (exit status 1; not expected)\r\n2018-08-28 16:20:56,369 INFO spawned: 'celery_google' with pid 48XX\r\n2018-08-28 16:20:56,999 INFO success: celery_master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n2018-08-28 16:20:57,063 INFO exited: celery_default (exit status 1; not expected)\r\n2018-08-28 16:20:57,370 INFO spawned: 'celery_default' with pid 48XX\r\n2018-08-28 16:20:57,370 INFO success: celery_google entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n2018-08-28 16:20:58,100 INFO exited: celery_dashboard (exit status 1; not expected)\r\n2018-08-28 16:20:58,400 INFO success: celery_default entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n```\r\n\r\nWe can see more by looking at the individual logs, but the actual source of the error isn't present.  Here is `/tmp` first:\r\n\r\n```bash\r\n$ ls -lahFG --sort=time /tmp/\r\n-rw-r--r--.  1 deploy  15M Aug 28 16:21 supervisor.log\r\n-rw-------.  1 root   1.1K Aug 28 16:21 celery_webcrawler-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    18K Aug 28 16:21 celery_google-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    17K Aug 28 16:21 celery_master-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    34K Aug 28 16:21 celery_master-stderr---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    16K Aug 28 16:21 celery_webcrawler-stderr---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    33K Aug 28 16:21 celery_google-stderr---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    435 Aug 28 16:20 celery_beat-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    203 Aug 28 16:20 celery_beat-stderr---supervisor-XXXXXXXXX.log\r\n-rw-r--r--.  1 root    32K Aug 28 16:20 celerybeat-schedule-refactor\r\n-rw-------.  1 root    17K Aug 28 16:20 celery_dashboard-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    29K Aug 28 16:20 celery_dashboard-stderr---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    16K Aug 28 16:20 celery_default-stdout---supervisor-XXXXXXXXX.log\r\n-rw-------.  1 root    30K Aug 28 16:20 celery_default-stderr---supervisor-XXXXXXXXX.log\r\n-rw-r--r--.  1 root      1 Aug 27 08:17 analytics.log\r\n```\r\n\r\nNow, every one of those `stderr` logs shows the same pattern that simply cycles over and over again, in parallel with `supervisor.log`:\r\n\r\n```\r\n/path/to/project_name/py36/lib64/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\r\n  return f(*args, **kwds)\r\n/path/to/project_name/py36/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is\r\nabsolutely not recommended!\r\n\r\nPlease specify a different user using the --uid option.\r\n\r\nUser information: uid=0 euid=0 gid=0 egid=0\r\n\r\n  uid=uid, euid=euid, gid=gid, egid=egid,\r\n\r\nINFO:celery.worker.consumer.connection [connection.py start:24]\r\n    asctime: 28/Aug/2018 20:20:52\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/connection.py\r\nConnected to redis://<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com:6379/0\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:43]\r\n    asctime: 28/Aug/2018 20:20:52\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: searching for neighbors\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:47]\r\n    asctime: 28/Aug/2018 20:20:53\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: sync with 1 nodes\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:50]\r\n    asctime: 28/Aug/2018 20:20:53\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: sync complete\r\n\r\n\r\nINFO:celery.apps.worker [worker.py on_consumer_ready:160]\r\n    asctime: 28/Aug/2018 20:20:53\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/apps/worker.py\r\ncelery@masterip-XX-XX-XX-XX.ec2.internal-0 ready.\r\n\r\n\r\nINFO:celery.worker.control [control.py hello:314]\r\n    asctime: 28/Aug/2018 20:20:53\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/control.py\r\nsync with celery@defaultip-XX-XX-XX-XX.ec2.internal-0\r\n\r\n\r\nWARNING:celery.redirected [log.py write:235]\r\n    asctime: 28/Aug/2018 20:20:55\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/utils/log.py\r\nRestoring 1 unacknowledged message(s)\r\n```\r\n\r\nUnfortunately none of this hints at (as far as I can tell) the actual cause of the \"exit status 1\" logs from `supervisor.log`.\r\n\r\nAs a result, the tasks are not running as they are scheduled by `celery.schedules.crontab`.\r\n\r\nHere is an output from `celery worker` with loglevel=DEBUG:\r\n\r\n```\r\n$ /path/to/project_name/py36/bin/celery -A project worker --autoscale=5,1 --loglevel=DEBUG -Q default -n default%h-%i\r\n/path/to/project_name/py36/lib64/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\r\n  return f(*args, **kwds)\r\n/path/to/project_name/py36/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is\r\nabsolutely not recommended!\r\n\r\nPlease specify a different user using the --uid option.\r\n\r\nUser information: uid=0 euid=0 gid=0 egid=0\r\n\r\n  uid=uid, euid=euid, gid=gid, egid=egid,\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Preparing bootsteps.\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Building graph...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer}\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Preparing bootsteps.\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Building graph...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}\r\n\r\n \r\n -------------- celery@defaultip-XX-XX-XX-XX.ec2.internal-0 v4.2.1 (windowlicker)\r\n---- **** ----- \r\n--- * ***  * -- Linux-XXXXXX-with-redhat-7.1-Maipo 2018-08-29 16:36:32\r\n-- * - **** --- \r\n- ** ---------- [config]\r\n- ** ---------- .> app:         project:0x7f26cc1708d0\r\n- ** ---------- .> transport:   redis://<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com:6379/0\r\n- ** ---------- .> results:     disabled://\r\n- *** --- * --- .> concurrency: {min=1, max=5} (prefork)\r\n-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)\r\n--- ***** ----- \r\n -------------- [queues]\r\n                .> default          exchange=default(direct) key=default\r\n                \r\n\r\n[tasks]\r\n  . celery.accumulate\r\n  . celery.backend_cleanup\r\n  . celery.chain\r\n  . celery.chord\r\n  . celery.chord_unlock\r\n  . celery.chunks\r\n  . celery.group\r\n  . celery.map\r\n  . celery.starmap\r\n  . dashboard.refresh_pickups\r\n  . dashboard.refresh_view\r\n  . fb.get_fb_ins\r\n  . google_analytics.events_all\r\n  . google_analytics.events_day\r\n  . google_analytics.metrics_all\r\n  . google_analytics.metrics_day\r\n  . master.insights\r\n  . master.insights_acct\r\n  . master.update\r\n  . master.update_acct\r\n  . singleton_write\r\n  . url.multiprocess_links\r\n  . urlanalytics.update_analytics\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Starting Hub\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Starting Pool\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Starting Autoscaler\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Starting Consumer\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Connection\r\n\r\n\r\nINFO:celery.worker.consumer.connection [connection.py start:24]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/connection.py\r\nConnected to redis://<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com:6379/0\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Events\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Mingle\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:43]\r\n    asctime: 29/Aug/2018 16:36:32\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: searching for neighbors\r\n\r\n\r\nINFO:celery.worker.consumer.mingle [mingle.py sync:52]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/mingle.py\r\nmingle: all alone\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Gossip\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Tasks\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Control\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting Heart\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py start:120]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n^-- substep ok\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Starting event loop\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Hub.register Autoscaler...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Hub.register Pool...\r\n\r\n\r\nINFO:celery.apps.worker [worker.py on_consumer_ready:160]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/apps/worker.py\r\ncelery@defaultip-10-0-0-85.ec2.internal-0 ready.\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Closing Hub...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Closing Pool...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Closing Autoscaler...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Closing Consumer...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Stopping Consumer...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Connection...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Events...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Mingle...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Gossip...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Tasks...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Control...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing Heart...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Closing event loop...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping event loop...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Heart...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Control...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Tasks...\r\n\r\n\r\nDEBUG:celery.worker.consumer.tasks [tasks.py stop:54]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/tasks.py\r\nCanceling task consumer...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Gossip...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Mingle...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Events...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Stopping Connection...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Stopping Autoscaler...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:33\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Stopping Pool...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Worker: Stopping Hub...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Heart...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Control...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Tasks...\r\n\r\n\r\nDEBUG:celery.worker.consumer.tasks [tasks.py stop:54]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/tasks.py\r\nCanceling task consumer...\r\n\r\n\r\nDEBUG:celery.worker.consumer.tasks [tasks.py shutdown:61]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/worker/consumer/tasks.py\r\nClosing consumer channel...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Gossip...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Events...\r\n\r\n\r\nDEBUG:celery.bootsteps [bootsteps.py _debug:262]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/bootsteps.py\r\n| Consumer: Shutdown Connection...\r\n\r\n\r\nWARNING:celery.redirected [log.py write:235]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/utils/log.py\r\nRestoring 1 unacknowledged message(s)\r\n\r\n\r\nDEBUG:celery.concurrency.asynpool [asynpool.py _help_stuff_finish:1281]\r\n    asctime: 29/Aug/2018 16:36:34\r\n    pathname: /path/to/project_name/py36/lib/python3.6/site-packages/celery/concurrency/asynpool.py\r\nremoving tasks from inqueue until task handler finished\r\n```\r\n\r\nAfter that final mysterious log message, **worker terminates suddenly without CTRL+C.**\r\n\r\n----\r\n\r\n**Version info**:\r\n\r\n```bash\r\n$ python --version\r\nPython 3.6.5\r\n\r\n$ python -c 'import django; print(django.__version__)'\r\n2.1\r\n\r\n$ celery --version\r\n4.2.1 (windowlicker)\r\n\r\n$ supervisord --version\r\n4.0.0.dev0\r\n\r\n$ redis-cli --version\r\nredis-cli 4.0.11 (git:230489b2)\r\n```\r\n\r\nSystem: Red Hat 4.8.5-28 on linux Amazon EC2 instance\r\n\r\n----\r\n\r\n**Project skeleton**:\r\n\r\n    project_name\r\n    ├── celeryconfig.py\r\n    ├── manage.py\r\n    ├── supervisord.conf\r\n    ├── py36                # virtual environment\r\n    ├── apps/               # houses Django apps with tasks.py, etc\r\n    └── project\r\n        ├── __init__.py\r\n        ├── celery_app.py\r\n        └── settings.py\r\n\r\n----\r\n\r\n**supervisord.conf** (most comments removed):\r\n\r\n    [supervisord]\r\n    environment=LOGGING=\"%(ENV_LOGGING)s\",DJ_LOGGING=\"%(ENV_DJ_LOGGING)s\",SECRET_KEY=\"%(ENV_SECRET_KEY)s\",DEBUG=\"%(ENV_DEBUG)s\",DATABASE_ENGINE=\"%(ENV_DATABASE_ENGINE)s\",DATABASE_NAME=\"%(ENV_DATABASE_NAME)s\",DATABASE_USER=\"%(ENV_DATABASE_USER)s\",DATABASE_PASSWORD=\"%(ENV_DATABASE_PASSWORD)s\",DATABASE_HOST=\"%(ENV_DATABASE_HOST)s\",USER_ID=\"%(ENV_USER_ID)s\",DATABASE_PORT=\"%(ENV_DATABASE_PORT)s\",EMAIL_HOST_USER=\"%(ENV_EMAIL_HOST_USER)s\",EMAIL_HOST_PASSWORD=\"%(ENV_EMAIL_HOST_PASSWORD)s\",GOOGLE_ANALYTICS_PROPERTY_ID=\"%(ENV_GOOGLE_ANALYTICS_PROPERTY_ID)s\",GOOGLE_ANALYTICS_DOMAIN=\"%(ENV_GOOGLE_ANALYTICS_DOMAIN)s\",REDIS_SERVER=\"%(ENV_REDIS_SERVER)s\"\r\n    logfile=/tmp/supervisor.log           ; supervisord log file\r\n    logfile_maxbytes=50MB                 ; maximum size of logfile before rotation\r\n    logfile_backups=10                    ; number of backed up logfiles\r\n    loglevel=info                         ; info, debug, warn, trace\r\n    user=root\r\n\r\n\r\n    [unix_http_server]\r\n    file=/tmp/supervisor.sock               ; path to socket file\r\n\r\n    [rpcinterface:supervisor]\r\n    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface\r\n\r\n    [supervisorctl]\r\n    serverurl=unix:///tmp/supervisor.sock\r\n\r\n    [program:celery_beat]\r\n    command=/path/to/project_name/py36/bin/celery -A project beat --loglevel WARNING --pidfile=/tmp/celerybeat-refactor.pid --schedule=/tmp/celerybeat-schedule-refactor\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [program:celery_webcrawler]\r\n    command=/path/to/project_name/py36/bin/celery -A project worker --autoscale=1,1 --loglevel INFO -Q webcrawler -n webcrawler%%h-%%i\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [program:celery_default]\r\n    command=/path/to/project_name/py36/bin/celery -A project worker --autoscale=5,1 --loglevel INFO -Q default -n default%%h-%%i\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [program:celery_google]\r\n    command=/path/to/project_name/py36/bin/celery -A project worker --autoscale=2,1 --loglevel INFO -Q google_analytics -n google_analytics%%h-%%i\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [program:celery_master]\r\n    command=/path/to/project_name/py36/bin/celery -A project worker --autoscale=5,1 --loglevel INFO -Q master -n master%%h-%%i\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [program:celery_dashboard]\r\n    command=/path/to/project_name/py36/bin/celery -A project worker --autoscale=1,1 --loglevel INFO -Q dashboard -n dashboard%%h-%%i\r\n    autostart=true\r\n    autorestart=true\r\n\r\n    [group:workers]\r\n    programs=celery_default,celery_google,celery_master,celery_dashboard,celery_beat,celery_webcrawler\r\n\r\n----\r\n\r\n**Module contents**:\r\n\r\n_celeryconfig.py_:\r\n\r\n```python\r\nimport os\r\n\r\nfrom celery.schedules import crontab\r\nfrom kombu import Exchange, Queue\r\n\r\nfrom project.settings import _verboselogfmt\r\n\r\nworker_hijack_root_logger = False\r\nworker_log_format = _verboselogfmt\r\nworker_task_log_format = _verboselogfmt\r\n\r\nbroker_url = 'redis://' + os.environ['REDIS_SERVER'] + ':6379/0'\r\n\r\ntask_default_queue = 'default'\r\ntask_default_exchange_type= 'direct'\r\ntask_default_routing_key = 'default'\r\n\r\ntask_serializer = 'pickle'\r\nresult_serializer = 'pickle'\r\n\r\ntask_queues = (\r\n    Queue('default', Exchange('default'), routing_key='default'),\r\n    Queue('facebook', Exchange('facebook'), routing_key='facebook'),\r\n    Queue('twitter', Exchange('twitter'), routing_key='twitter'),\r\n    Queue('webcrawler', Exchange('webcrawler'), routing_key='webcrawler'),\r\n    Queue('google_analytics', Exchange('google_analytics'), routing_key='google_analytics'),\r\n    Queue('master', Exchange('master'), routing_key='master'),\r\n    Queue('dashboard', Exchange('dashboard'), routing_key='dashboard'),\r\n    Queue('urlanalytics', Exchange('urlanalytics'), routing_key='urlanalytics')\r\n)\r\n\r\nbeat_schedule = {\r\n    'daily_google_analytics_events_all': {\r\n        'task': 'google_analytics.events_all',\r\n        'schedule': crontab(hour='6', minute=5),\r\n    },\r\n    'daily_google_analytics_events_day': {\r\n        'task': 'google_analytics.events_day',\r\n        'schedule': crontab(hour='6', minute=0),\r\n        'args': [1]\r\n    },\r\n    'daily_google_analytics_metrics_all': {\r\n        'task': 'google_analytics.metrics_all',\r\n        'schedule': crontab(hour='6', minute=10),\r\n    },\r\n    'daily_google_analytics_metrics_day': {\r\n        'task': 'google_analytics.metrics_day',\r\n        'schedule': crontab(hour='6', minute=15),\r\n        'args': [1]\r\n    },\r\n    'daily_social': {\r\n        'task': 'master.update',\r\n        'schedule': crontab(hour='6', minute=0),\r\n    },\r\n    'webcrawler_crawl': {\r\n        'task': 'webcrawler.crawl',\r\n        'schedule': crontab(hour='5,10,15,20', minute=0),\r\n    },\r\n    'dashboard_refresh': {\r\n        'task': 'dashboard.refresh_view',\r\n        'schedule': crontab(hour='5,10,15,20', minute=0),\r\n    },\r\n    'update_url_clicks': {\r\n        'task': 'urlanalytics.update_analytics',\r\n        'schedule': crontab(hour='5', minute=0),\r\n    },\r\n}\r\n\r\ntask_routes = {\r\n    'master.update': {\r\n        'queue': 'master',\r\n        'routing_key': 'master'\r\n    },\r\n    'master.update_acct': {\r\n        'queue': 'master',\r\n        'routing_key': 'master'\r\n    },\r\n    'webcrawler.crawl': {\r\n        'queue': 'webcrawler',\r\n        'routing_key': 'webcrawler'\r\n    },\r\n    'google_analytics.events_all': {\r\n        'queue': 'google_analytics',\r\n        'routing_key': 'google_analytics'\r\n    },\r\n    'google_analytics.events_day': {\r\n        'queue': 'google_analytics',\r\n        'routing_key': 'google_analytics'\r\n    },\r\n    'google_analytics.metrics_day': {\r\n        'queue': 'google_analytics',\r\n        'routing_key': 'google_analytics',\r\n    },\r\n    'google_analytics.metrics_all': {\r\n        'queue': 'google_analytics',\r\n        'routing_key': 'google_analytics'\r\n    },\r\n    'dashboard.refresh_view': {\r\n        'queue': 'dashboard',\r\n        'routing_key': 'dashboard'\r\n    },\r\n    'dashboard.refresh_pickups': {\r\n        'queue': 'dashboard',\r\n        'routing_key': 'dashboard'\r\n    },\r\n    'urlanalytics.update_analytics': {\r\n        'queue': 'urlanalytics',\r\n        'routing_key': 'urlanalytics'\r\n    },\r\n}\r\n```\r\n\r\n_project/celery_app.py_\r\n\r\n```python\r\nimport os\r\n\r\nfrom celery import Celery\r\nimport django\r\n\r\nimport celeryconfig\r\n\r\nos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')\r\ndjango.setup()\r\n\r\napp = Celery('app')\r\napp.config_from_object(celeryconfig)\r\napp.autodiscover_tasks(force=True)\r\n```\r\n\r\n_project/\\_\\_init\\_\\_.py_:\r\n\r\n```python\r\nfrom .celery_app import app\r\n__all__ = ['app']\r\n```\r\n\r\nPartial snippet from _project/settings.py_:\r\n\r\n```python\r\nimport logging\r\nimport logging.config\r\nimport os\r\nimport sys\r\n\r\n_ENV = os.environ\r\n\r\n_DJANGO_APPS = [\r\n    'django.contrib.admin',\r\n    'django.contrib.auth',\r\n    'django.contrib.contenttypes',\r\n    'django.contrib.sessions',\r\n    'django.contrib.messages',\r\n    'django.contrib.staticfiles',\r\n    'django.contrib.sites',\r\n    'django.contrib.postgres'\r\n]\r\n\r\n_THIRD_PTY_APPS = [\r\n    'allauth',\r\n    'allauth.account',\r\n    'allauth.socialaccount.providers.google',\r\n    'allauth.socialaccount',\r\n    'bootstrap3',\r\n    'corsheaders',\r\n    'envelope',\r\n    'rest_framework',\r\n]\r\n\r\n_PROJECT_APPS = [\r\n    'accounts',\r\n    'dashboard',\r\n    'google_analytics',\r\n    'lib',\r\n    'master',\r\n    'translate',\r\n    'url',\r\n    'urlanalytics',\r\n    'webcrawler',\r\n    # and a few others\r\n]\r\n\r\nLOGGING_CONFIG = None\r\n\r\n_verboselogfmt = \"\"\"\r\n%(levelname)s:%(name)s [%(filename)s %(funcName)s:%(lineno)s]\r\n    asctime: %(asctime)s\r\n    pathname: %(pathname)s\r\n%(message)s\r\n\"\"\"\r\n\r\nLOGGING = {\r\n    'version': 1,\r\n    'disable_existing_loggers': True,\r\n    'formatters': {\r\n        'verbose': {\r\n            'format': _verboselogfmt,\r\n            'datefmt': '%d/%b/%Y %H:%M:%S'\r\n        },\r\n        'simple': {\r\n            'format': logging.BASIC_FORMAT\r\n        },\r\n    },\r\n    'handlers': {\r\n        'file': {\r\n            'level': 'ERROR',\r\n            'class': 'logging.FileHandler',\r\n            'filename': '/tmp/analytics.log',\r\n            'formatter': 'verbose'\r\n        },\r\n        'console': {\r\n            'class': 'logging.StreamHandler',\r\n            'formatter': 'verbose'\r\n        },\r\n    },\r\n    'loggers': {\r\n        # Root\r\n        '': {\r\n            'level': 'WARNING',\r\n            'handlers': ['console']\r\n        },\r\n        'django': {\r\n            'handlers': ['console'],\r\n            'level': _ENV.get('DJ_LOGGING', 'INFO').upper(),\r\n            'propagate': True\r\n        },\r\n        'app': {\r\n            'handlers': ['console', 'file'],\r\n            'level': _ENV.get('LOGGING', 'INFO').upper(),\r\n            'propagate': False\r\n        },\r\n    }\r\n}\r\n\r\nlogging.config.dictConfig(LOGGING)\r\n```\r\n\r\nA few environment variables:\r\n\r\n```bash\r\nexport DEBUG=False\r\nexport LOGGING='DEBUG'\r\nexport DJ_LOGGING='WARNING'\r\nexport REDIS_SERVER='<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com'\r\n```\r\n\r\n## Output of ``celery -A project report``\r\n\r\n```\r\n$ celery -A project report\r\n/path/to/project_name/py36/lib64/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\r\n  return f(*args, **kwds)\r\n\r\nsoftware -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:3.6.5\r\n            billiard:3.5.0.4 redis:2.10.6\r\nplatform -> system:Linux arch:64bit, ELF imp:CPython\r\nloader   -> celery.loaders.app.AppLoader\r\nsettings -> transport:redis results:disabled\r\n\r\nExchange: <class 'kombu.entity.Exchange'>\r\nQueue: <class 'kombu.entity.Queue'>\r\nbeat_schedule: {\r\n    'daily_google_analytics_events_all': {   'schedule': <crontab: 5 6 * * * (m/h/d/dM/MY)>,\r\n                                             'task': 'google_analytics.events_all'},\r\n    'daily_google_analytics_events_day': {   'args': [1],\r\n                                             'schedule': <crontab: 0 6 * * * (m/h/d/dM/MY)>,\r\n                                             'task': 'google_analytics.events_day'},\r\n    'daily_google_analytics_metrics_all': {   'schedule': <crontab: 10 6 * * * (m/h/d/dM/MY)>,\r\n                                              'task': 'google_analytics.metrics_all'},\r\n    'daily_google_analytics_metrics_day': {   'args': [1],\r\n                                              'schedule': <crontab: 15 6 * * * (m/h/d/dM/MY)>,\r\n                                              'task': 'google_analytics.metrics_day'},\r\n    'daily_social': {   'schedule': <crontab: 0 6 * * * (m/h/d/dM/MY)>,\r\n                        'task': 'master.update'},\r\n    'dashboard_refresh': {   'schedule': <crontab: 0 5,10,15,20 * * * (m/h/d/dM/MY)>,\r\n                             'task': 'dashboard.refresh_view'},\r\n    'update_url_clicks': {   'schedule': <crontab: 0 5 * * * (m/h/d/dM/MY)>,\r\n                             'task': 'urlanalytics.update_analytics'},\r\n    'webcrawler_crawl': {   'schedule': <crontab: 0 5,10,15,20 * * * (m/h/d/dM/MY)>,\r\n                            'task': 'webcrawler.crawl'}}\r\nbroker_url: 'redis://<elastic_cache_redis_server_name>.abcdef.ng.XXXX.use1.cache.amazonaws.com:6379/0'\r\ncrontab: <class 'celery.schedules.crontab'>\r\nos: <module 'os' from '/path/to/project_name/py36/lib64/python3.6/os.py'>\r\nresult_serializer: 'pickle'\r\ntask_default_exchange_type: 'direct'\r\ntask_default_queue: 'default'\r\ntask_default_routing_key: '********'\r\ntask_queues: \r\n    (<unbound Queue default -> <unbound Exchange default(direct)> -> default>,\r\n <unbound Queue facebook -> <unbound Exchange facebook(direct)> -> facebook>,\r\n <unbound Queue twitter -> <unbound Exchange twitter(direct)> -> twitter>,\r\n <unbound Queue webcrawler -> <unbound Exchange webcrawler(direct)> -> webcrawler>,\r\n <unbound Queue google_analytics -> <unbound Exchange google_analytics(direct)> -> google_analytics>,\r\n <unbound Queue master -> <unbound Exchange master(direct)> -> master>,\r\n <unbound Queue dashboard -> <unbound Exchange dashboard(direct)> -> dashboard>,\r\n <unbound Queue urlanalytics -> <unbound Exchange urlanalytics(direct)> -> urlanalytics>)\r\ntask_routes: {\r\n    'dashboard.refresh_pickups': {   'queue': 'dashboard',\r\n                                     'routing_key': '********'},\r\n    'dashboard.refresh_view': {'queue': 'dashboard', 'routing_key': '********'},\r\n    'google_analytics.events_all': {   'queue': 'google_analytics',\r\n                                       'routing_key': '********'},\r\n    'google_analytics.events_day': {   'queue': 'google_analytics',\r\n                                       'routing_key': '********'},\r\n    'google_analytics.metrics_all': {   'queue': 'google_analytics',\r\n                                        'routing_key': '********'},\r\n    'google_analytics.metrics_day': {   'queue': 'google_analytics',\r\n                                        'routing_key': '********'},\r\n    'master.update': {'queue': 'master', 'routing_key': '********'},\r\n    'master.update_acct': {'queue': 'master', 'routing_key': '********'},\r\n    'urlanalytics.update_analytics': {   'queue': 'urlanalytics',\r\n                                         'routing_key': '********'},\r\n    'webcrawler.crawl': {'queue': 'webcrawler', 'routing_key': '********'}}\r\ntask_serializer: 'pickle'\r\nworker_hijack_root_logger: False\r\nworker_log_format: ('\\n'\r\n '%(levelname)s:%(name)s [%(filename)s '\r\n '%(funcName)s:%(lineno)s]\\n'\r\n '    asctime: %(asctime)s\\n'\r\n '    pathname: %(pathname)s\\n'\r\n '%(message)s\\n')\r\nworker_task_log_format: ('\\n'\r\n '%(levelname)s:%(name)s [%(filename)s '\r\n '%(funcName)s:%(lineno)s]\\n'\r\n '    asctime: %(asctime)s\\n'\r\n '    pathname: %(pathname)s\\n'\r\n '%(message)s\\n')\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/417855477","html_url":"https://github.com/celery/celery/issues/5013#issuecomment-417855477","issue_url":"https://api.github.com/repos/celery/celery/issues/5013","id":417855477,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzg1NTQ3Nw==","user":{"login":"thedrow","id":48936,"node_id":"MDQ6VXNlcjQ4OTM2","avatar_url":"https://avatars2.githubusercontent.com/u/48936?v=4","gravatar_id":"","url":"https://api.github.com/users/thedrow","html_url":"https://github.com/thedrow","followers_url":"https://api.github.com/users/thedrow/followers","following_url":"https://api.github.com/users/thedrow/following{/other_user}","gists_url":"https://api.github.com/users/thedrow/gists{/gist_id}","starred_url":"https://api.github.com/users/thedrow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thedrow/subscriptions","organizations_url":"https://api.github.com/users/thedrow/orgs","repos_url":"https://api.github.com/users/thedrow/repos","events_url":"https://api.github.com/users/thedrow/events{/privacy}","received_events_url":"https://api.github.com/users/thedrow/received_events","type":"User","site_admin":false},"created_at":"2018-09-01T12:11:01Z","updated_at":"2018-09-01T12:11:01Z","author_association":"CONTRIBUTOR","body":"Note that Celery 3.x is not supported."}}
{"actor":{"display_login":"thedrow"}}