{"action":"opened","issue":{"url":"https://api.github.com/repos/cundi/blog/issues/19","labels_url":"https://api.github.com/repos/cundi/blog/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/cundi/blog/issues/19/comments","events_url":"https://api.github.com/repos/cundi/blog/issues/19/events","html_url":"https://github.com/cundi/blog/issues/19","id":130588625,"number":19,"title":"使用Celery和Redis为Django设置异步任务队列［翻译］","user":{"login":"cundi","id":10941075,"avatar_url":"https://avatars.githubusercontent.com/u/10941075?v=3","gravatar_id":"","url":"https://api.github.com/users/cundi","html_url":"https://github.com/cundi","followers_url":"https://api.github.com/users/cundi/followers","following_url":"https://api.github.com/users/cundi/following{/other_user}","gists_url":"https://api.github.com/users/cundi/gists{/gist_id}","starred_url":"https://api.github.com/users/cundi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cundi/subscriptions","organizations_url":"https://api.github.com/users/cundi/orgs","repos_url":"https://api.github.com/users/cundi/repos","events_url":"https://api.github.com/users/cundi/events{/privacy}","received_events_url":"https://api.github.com/users/cundi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-02T06:29:51Z","updated_at":"2016-02-02T06:29:51Z","closed_at":null,"body":"原始链接：http://michal.karzynski.pl/blog/2014/01/18/performance-testing-django-applications-with-the-grinder/  \r\n\r\n# Setting up an asynchronous task queue for Django using Celery and Redis\r\n`MAY 18TH, 2014`\r\n\r\nCelery is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background. A Celery powered application can respond to user requests quickly, while long-running tasks are passed onto the queue. In this article we will demonstrate how to add Celery to a Django application using Redis.  \r\n\r\n>Celery uses a broker to pass messages between your application and Celery worker processes. In this article we will set up Redis as the message broker. You should note that persistence is not the main goal of this data store, so your queue could be erased in the event of a power failure or other crash. Keep this in mind and don’t use the job queue to store application state. If you need your queue to be have persistence, use another message broker such as RabbitMQ.  \r\n\r\n## Prerequisites 前置知识\r\n\r\nIn this article we will add Celery to a Django application running in a Python virtualenv. I will assume that the virtual environment is located in the directory /webapps/hello_django/ and that the application is up an running. You can follow steps in my previous article to set up Django in virtualenv running on Nginx and Gunicorn.  \r\n\r\nThis article was tested on a server running Debian 7, so everything should also work on an Ubuntu server or other Debian-based distribution. If you’re using an RPM-based distro (such as CentOS), you will need to replace the aptitude commands by their yum counterparts and if you’re using FreeBSD you can install the components from ports. If you don’t have a server to play with, I can recommend the inexpensive VPS servers offered by Digital Ocean.  \r\n\r\n## Update your system\r\n\r\nLet’s get started by making sure your system is up to date.  \r\n\r\n```shell\r\n$ sudo aptitude update\r\n$ sudo aptitude upgrade\r\n```\r\n\r\n## Install Redis\r\n\r\nThe first piece of software we’ll install is Redis.  \r\n\r\n```shell\r\n$ sudo aptitude install redis-server\r\n$ redis-server --version\r\nRedis server version 2.4.14 (00000000:0)\r\n```\r\n\r\nCheck if Redis is up and accepting connections:  \r\n\r\n```shell\r\n$ redis-cli ping\r\nPONG\r\n```\r\n\r\n## Installing Celery in your aplication’s virtualenv\r\n\r\nLet’s add Celery to your application’s virtual Python environment. \r\n\r\nFirst we’ll switch to the application user and activate the `virtualenv`  \r\n\r\n```shell\r\n$ sudo su - hello\r\nhello@django:~$ source bin/activate\r\n```\r\n\r\nNow we can use `pip` to install Celery along with its Redis bindings and dependencies:  \r\n\r\n``shell\r\n(hello_django)hello@django:~$ pip install celery[redis]\r\nDownloading/unpacking celery[redis]\r\n(...)\r\nSuccessfully installed celery pytz billiard kombu redis anyjson amqp\r\nCleaning up...\r\n```\r\n\r\n## Setting up Celery support in your Django application\r\n\r\nIn order to use Celery as part of your Django application you’ll need to create a few files and tweak some settings. Let’s start by adding Celery-related configuration variables to `settings.py`  \r\n\r\n```python\r\n/webapps/hello_django/hello/hello/settings.py\r\n# CELERY SETTINGS\r\nBROKER_URL = 'redis://localhost:6379/0'\r\nCELERY_ACCEPT_CONTENT = ['json']\r\nCELERY_TASK_SERIALIZER = 'json'\r\nCELERY_RESULT_SERIALIZER = 'json'\r\n```\r\n\r\nNow we’ll create a file named `celery.py`, which will instantiate Celery, creating a so called Celery application. You can find more information about available Celery application settings in the documentation.  \r\n\r\n```python\r\n/webapps/hello_django/hello/hello/celery.py\r\nfrom __future__ import absolute_import\r\n\r\nimport os\r\nfrom celery import Celery\r\nfrom django.conf import settings\r\n\r\n# set the default Django settings module for the 'celery' program.\r\nos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hello.settings')\r\n\r\napp = Celery('hello_django')\r\n\r\n# Using a string here means the worker will not have to\r\n# pickle the object when using Windows.\r\napp.config_from_object('django.conf:settings')\r\napp.autodiscover_tasks(lambda: settings.INSTALLED_APPS)\r\n```\r\n\r\nIn order to instantiate the Celery app every time our Django application is started, we can add the following lines to the `__init__.py` file in our Django `proj.proj` module. This will make sure that celery task use this app.  \r\n\r\n```python\r\n/webapps/hello_django/hello/hello/__init__.py\r\nfrom __future__ import absolute_import\r\nfrom .celery import app as celery_app\r\n```\r\n\r\n## Create your first tasks\r\n\r\nWe will now add an app called `testapp` to our Django project and add some tasks to this app. Let’s start by creating the app:  \r\n\r\n```python\r\n(hello_django)hello@django:~/hello$ python manage.py startapp testapp\r\n```\r\n\r\nMake sure that the app is added to `INSTALLED_APPS` in `settings.py`  \r\n\r\n```python\r\n/webapps/hello_django/hello/hello/settings.py\r\nINSTALLED_APPS = (\r\n    # (...)\r\n    'testapp',\r\n)\r\n```\r\n\r\nCreate a file called `tasks.py` in your apps’s directory and add the code of your first Celery task to the file.  \r\n\r\n```python\r\n/webapps/hello_django/hello/testapp/tasks.py\r\nfrom __future__ import absolute_import\r\n\r\nfrom celery import shared_task\r\n\r\n@shared_task\r\ndef test(param):\r\n    return 'The test task executed with argument \"%s\" ' % param\r\n```\r\n\r\nFind more information about writing task functions in the docs.  \r\n\r\nIf you created all files as outlined above, you should see the following directory structure:  \r\n\r\n```shell\r\n/webapps/hello_django/hello\r\n├── hello\r\n│   ├── celery.py       # The Celery app file\r\n│   ├── __init__.py     # The project module file we modified\r\n│   ├── settings.py     # Settings go here, obviously :)\r\n│   ├── urls.py\r\n│   └── wsgi.py\r\n├── manage.py\r\n└── testapp\r\n    ├── __init__.py\r\n    ├── models.py\r\n    ├── tasks.py        # File containing tasks for this app\r\n    ├── tests.py\r\n    └── views.py\r\n```\r\n\r\nYou can find a[ complete sample Django project](https://github.com/celery/celery/tree/master/examples/django) on Celery’s GitHub.  \r\n\r\n## Testing the setup\r\n\r\nIn production we will want Celery workers to be daemonized, but let’s just quickly start the workers to check that everything is configured correctly. Use the `celery` command located in your virtualenv’s `bin` directory to start the workers. Make sure that the module path `hello.celery:app` is available on your `PYTHONPATH`.  \r\n\r\n>It’s important to understand how Celery names tasks which it discovers and how these names are related to Python module paths. If you run into `NotRegistered` or `ImportError` exceptions make sure that your apps and tasks are imported in a consistent manner and your `PYTHONPATH` is set correctly.  \r\n\r\n```shell\r\n$ export PYTHONPATH=/webapps/hello_django/hello:$PYTHONPATH\r\n$ /webapps/hello_django/bin/celery --app=hello.celery:app worker --loglevel=INFO\r\n\r\n -------------- celery@django v3.1.11 (Cipater)\r\n---- **** -----\r\n--- * ***  * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.5\r\n-- * - **** ---\r\n- ** ---------- [config]\r\n- ** ---------- .> app:         hello_django:0x15ae410\r\n- ** ---------- .> transport:   redis://localhost:6379/0\r\n- ** ---------- .> results:     disabled\r\n- *** --- * --- .> concurrency: 2 (prefork)\r\n-- ******* ----\r\n--- ***** ----- [queues]\r\n -------------- .> celery           exchange=celery(direct) key=celery\r\n\r\n[tasks]\r\n  . testapp.tasks.test\r\n\r\n[2014-05-20 13:53:59,740: INFO/MainProcess] Connected to redis://localhost:6379/0\r\n[2014-05-20 13:53:59,748: INFO/MainProcess] mingle: searching for neighbors\r\n[2014-05-20 13:54:00,756: INFO/MainProcess] mingle: all alone\r\n[2014-05-20 13:54:00,769: WARNING/MainProcess] celery@django ready.\r\n```\r\n\r\nIf everything worked, you should see a splash screen similar to the above and the ` [tasks]` section should list tasks discovered in all the apps of your project.  \r\n\r\n```shell\r\n[tasks]\r\n  . testapp.tasks.test\r\n```\r\n\r\n## Submitting a task to the queue for execution\r\n\r\nIn another terminal, activate the virtualenv and start a task from your project’s shell.  \r\n\r\n```shell\r\n$ sudo su - hello\r\nhello@django:~$ source bin/activate\r\n(hello_django)hello@django:~$ cd hello/\r\n(hello_django)hello@django:~/hello$ python manage.py shell\r\nPython 2.7.3 (default, Mar 13 2014, 11:03:55)\r\n[GCC 4.7.2] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n(InteractiveConsole)\r\n>>> from testapp.tasks import test\r\n>>> test.delay('This is just a test!')\r\n<AsyncResult: 79e35cf7-0a3d-4786-b746-2d3dd45a5c16>\r\n```\r\n\r\nYou should see messages appear in the terminal where Celery workers are started:  \r\n\r\n```shell\r\n[2014-05-18 11:43:24,801: INFO/MainProcess] Received task: testapp.tasks.test[79e35cf7-0a3d-4786-b746-2d3dd45a5c16]\r\n[2014-05-18 11:43:24,804: INFO/MainProcess] Task testapp.tasks.test[79e35cf7-0a3d-4786-b746-2d3dd45a5c16] succeeded in 0.00183034200018s: u'The test task executed with argument \"This is just a test!\" '\r\n```\r\n\r\nYou can find more information about [calling Celery tasks](http://celery.readthedocs.org/en/latest/userguide/calling.html) in the docs.  \r\n\r\n## Running Celery workers as daemons\r\n\r\nIn production we can use [supervisord](http://supervisord.org/) to start Celery workers and make sure they are restarted in case of a system reboot or crash. Installation of Supervisor is simple:   \r\n\r\n```shell\r\n$ sudo aptitude install supervisor\r\n```\r\n\r\nWhen Supervisor is installed you can give it programs to start and watch by creating configuration files in the `/etc/supervisor/conf.d` directory. For our `hello-celery` worker we’ll create a file named `/etc/supervisor/conf.d/hello-celery.conf` with this content:  \r\n\r\n```shell\r\n/etc/supervisor/conf.d/hello-celery.conf\r\n[program:hello-celery]\r\ncommand=/webapps/hello_django/bin/celery --app=hello.celery:app worker --loglevel=INFO\r\ndirectory=/webapps/hello_django/hello\r\nuser=hello\r\nnumprocs=1\r\nstdout_logfile=/webapps/hello_django/logs/celery-worker.log\r\nstderr_logfile=/webapps/hello_django/logs/celery-worker.log\r\nautostart=true\r\nautorestart=true\r\nstartsecs=10\r\n\r\n; Need to wait for currently executing tasks to finish at shutdown.\r\n; Increase this if you have very long running tasks.\r\nstopwaitsecs = 600\r\n\r\n; When resorting to send SIGKILL to the program to terminate it\r\n; send SIGKILL to its whole process group instead,\r\n; taking care of its children as well.\r\nkillasgroup=true\r\n\r\n; if rabbitmq is supervised, set its priority higher\r\n; so it starts first\r\npriority=998\r\n```\r\n\r\nThis configuration is based on[ a sample config](https://github.com/celery/celery/blob/3.1/extra/supervisord/celeryd.conf) provided by the makers of Celery. You can set [many other options](http://supervisord.org/configuration.html#program-x-section-settings).  \r\n\r\nCreate a file to store your application’s log messages:   \r\n\r\n```shell\r\nhello@django:~$ mkdir -p /webapps/hello_django/logs/\r\nhello@django:~$ touch /webapps/hello_django/logs/celery-worker.log\r\n```\r\n\r\nAfter you save the configuration file for your program you can ask supervisor to reread configuration files and update (which will start your the newly registered app).  \r\n\r\n```shell\r\n$ sudo supervisorctl reread\r\nhello-celery: available\r\n$ sudo supervisorctl update\r\nhello-celery: added process group\r\n```\r\n\r\nYou can now monitor output of Celery workers by following the `celery-worker.log` file:  \r\n\r\n```shell\r\n$ tail -f /webapps/hello_django/logs/celery-worker.log\r\n```\r\n\r\nYou can also check the status of Celery or start, stop or restart it using supervisor.  \r\n\r\n```shell\r\n$ sudo supervisorctl status hello                       \r\nhello                            RUNNING    pid 18020, uptime 0:00:50\r\n$ sudo supervisorctl stop hello  \r\nhello: stopped\r\n$ sudo supervisorctl start hello                        \r\nhello: started\r\n$ sudo supervisorctl restart hello \r\nhello: stopped\r\nhello: started\r\n```\r\n\r\nCelery workers should now be automatically started after a system reboot and automatically restarted if they ever crashed for some reason.  \r\n\r\n## Inspecting worker tasks\r\n\r\nYou can check that Celery is running by issuing the celery status command:  \r\n\r\n```shell\r\n$ export PYTHONPATH=/webapps/hello_django/hello:$PYTHONPATH\r\n$ /webapps/hello_django/bin/celery --app=hello.celery:app status\r\ncelery@django: OK\r\n\r\n1 node online.\r\n```\r\n\r\nYou can also inspect the queue using a friendly curses monitor:  \r\n\r\n```shell\r\n$ export PYTHONPATH=/webapps/hello_django/hello:$PYTHONPATH\r\n$ /webapps/hello_django/bin/celery --app=hello.celery:app control enable_events\r\n$ /webapps/hello_django/bin/celery --app=hello.celery:app events\r\n```\r\n\r\n![celery-curses-monitor](https://cloud.githubusercontent.com/assets/10941075/12741647/5edfdff4-c9b9-11e5-9db5-bc1a1bd1bc16.png)\r\n\r\n\r\n*Celery Worker monitor*  \r\n\r\nI hope that’s enough to get you started. You should probably read the [Celery User Guide ](http://docs.celeryproject.org/en/latest/userguide/)now. Happy coding!  \r\n"}}