According to the docs, Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.
It’s a task queue with focus on real-time processing, while also supporting task scheduling.
What is it used for?
It is mainly used for the following things:
- Running something in the background
- Asynchronous execution of code
- Scheduling periodic work
Use case example
Your web app needs to send an email. That is a very slow operation. While users can put up with 4 or 5 seconds until an email is sent, it might leave a bad impression on them. Solution? Celery (singing... "I came in like a wrecking ball...(by Miley Cyrus)")! It will take this operation out of the main thread and executes it in the background. This gives the user the impression of good performance and “snappiness”, even though the real work might actually take some time.
Read Queue everything and delight everyone for additional info on why task queues can be useful.
Now that we know what Celery is and what it is used for, let's jump in and see how to use it with Django (other cases should be similar). We will see how to send an email with Celery.
First, create a new
proj/proj/celery.py module that defines a Celery instance
from __future__ import absolute_import import os from celery import Celery # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') from django.conf import settings # noqa app = Celery('proj') # Using a string here means the worker will not have to # pickle the object when using Windows. app.config_from_object('django.conf:settings') app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
Then, import this app in
from __future__ import absolute_import # This will make sure the app is always imported when # Django starts so that shared_task will use this app. from .celery import app as celery_app # noqa
Then, create an ordinary function
def send_email(): message = EmailMessage('Subject', 'Message', to=['firstname.lastname@example.org']) message.send() print('Email is sent')
Note that tasks are normally placed in
tasks.py file inside django apps:
- app1/ - app1/tasks.py - app1/models.py - app2/ - app2/tasks.py - app2/models.py
However, for this simple example, I put the
send_email() function inside
Now, let's check our function by executing it in the shell
>>> from proj.celery import send_email >>> send_email()
After a couple of seconds, you will see
Email is sent message as long as you properly configured email settings. But these seconds are too long when you can easily get rid of them. To do that, we now need to transform this function into a celery task by simply using
@shared_task() def send_email(): message = EmailMessage('Subject', 'Message', to=['email@example.com']) message.send() print('Email is sent')
Now, we start celery in the command line by executing this:
celery -A proj worker -l info
Now, we call our task from the shell with
delay() method of celery:
>>> from proj.celery import send_email >>> send_email.delay()
You will immediately see that the method returned! That means that users will see the results right away! If you check you Celery logs, you will see something like this:
[2016-04-28 06:54:59,920: INFO/MainProcess] Received task: proj.celery.send_email[1d2b9446-4791-4da4-8136-ee74d78cf394] [2016-04-28 06:55:02,470: WARNING/Worker-3] Email is sent [2016-04-28 06:55:02,471: INFO/MainProcess] Task proj.celery.send_email[1d2b9446-4791-4da4-8136-ee74d78cf394] succeeded in 2.550240921s: None
Awesome! Pretty fast!
This was a simple example of how to use Celery. Please note that this post does not discuss the installation process of Celery (or RabbitMQ) and is only intended to serve as a fast introduction to the tool.
In the next post, I will discuss how Celery can be used for periodic tasks (think cron jobs)