Scheduling Tasks with Celery Beat

In the past I have shared how to start a Django project and how to utilize Celery to run asynchronous background tasks.

However, neither of those presentations answer the question: "how I setup scheduled asynchronous jobs in a Django site?"

I aim to answer that here.

Just Beat It

Before you dive into this short tutorial, you might checkout the videos I linked above. You are going to need a running Django site, with Celery setup.

Once you have those things, you'll need to setup the Beat scheduler.

In my celery.py file in my project directory, I added some additional content, so this is my entire celery.py file:

import os
from celery import Celery
from celery.schedules import crontab
from django.conf import settings
from celery import shared_task

# APP
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'example.settings')

app = Celery('example')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Schedule
app.conf.beat_schedule = {
    "Calc minutely": {
        "task": "example.celery.my_task",
        "args": (3, 40),
        "schedule": crontab(),
    },
}

# Tasks
@shared_task
def my_task(arg1, arg2):
    result = arg1 + arg2
    return result

If you watched my video from before, you'll see that the only difference here is the newly added app.conf.beat_schedule variable being set.

Note, if you are doing this from scratch, ensure to edit your projects directory's __init__.py file, and add the following content (this file for me is at example/__init__.py):

from .celery import app as celery_app

__all__ = ["celery_app"]

Start the Beater

Now it's time to start the beat scheduler. You can run the Celery worker and scheduler as two separate processes, but I want to make this tutorial as easy as possible. So I am just going to run the beat scheduler in the same process as my celery worker.

Here is the command I was using before:

celery -A example worker -l info

And here is the command I am using now, using the -B flag to invoke the scheduler as well:

celery -A example worker -l info -B

Conveniently, I have the scheduler info setup in the same celery.py file, so it's all wrapped up into a neat package.

If you are using Docker (or Docker Compose), then you will want to add another container that specifically runs the celery worker command. And if you want to separate the scheduler from the worker, then you can do that as well.

Making it Complicated

I won't delve any further in at this point into all the settings. For many projects, this may suffice.

For further reading, check out the docs for the django-celery-beat package.