This article belongs to Flask Web Development Field series. This article describes how to integrate Celery in a Flask project.

Create Celery program

The first step is to create an instance of the Celery program. Since the Flask instance will normally be named app, we will normally name tasks like Celery or celery_app to avoid collisions:

from celery import Celery

celery = Celery(__name__, broker='pyamqp://guest@localhost//')

@celery.task
def add(x, y):
    return x + yCopy the code

Organize and load the configuration

Most tutorials, including the current Flask documentation, describe how to load a configuration as follows:

Celery. Conf. Update (app.config) #Copy the code

That is, write the Celery configuration and the Flask configuration together and then update the configuration from the configuration dictionary of the Flask application instance.

However, the problem is that Celery starts with a new lowercase profile name and some profiles are replaced with the new name. The old upper-case configuration is still supported, but if you plan to use lower-case configuration names, or if you plan to migrate them in the future, this configuration loading method won’t work because Flask only imports upper-case configuration variables when importing configurations from files or objects.

Therefore I recommend putting the Celery configuration in a separate file and not mixing it with the Flask configuration. You can create a celeryconfig.py file (or another name) in the current directory to save the configuration as shown in the Celery file example:

broker_url = 'pyamqp://'
result_backend = 'rpc://'

task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'Europe/Oslo'
enable_utc = TrueCopy the code

Then load the configuration using the following method (use a different module name, or in a different path, remember to modify the string passed in) :

celery.config_from_object('celeryconfig')Copy the code

If you need to pass in the broker and backend configuration when creating Celery instances, you can either write it out directly or import it from the configuration module:

from celeryconfig import broker_url

celery = Celery(__name__, broker=broker_url)Copy the code

Initialize Celery in the Flask procedure

You can create tasks separately but we will usually need to add Flask context support to them as sometimes your tasks will use variables that depend on the Flask context.

Let’s create tasks with tasks in the factory function, load tasks in the factory function and implement Flask application context support:

from flask import Flask
from celery import Celery

from celeryconfig import broker_url


def make_celery(app):
    celery = Celery(__name__, broker=broker_url)
    celery.config_from_object('celeryconfig')

    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask
    return celery

app = Flask(__name__)

celery = make_celery(app)Copy the code

In the module that defines Celery tasks, e.g. tasks.py, you can import the Celery instance:

from app import celery

@celery.task
def add(x, y):
    return x + yCopy the code

Initialize Celery in the Flask program using the factory function

When Flask is also created using factory functions, we can create tasks instance globally and then update the tasks configuration and set the context in the factory function that creates the Flask instance:

from flask import Flask
from celery import Celery

from celeryconfig import broker_url
 
celery = Celery(__name__, broker=broker_url)
 

def create_app():
    app = Flask(__name__)
 
    register_celery(app)
    return app
 

def register_celery(app):
    celery.config_from_object('celeryconfig')
 
    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)
 
    celery.Task = ContextTaskCopy the code

Also import tasks directly and create tasks:

from app import celery

@celery.task
def add(x, y):
    return x + yCopy the code

This was originally a complete starter of Celery, but due to a hard disk damage last year, the corresponding sample program was lost and I didn’t have the willpower to rewrite it, so this article only extracted the contents related to Flask.

Since it has been more than half a year since the writing of the first draft, the latest version of Celery is already 4.3.0. If there are any omissions in the article or there is a better way to implement them, please comment.