background

In development, we often encounter time-consuming tasks, for example:

Upload and parse an Excel file of 1W data, and persist it to the database.

In my application, this task took about 6s, and waiting for 6s was a disaster for users.

A better way to deal with it is:

  1. Receive the request for this task
  2. Add the task to the queue
  3. The message “Operation succeeded, processing in background” is displayed immediately
  4. The background consumes the queue, performs the task

We follow this idea and implement with Celery.

implementation

The environment used in this article is as follows:

  • Python 3.6.7
  • The RabbitMQ 3.8
  • Celery 4.3

Install RabbitMQ using Docker

Celery relies on a message back, options include RabbitMQ, Redis, etc. This article uses RabbitMQ.

For easy installation, I will install RabbitMQ directly using Docker:

docker run -d --name anno-rabbit -p 5672:5672 rabbitmq:3
Copy the code

Once successfully started, the message queue can be accessed through amqp://localhost.

Install and configure Celery

Celery is a Python implemented tool and setup can be done directly via Pip:

pip install celery
Copy the code

And let’s also say that my current project folder is proj, my project name is MyProj, and my application name is MyApp

When setup is complete, create a celery. Py file under proj/myproj/ to instantiate celery instances:

proj/myproj/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery, platforms

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE'.'myproj.settings')

app = Celery('myproj',
             broker='amqp://localhost//',
             backend='amqp://localhost//')

# Using a string here means the worker don't have to serialize
# the configuration object to child processes.s
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
Copy the code

Add a reference to Celery in proj/myproj/__init__. Py to make sure Django starts with Celery:

proj/myproj/__init__.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app'.)Copy the code

Without other special configurations, this is the basic configuration of Celery.

Write a time-consuming task

To simulate a time-consuming task we directly create a method that ‘sleeps’ 10s and set it to Celery task:

proj/myapp/tasks.py

import time
from myproj.celery import app as celery_app

@celery_app.task
def waste_time(a):
    time.sleep(10)
    return "Run function 'waste_time' finished."
Copy the code

Start the Celery Worker

Celery configuration complete and after task creation we start Celery in asynchronous task mode:

celery -A myproj worker -l info
Copy the code

Note that I highlighted asynchronous mode because Celery supports asynchronous tasks as well as timed tasks, so specify when starting.

Also note that once tasks are started, the modifications to Task(here waste_time) must restart Celery to take effect.

Task calls

In the logical code for request processing, invoke the task created above:

proj/myapp/views.py

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from .tasks import waste_time

@require_http_methods(["POST"])
def upload_files(request):
    waste_time.delay()
    # Status code 202: Accepted, indicating that the asynchronous task has been Accepted and may still be processing
    return JsonResponse({"results": "Operation succeeded, uploading, please wait..."}, status=202)
Copy the code

Waste_time.delay () after calling waste_time.delay(), waste_time will be added to the task queue waiting for the idle Celery Worker to call.

The effect

When we send a request, the interface returns {” Results “: “Operation succeeded, uploading, please wait…” } instead of being stuck for ten seconds, the user experience is much better.

conclusion

Tasks like Celery are a common Python method, although the actual elapsed time is constant or even increased (e.g. busy Worker causes delays), it is more acceptable for the user experience to continue processing other tasks after clicking on upload large files without waiting on the page.

There are more uses of Celery not covered in this article, the documentation is very detailed and can be directly referenced if necessary.

reference

  • Docs.celeryproject.org/en/latest/d…
  • hub.docker.com/_/rabbitmq