celery

The use of celery and its configuration in Django are not covered in detail, the pits used in Django are mainly recorded.

Pit point

Time zone problem

Default time zone of celery is world Standard time, 8 hours slower than e8, if you post timed tasks, be sure to keep an eye on timed tasks otherwise you might use the correct method but not call successfully

Setup celery time zone you can add a setting to settings.py in your Django project

CELERY_TIMEZONE = 'Asia/Shanghai'
Copy the code

Django-celery can identify the time zone in the setup

You can also specify the current timezone when Posting a scheduled task, using Djangos get_current_timezone()

Convert the required time to the current time zone
from django.utils.timezone import get_current_timezone
import datetime

send_time = datetime.datetime.now() + datetime.timedelta(days=1)
tz = get_current_timezone()
send_time = tz.localize(send_time)
Copy the code

Pass the converted time to the parameter when using an asynchronous task

celery_task.apply_async(args=[], kwdg={}, eta=send_time)
Copy the code

Of course, you can also execute asynchronous tasks with intervals, corresponding to the countdown parameter in apply_async()

celery_task.apply_async(countdown=seconds)
Copy the code

Celery serialization problem

Celery provides two serialization formats, pickle and json. Pickle is a Python serialization library that allows you to sequence and deserialize data in various formats, corresponding to pickle and unpickle

The data format that celery accepts can be specified in the setup as well as the serializers for tasks and results

# settings.py
# celery allows data to be received in a string such as 'json'
CELERY_ACCEPT_CONTENT = ['pickle'.'json']
Serializers for asynchronous tasks can also be JSON
CELERY_TASK_SERIALIZER = 'pickle'
The data format of the task result can also be JSON
CELERY_RESULT_SERIALIZER = 'pickle'
Copy the code

This is especially important with Django. If you need to pass a Queryset to an asynchronous task, set the received format and serializer to ‘pickle’, as shown above

Not recommended to pass ORM object to celery asynchronous task, get expired data, recommend to pass ID

The results of

If the results of asynchronous tasks do not need to be processed i.e. the results of asynchronous tasks have little relation to the business logic, it is recommended not to store the results of asynchronous tasks.

If retained as a result, celery will build a queue for task results, and wait until the asynchronous task given the results will be removed from the queue to task, expensive to create and manage the task, you can see in this blog: www.cnblogs.com/blaketairan…

Set Django Settings to ignore celery task execution results

CELERY_IGNORE_RESULT = True
Copy the code

Use different queues

If task A is more important than task B and the amount of task B is very large, important task A needs to wait for task B to finish before continuing. In this case, different queues can be used to save tasks and different workers can perform the two tasks

CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('for_task_A', Exchange('for_task_A'), routing_key='for_task_A'),
    Queue('for_task_B', Exchange('for_task_B'), routing_key='for_task_B'),Copy the code

You can then customize the router to perform different tasks


CELERY_ROUTES = {
    'my_taskA': {'queue': 'for_task_A'.'routing_key': 'for_task_A'},
    'my_taskB': {'queue': 'for_task_B'.'routing_key': 'for_task_B'}},Copy the code

Then, when starting celery, specify different workers

celery worker -E -l INFO -n workerA -Q for_task_A celery worker -E -l INFO -n workerB -Q for_task_B
Copy the code