Start and advance tasks with distributed tasks

A list,

Celery is a simple, flexible and reliable distributed task queue developed by Python, which is essentially a producer-consumer model where producers send tasks to message queues and consumers are responsible for processing tasks. Celery focuses on real-time operation but is also good for scheduling support and can handle millions of tasks per day. Features:

  • Simple: easy to configure and use after being familiar with the working flow of celery

  • High availability: When task execution fails or a connection break occurs during execution, celery will automatically try to resume task

  • Quick: a single celery progression can handle up to a million tasks per minute

  • Flexibility: almost every component of the celery can be expanded and customised

Application scenarios:

  • 1. Web application: when the user takes a long time to complete an operation on the site, we can give tasks to Celery execution and return them directly to the user. When Celery execution is complete, we can notify the user, greatly improve the concurrency of the site and the user experience.

  • Task scenario: for example, in operation and maintenance scenario, some commands or tasks need to be executed in batches on hundreds of machines, then Celery can be easily done.

  • Regular tasks: similar scenarios to regular tasks according to reports and regular send notifications, although Linux scheduled tasks can help me to implement but not very conducive to management, while Celery can provide a management interface and rich API.

Second, structure & working principle

Celery consists of the following three parts: Broker, task execution unit Worker and result store as shown in the figure below:

Working principle:

  1. Task Module Task Includes asynchronous tasks and scheduled tasks. Wherein, asynchronous tasks are usually triggered in the business logic and sent to the message queue, while timed tasks are periodically sent to the message queue by the Celery Beat process.

  2. The task execution unit Worker monitors the message queue in real time to obtain the task execution in the queue;

  3. After executing tasks, Woker stores the results in Backend.

Message middleware Broker

RabbitMQ, Redis, Amazon SQS, MongoDB, Memcached, and so on are recommended.

Task execution unit Worker

Worker is a task execution unit that is responsible for taking tasks out of the message queue for execution. It can be started in one or more or in different machine nodes, which is the core of its distributed implementation.

Result Storage Backend

Backend supports RabbitMQ, Redis, Memcached,SQLAlchemy, Django ORM, Apache Cassandra, and Elasticsearch.