I. Brief introduction to builtcelery distributed system

Celery profile

Celery is a simple, flexible and reliable distributed system that processes large numbers of messages, focuses on asynchronous task queues for real-time processing and also supports task scheduling. Celery a worker is a daemon.

2, components,

There are three components in the Celery frame: Celery+RabbitMQ+Redis, there are three components in the frame: Celery+RabbitMQ+Redis And explain in detail how the task flow works.

3. The framework in this case

Celery broker = RabbitMQ broker is a message-intermediate that caches tasks in queues and blocks rows. Backend = Redis backend is a result store which mainly stores data after tasks have been completed and also stores task execution results. If no results need to be stored, no setting is required.

Celery simple configuration file

The RabbitMQ account password is root and dispatch_server_host is a user-defined RabbitMQ virtual host or queue.
Redis select Redis 1 database db1
Rabbitmq dispatch_server_host is a custom rabbitMQ virtual host
BROKER_URL = 'it: / / root: [email protected] / dispatch_server_host'

# Redis address, select Redis library 1
CELERY_RESULT_BACKEND = "Redis: / / : [email protected]:6379/1"

Register the address of the task
CELERY_INCLUDE = ['logic.celery_core.task_register']
Copy the code

These are the two main functions of the test

5. Distribute tasks

The test case selected here is one that was written in the previous blog, ranking players by their chat history. For a specific task build, look at the previous blog build process. Here is a brief introduction to 20200406,20200407 two days 1-450 451-900 level of the chat records of players for distributed computing. There are four tasks (Task_1:20200406 1-450) to perform jieba word calculation on the chat records of players task_2:20200406 451-900) to perform jieba word calculation on the chat records of players…… Elongate ([worker_1, worker_2, worker_3, worker_4])

2. Start RabbitMQ, Redis

If there is a need to install the start command again private letter I again a detailed deployment tutorial, here is not said.

Connect to Redis and use the RedisDesktopManager tool

Third, submit the task flow

1. RabbitMQ receives the task

RPC queues pass parameters to receivers via RabbitMQ

Start Celery server

Start queue named celery_test to consume tasks in RabbitMQ, here start 4 worker celery -A main worker -n celery_test -c 4 -l info -q celery_test

3. Storage of Redis

Let’s take a look at Redis DB1

Four,

The Celery distributed frame is a relatively simple microserver frame, but if you want to optimize it in a targeted way you still need to figure out how the specific data in the whole frame goes. Celery+RabbitMQ+Redis+MySQL should be the more common configuration but these can be selected according to the business situation. For the management of the overall framework, I use flower+ Supervisor to view and manage queues.
The RabbitMQ RPC queue will be used for more details if you are interested.

If it is helpful to you, please give a thumbs-up to support it, it is very important! Thank you!

The whole framework is a personal understanding, if there is a big boss found where there is a problem please be sure to tell me, if there is a good optimization framework ideas can also be said to discuss together. Thank you!