I. Background introduction

In the early stage of system development, it is easy to encounter such a situation: different business line developers, due to the impact of technology stack and version time, will give priority to the selection of familiar, such as MQ middleware commonly used: Kafka, Rocket, Rabbit, etc., so it is easy to ignore the component differences between each project;

In middle and later periods of the system development, the business is relatively stable, usually take up higher module for resources gradually refactoring, integrate the management of public service, so as to make the system more integrity, in the process, solve different middleware differences often bear the brunt of the project, such as common cache center, MQ message management, etc.;

This situation is generally difficult to avoid. In the early stage of the system, in order to quickly support the business, many holes are buried. Once the business can develop stably and the sustainability is verified, module reconstruction will be properly considered to reduce the cost.

Second, the idea of reconstruction

2.1 Initial Problems

In the early stage of the development of a startup company, there were five projects in parallel development on the business line. The usage of MQ at that time was as follows:

  • Rocket: 3 core business projects with different versions;
  • Kafka: The data weight is too high, 1 item is used;
  • Redis: Python based connection, queue message mode;

At the beginning, the whole was under control because it was not used much. Later, as the business continued to iterate, the communication between projects began to be chaotic and difficult to maintain. Then, it was forced to reconstruct and unify the message components.

2.2 Secondary selection

Based on the comprehensive consideration of the business, SEVERAL existing projects were redesigned MQ, forming the overall architecture idea as follows:

  • MQ component selection: RocketMQ;
  • Replace the queue mode of Redis components.
  • Converting python-based systems to Java;
  • Provide both message production and consumption services;
  • The functionality of MQ is uniformly maintained by the above services;

There is no change in component selection for the core line of business. One of the reasons for switching kafka is that it involves a lot of settlement, the Redis queue mode is deprecated, and the Python-based management system is not very functional. After such design, look from integral train of thought can be reasonable a lot of.

3. Transformation process

3.1 Overall Thinking

Description of core roles involved, from left to right:

  • Production client: the node that needs to request the communication of the server can call the message sending interface encapsulated by the production server.
  • Production server: encapsulates message sending API, maintains route management, permission recognition, message storage, etc.
  • Message storage layer: mainly based on message middleware for storage, database layer is used to deal with the secondary scheduling under specific circumstances;
  • Consuming server: encapsulates message receiving API, and requests the specified consuming interface according to routing identification to complete communication;
  • Consumption client: responds to the request of consumption server and encapsulates the specific business logic of consumption;

There is not much difference in overall technical difficulty, but the introduction of two services [production and consumption], used to manage MQ communication flow, adapt to specific business logic, the introduction of a database to do a landing storage, more flexible in the handling of abnormal flow, so that the whole message module has strong scalability.

3.2 Details

  • Component selection

There were a number of messaging middleware options, but RocketMQ components were selected in consideration of the familiarity of the on-line developers and the test comparison reports provided by various parties. RocketMQ’s characteristics of high performance, high reliability, and support for distributed transactions were also core considerations.

  • Microservices Architecture

Micro service based on the current mode of architecture, the MQ integration in two core service function itself, the unified management and iteration, and version control components, for all the news of the production, control the global routing, as well as the specific circumstances, through application service level, function design, implementing message delay spending, as well as the failure message scheduling execution, And partial manual triggering of single messages.

  • Data is stored

To secondary storage of news entities, mainly adapted some specific function point, some messages can delay processing, such as when the MQ queue appeared accumulation, or to monitor line, can be configured, intervention only part of the message storage warehouse, don’t push the MQ, when waiting for the service are relatively free to send again.

As a stable support for decoupling between systems, message-oriented middleware needs to have a clear design route and monitor and record the key nodes of the process to ensure the stability and fault tolerance of the whole link when managing at the service level.

Source code address

GitEE address https://gitee.com/cicadasmile Wiki, https://gitee.com/cicadasmile/butte-java-note/wikisCopy the code