On how to build a high concurrency system some experience summary, only for reference, welcome to exchange.

• preface
• infrastructure
• The database
• architecture
• application
• specification
• conclusion

preface

Leave hungry for some time, starting in 2017 during the hungry? Take waybill system development and maintenance work, from the earliest per million, to leave the average daily must single, the characteristics of the rapid development of business plus delivery business is business focused on the noon peak and late peak two peak, the magnitude of the peak concurrent requests is a rising tide lifts all boats, High concurrency is a daily challenge. Take the waybill system for example, the QPS of daily midday peak core query service is more than 200,000, the QPS of Redis cluster is more than one million, the QPS of database is more than 100,000, and the TPS is more than 20,000.

In such a large flow, the main work is also around how to build system stability and improve capacity, the following mainly from the infrastructure, database, architecture, application, specification of these aspects to talk about how to build a high concurrency system. The following are some of the lessons I’ve learned over the years. Architecture doesn’t have a silver bullet, so it’s not a best practice.

infrastructure

In a layered architecture, the lowest level is infrastructure. Basic Settings generally include physical servers, IDCs, deployment modes, and so on. Just like a pyramid, the infrastructure is the base of the pyramid, and only when the base is stable can the top be stable.

Different live

In 2017, Ele. me launched the live in different locations project. The whole project was led by the infrastructure department of the company, and the business side needed to make corresponding transformation with the live Plan.

Live more can be divided into more live in the same city, live more in different places and so on, there are many ways to achieve, such as ali using the unitary scheme, Ele. me using the multi-center scheme, on ele. me live more can refer to ele. me live more live technology. At that time, the main purpose of multi-activity is to ensure the high availability of the system and avoid the single point of failure of a single IDC. At the same time, the traffic in each equipment room is 1/N of the total traffic, which also increases the system capacity and can resist more traffic in high concurrency scenarios. Below is the overall architecture of The Hungry Me More implementation, from the shared article above.

The database

Database is one of the most important parts of the whole system. Under the scenario of high concurrency, a large part of the work is carried out around database. The main problem to be solved is how to improve the database capacity.

Reading and writing separation

Most Services on the Internet are characterized by more reads and fewer writes. Therefore, the read/write separation architecture can effectively reduce the database load and improve system capacity and stability. The core idea is that the master library bears the write traffic and the slave library bears the read traffic. In the read/write separation architecture, there is usually a configuration of one master and multiple slave libraries, and multiple slave libraries share the high concurrent query traffic. For example, there are now 10,000 QPS and 1K TPS, assuming that in the configuration of 1 master and 5 slaves, the master library only bears 1K TPS, and each slave library bears 2K QPS. This level of magnitude is completely acceptable for DB, and the pressure on DB is much smaller than before the read-write separation transformation.

The advantage of this pattern is that it is simple, with little or no code modification cost and only requires the configuration of the database master and slave. The disadvantages are equally obvious:

Master-slave delay

MySQL’s default primary/secondary replication is asynchronous. If you query data from the secondary database immediately after the primary database is inserted, you may not find the data. In normal cases, the master/slave replication has a millisecond delay, and may have a second or longer delay when the DB load is high. However, even the millisecond delay cannot be ignored for services with high real-time requirements. Therefore, in some critical query scenarios, we will bind query requests to the master library to avoid the master-slave delay problem. There are also a lot of articles on the web about the optimization of master-slave delay, which will not be repeated here.

The number of slave libraries is limited

There is a limit to the number of slave libraries that can be mounted by a master library, and there is no way to achieve infinite horizontal scaling. The more slave libraries, the higher QPS it can withstand theoretically, but too many slave libraries will lead to the greater pressure of master slave replication IO of the master library, resulting in higher latency, thus affecting business. Therefore, only a limited number of slave libraries will be mounted behind the master library in general.

Can’t solve the problem of high TPS

Although the slave library can solve the problem of high QPS, it cannot solve the problem of high TPS. All write requests can only be processed by the master library. Once the TPS is too high, DB still has the risk of downtime.

Depots table

When read/write separation does not meet the business needs, we need to consider using the separate database and table pattern. When deciding to optimize the database, we should give priority to the mode of read/write separation. Only when the mode of read/write separation can no longer withstand the traffic, we consider the mode of database and table separation. The net effect of the separate database and table pattern is to convert a single database and table into multiple databases and tables, as shown below.

First of all, the sub-table can be divided into vertical split and horizontal split. Vertical split is split according to the business dimension. Suppose there is an original order table with 100 fields, it can be split into multiple tables according to different business latitude, such as a table of user information, a table of payment information, etc., so that the fields of each table are relatively not too many.

Horizontal split is to split a table into N tables, for example, split an order table into 512 order sub-tables.

In practice, it is possible to do only horizontal split or vertical split, or both horizontal and vertical split.

Having said the sub-table, what is the sub-library? Database partition is to split the tables originally in one DB instance into N DB instances according to certain rules. Each DB instance has a master, which is similar to the architecture of multiple mater. At the same time, to ensure high availability, each master must have at least one slave. To ensure that when the master goes down, the slave can take over in time, and also to ensure that data is not lost. After splitting, there are only partial tables in each DB instance.

Because of the multi-master architecture, the sub-table contains all the advantages of the read-write separation mode, but also can solve the problem of high TPS that cannot be solved in the read-write separation architecture. At the same time, the sub-table can be extended horizontally indefinitely in theory, and also solves the problem of the limited number of slave libraries in the read-write separation architecture. Of course, in actual engineering practice, it is generally necessary to estimate the capacity in advance, because the database is stateful. If the capacity is found to be insufficient, it is very troublesome to expand, and should be avoided as far as possible.

In the mode of database separation and table separation, you can avoid the problem of master/slave delay by not enabling the query of the slave database. That is, all the read and write data are in the master database. After the database separation, the traffic on each master database only accounts for 1/N of the total traffic. Perform a master/slave switchover to replace the master when the master library is down. Said the benefits, and then said that the sub-database sub-table will bring problems, mainly the following points:

High renovation cost

The database and table are generally supported by middleware, and there are two common modes: client mode and proxy mode. The client mode will realize the logic of library and table by referring to the client package on the service, which is represented by the open source Sharding JDBC. Proxy mode means that all services do not directly connect to MySQL, but connect to proxy and then connect to MySQL. The proxy needs to implement the protocol related to MySQL.

The two modes have their own advantages and disadvantages. The proxy mode is relatively more complex, but because there is an additional layer of agents, more things can be done in the proxy layer, which is convenient for upgrading, and the database connection number can be stable through the proxy. The advantage of using client mode is that it is relatively simple to implement, no intermediate proxy, and theoretically better performance, but the upgrade is more difficult than the agent mode because the business needs to modify the code during the upgrade. The proxy mode is used in Ele. me. There is a unified database access middleware — DAL, which is responsible for proxy of all databases and realizes the logic of sub-database and sub-table, so as to keep transparent to business.

Transaction issues

In business, we use transactions to handle multiple database operations. The correctness of business processes is guaranteed by the four characteristics of transactions: consistency, atomicity, persistence, and isolation. After the database is split into tables, a table is split into N subtables, which may be in different DB instances, so that although it logically looks like a table, it is no longer in a DB instance, causing the problem of not being able to use transactions.

Is one of the most common in batch operations, we can at the same time in front of the depots table to the operation of the multiple orders placed in a transaction, but after the depots table will not be able to do that, because of different orders can belong to different users, suppose we to depots table as the user, so order table of different users in different DB instance, Multiple DB instances obviously cannot be handled by a single transaction, so some other means are needed to solve this problem. Such cross-DB instance operation should be avoided as far as possible after the sub-database sub-table. If it must be used in this way, compensation and other methods should be given priority to ensure the final consistency of data. If consistency must be improved, the commonly used scheme is distributed transaction.

Multidimensional queries are not supported

Generally, the sub-database sub-table can only be divided according to 1-2 latitude, which is the so-called Sharding key. Commonly used dimensions such as users and merchants dimension, if according to user’s dimension table, the most simple way to accomplish this is according to the user ID to the modulus to positioning in which depots which table, this means that all read and write requests must be take after user ID, but inevitably exists in real business multiple dimensions of the query, Not all queries will have user IDS, which requires us to modify the system.

To support after the depots table multidimensional query, there are two kinds of common solution, the first is to introduce a index table, this index table is no depots of the table, or according to the user ID depots table as an example, the index table records on the mapping relationship between various dimensions and user ID, request need through other dimension query index table to get the user ID, Then query the table behind the sub-database sub-table by user ID. This, on the one hand, requires more IO, and on the other hand, the index table is easy to become a system bottleneck because there is no sub-database sub-table.

The second solution is to introduce NoSQL. The most common combination is ES+MySQL, or HBase+MySQL. In essence, this solution uses NoSQL to play the role of the index table in the first solution. NoSQL has better horizontal scalability and scalability, and is less likely to become a bottleneck if it is properly designed.

Data migration

Generally, data migration is required for database and table subdivision. The original single table data is migrated to the database table after database and table subdivision through data migration. There are two kinds of data migration scheme of common, the first is a migration downtime, as the name suggests, this method simple and crude, the advantage is that can one pace reachs the designated position, the migration period is short, and can guarantee the data consistency, is bad to hurt, some key business may not be able to accept a few minutes or longer downtime migration of business losses.

The other solution is double write, which is mainly for new incremental data, the stock data can be directly synchronized, about how to double write migration has been shared on the Internet, there is no need to repeat, the core idea is to write the old library and new library at the same time. Dual-write has a small impact on services, but it is more complex, takes a longer migration period, and is prone to data inconsistency. Therefore, a complete data consistency guarantee scheme is required.

summary

Read/write separation mode and database and table separation mode The read/write separation mode is recommended. The database and table separation mode is recommended only when service requirements are not met. Reason is that although depots table model can significantly improve the capacity of the database, but can increase the system complexity, and can only support a few dimensions, speaking, reading and writing, in a sense for the business system is a kind of limit, so when design depots table plan needs to combined with the specific business scenarios, a more comprehensive consideration.

architecture

Architecture is also very important in the construction of highly concurrent systems. Here are some lessons from the patterns of caching, message queuing, resource isolation, and so on.

The cache

Caching is one of the most effective tools in a highly concurrent architecture. The maximum effect of cache is to improve system performance, protect back-end storage from heavy traffic, and improve system scalability. The concept of cache originated from CPU. In order to improve the processing speed of CPU, L1, L2, and L3 cache are introduced to speed up access. The cache used in the current system is also based on the practice of CPU cache.

Caching is such a big topic that I could write a book on it. Here I summarize some of the problems and solutions I encountered when designing and implementing caching for waybill systems. Cache is mainly divided into local Cache and distributed Cache. Local Cache such as Guava Cache and EHCache, and distributed Cache such as Redis and Memcached are mainly used in the waybill system.

How to ensure data consistency between cache and database

The first is how to ensure the data consistency between the cache and the database, basically when using the cache will encounter this problem, but it is also a frequent interview question. This problem becomes more prominent when I use cache in the waybill system I am in charge of. Firstly, the waybill is frequently updated, and the waybill system has very high requirements on data consistency, so it is almost impossible to accept data inconsistency, so the cache cannot be invalidated simply by setting an expiration time.

For the mode of cache read and Write, it is recommended to read Uncle Mouse’s article “Cache Update Routines”, which summarizes several commonly used cache read and Write routines. My cache read and Write mode in the waybill system also refers to the Write Through mode in the article, which is roughly like this through pseudo-code:

Lock (waybill ID) {//... // deleteCache deleteCache(); // updateDB updateDB(); ReloadCache ()}Copy the code

Since it is Write Through mode, updates to the cache are made in the Write request. First of all, in order to prevent concurrency issues, write requests need to be distributed lock, lock granularity is waybill ID as the key, after the execution of the business logic, first remove the cache, to update the DB, and then rebuild the cache, these operations are synchronous, reading request query cache first, if the cache hit the return directly, if you don’t hit the query DB cache, In this way, the cache operations are converged in the write request, and the write request is locked, which effectively prevents the problem of writing dirty cache data caused by concurrent read and write.

Cache data structure design

Caching avoids the problem of large and hot keys. For example, if you use the hash data structure in Redis, you’re more likely to have large and hot keys than regular string keys, so if you don’t have to do something specific with the hash, consider breaking up the hash into separate key/value pairs. Common string keys are used to store hash keys. This prevents large keys caused by too many hash elements and avoids overheating of single hash keys.

Read and write performance

The write performance is mainly affected by the size of key/value data. JSON serialization can be used to store data in simple scenarios. However, in high-concurrency scenarios, JSON cannot meet performance requirements and occupies a large amount of storage space. The more common alternatives are Protobuf, Thrift, etc. There are also some performance comparisons for these serialization/deserialization schemes available online at code.google.com/p/thrift-pr…

The main factor affecting read performance is the size of packets read at a time. In practice, it is recommended to use redis pipeline+ batch operation, for example, if it is a string key, that is pipeline+ MGET mode, assuming that one Mget 10 keys, 100 MGET for a batch of pipeline, At that time, network IO can query 1000 cache keys. Of course, the number of a specific batch depends on the packet size of cache keys, and there is no unified value.

Appropriate redundancy

Proper redundancy means that we can do some redundancy when designing the external business query interface. The experience comes from when we design the waybill system external query interface, the pursuit of generality, the return value of the interface design into a large object, put all the fields on the waybill in the large object directly exposed outside, inside this has the advantage of not need different according to different development of query interface, anyway the fields in the interface, Take whatever you want.

Is fine to do so at first, but we need to query interface increase cache to find, because all business partner through this interface to query the waybill data, we can’t know their business scenarios, also don’t know them, what about the requirements of the interface data consistency, such as whether to accept a short data consistency, And we also don’t know their specific use what field in the interface, the interface will not change, some field in some fields will be change frequently, according to different update frequency actually can adopt different cache design plan, but it’s a pity, because when we design the interface too pursuit of universality, when doing the cache optimization is very trouble, The solution had to be designed for the worst case scenario, where all business parties had high requirements for data consistency, resulting in the final solution spending a lot of energy on data consistency.

If we start to design external query interface can do some appropriate redundancy and distinguish between different business scenarios, though this will cause some of the interface function is similar, but when adding in the cache can be targeted, to design different solution for different business scenarios, such as the key process to pay attention to the guarantee of data a, Non-critical scenarios allow temporary data inconsistencies to reduce the cost of caching implementations. Will be updated in the interface at the same time also can best fields and a distinction must be won’t update field, so in the design of the cache scheme, in view of the field is not updated, can set a long overdue, and will update the field, only set shorter expiration time, and need to be cache update scheme design to guarantee the data consistency.

The message queue

In the architecture of high concurrency system, message queue (MQ) is essential. When the heavy traffic comes, we increase the scalability of the system by the asynchronous processing of message queue and the characteristics of peak cutting and valley filling. In addition, the use of message queue can also achieve the purpose of full decoupling between systems.

The core model of message queue is composed of Producer, Consumer and message Broker. The industry commonly used open source solutions have ActiveMQ, RabbitMQ, Kafka, RocketMQ and more fire Pulsar in recent years, the contrast can reference articles about various message middleware: zhuanlan.zhihu.com/p/401416746…

After using the message queue, synchronous processing request, can be changed by MQ message asynchronous consumption, which reduces system to deal with the pressure, increase the system throughput, share about how to use the message queue, there are many articles, here is my experience when considering using the message queue to combine specific business scenarios to determine whether the introduction of the message queue, Because after using the message queue is actually increased the complexity of the system, the original through a synchronous request can fix things, need to introduce additional dependencies, and news consumption is asynchronous, asynchronous naturally is more complex than synchronization, also need to consider additional message out-of-order, delay, loss and other issues, how to solve these problems is a big topic, There is no such thing as a free lunch, and any architectural design is a process of making and losing choices. You need to carefully consider the pros and cons before making a decision.

Service governance

Service governance is a big topic that can be taken out in its own right, and I’ll put it under architecture here as well. Service governance is defined as

Generally, it is independent of the business logic to provide some system guarantee measures for the reliable operation of the system.

Common safeguard measures include service registration and discovery, observability (monitoring), flow limiting, timeout, circuit breaker, etc. In microservice architecture, service governance is generally accomplished through service governance framework, and open source solutions include Spring Cloud, Dubbo, etc.

In a high-concurrency system, service governance is a very important part. Compared with cache and database, service governance is more about details, such as whether the timeout setting for the interface is 1 second or 3 seconds, how to do monitoring, etc. There is a saying that details determine success or failure. I’ve seen a lot of failures happen just because an interface timeout is set improperly, especially in a high-concurrency system. Pay attention to these details.

timeout

The rule for timeouts is: Everything has timeouts. Whether it’s RPC calls, Redis operations, consume/send messages, DB operations, etc., there are timeouts. Before, ELE. me encountered dependence on external components, but did not set a reasonable timeout. When external dependency fails, all threads of the service are blocked, resulting in resource exhaustion and failure to respond to external requests, which is a “blood” lesson.

In addition to setting timeouts, it is also important to set reasonable timeouts, such as the one mentioned above. Even if timeouts are set, too long will still bring down the service due to external dependency failures. How to set a reasonable timeout is very specific. You can consider whether it is critical to business scenarios and whether it is strongly dependent. There is no general rule and it needs to be based on specific business scenarios. For example, in some C-side presentation interfaces, setting a timeout of 1 second may seem fine, but in some performance sensitive scenarios, 1 second may be too long. In short, it needs to be set for specific business scenarios, but the principle remains the same: everything has a timeout.

monitoring

Monitoring is the eyes of the system, without monitoring the system is like a black box, from the outside do not know the operation of the inside, we can not manage and operate the system. So, monitoring systems are very important. System observability mainly includes three parts: logging, tracing, and metrics. The hungry before yao is mainly use the research of monitoring system, have had to really is very good, detailed introduction can be reference: mp.weixin.qq.com/s/1Vc5hUX7X… In the construction of high concurrent systems, we must have the perfect monitoring system, including system level monitoring (CPU, memory, network, etc.), the application layer of the monitor (JVM, performance, etc.), business level monitoring (curve of various business, etc.), etc., in addition to monitoring have perfect alarm, because someone can’t 24 hours staring at the monitor, Once there is any risk must alarm out, timely intervention, prevent risks in the future.

fusing

Circuit breakers are typically built into microservice frameworks to protect their own services in the event of downstream service failures. Generally, a Crit Breaker is used to determine whether a fuse is triggered according to the success rate/number of interfaces and other rules. The Breaker will control the state of the fuse from closed to open to half open. The recovery of the circuit breaker will go through the mechanism of time window. The circuit breaker will go through the half-open state first. If the success rate reaches the threshold, the circuit breaker will be closed.

If there is no special need in a business system, there is usually no need to do anything about fuses. The framework will automatically turn fuses on and off. One thing you might want to be careful about is avoiding invalid circuit breakers. What is an invalid circuit breaker? I encountered a fault in the past. The service provider threw unreasonable exceptions (such as system exceptions) during some normal service verification. As a result, interface fusing affected normal services. Therefore, when you throw an exception or return an exception code on an interface, you must distinguish between service exceptions and system exceptions. Generally speaking, service exceptions do not need to be fused. If a system exception is thrown because of a service exception, it will be fused, and normal service processes will be affected.

demotion

Demotion is not a specific technology, but more like a methodology of architecture design, which is a strategy to lose the enemy and protect the enemy. The core idea is to limit some of their own capabilities in abnormal circumstances to ensure the availability of core functions. Degradation can be implemented in many ways, such as configuration, switching, limiting, and so on. Downgrades can be active or passive.

When the e-commerce system is greatly promoted, we will temporarily shut down some non-core functions to ensure the stability of core functions, or when the downstream service fails and cannot be recovered in a short time, we will degrade the downstream service to ensure the stability of our own service, which are all active degradation.

Passive degradation refers to that, for example, an interface downstream is called, but the interface times out. In this case, in order to keep the business process running, we usually choose to catch an exception in the code, print an error log, and then continue to execute the business logic. This degradation is passive.

It is important to do a good job of refactoring in highly concurrent systems. For example, when there is a large number of requests, there will inevitably be a timeout. If the business process is interrupted every time the timeout occurs, the normal business will be greatly affected. It is reasonable that we should carefully distinguish between strong and weak dependencies and adopt passive degradation for weak dependencies, while we cannot degrade for strong dependencies. Downgrade is similar to circuit breaker, which is also to protect its own services from external dependency failure. Therefore, we should make sufficient plans for downgrade.

Current limiting

There are also many articles and introductions about limiting traffic on the Internet, and specific technical implementation can refer to online articles. My personal experience about current limiting is that before setting current limiting, it is necessary to fully estimate system capacity through pressure measurement and other methods. Do not beat your head. Current limiting is generally harmful to user experience and should be used as a backstop method rather than a conventional one.

Resource isolation

There are various types of resource isolation, including server resources and middleware resources at the physical level, thread pools at the code level, and connection pools at the code level. The resource isolation described here is mainly at the application deployment level, such as set-based, etc. The remote hyperactivity mentioned above is also a kind of Set.

I also made some similar optimization on resource isolation when I was in charge of the waybill system in Ele. me. An online fault occurs because the servers deployed for a service are in the same cluster and are not divided into separate clusters based on traffic. As a result, the critical service traffic and non-critical service traffic affect each other. Therefore, AFTER this failure, I also decided to deploy servers in isolation by cluster. The isolation dimensions are mainly divided into key cluster, sub-critical cluster, and non-critical cluster based on service scenarios, so as to avoid the interaction between key and non-critical services.

summary

In terms of architecture, I am not a professional architect, and I have been learning related technologies and methodologies. Many of the technologies and architecture design patterns introduced above are learned and practiced at work. If I have to draw one lesson from my experience, I think it’s attention to detail. In my opinion, architecture is not only about lofty methodology, but also about technical details. Details determine success or failure. Sometimes forgetting to set a small timeout may lead to the collapse of the whole system.

application

In a high concurrency system, there is a lot of optimization that can be done at the application level. This section mainly shares the optimization of compensation, idempotence, asynchronization, preheating and other aspects.

The compensation

Under micro service architecture, can according to the different service areas of business split, service and service before through RPC requests or MQ message interaction, in a way that will inevitably exist call fails in a distributed environment, especially in high concurrent systems, due to the server load, higher failure probability is bigger, so the compensation is more important. There are two common compensation modes: scheduled task mode or message queue mode.

Scheduled task mode

Timing task compensation model is usually need to cooperate with the database, the compensation when a timer task, timing task execution time will scan, there is a need to compensate the data in the database if there is the executive compensation logic, the advantages of this scheme is due to the persistent data in a database, are relatively stable, not easy to a problem, The deficiency is due to the dependence on the database, which will cause certain pressure to the database when the data volume is large, and the scheduled task is periodically executed, so the general compensation will have a certain delay.

Message queue pattern

The message queue compensation pattern typically uses the message delay feature of the message queue. Handle failure, then sends a message delay, delay N minutes/seconds/hours later and try again, the benefits of this approach is more lightweight, in addition to the MQ no external dependencies, implementations are relatively simple, relatively more real-time, insufficient place is because there is no persistence in the database, there is the risk of loss of data, not stable enough. Therefore, my personal experience is to use the mode of scheduled task in critical link compensation, and the mode of message queue can be used in non-critical link compensation. In addition, there is a particularly important point in compensation that is idempotent design.

Power etc.

An idempotent operation has the same impact as a single operation when executed for multiple times. This is reflected in the business that the results of a single or multiple request for the same operation are the same without side effects. In distributed system, system error is inevitable. When errors occur, retry and compensation will be used to improve fault tolerance. In high concurrency system, the probability of system error is higher, so interface idempotence is very important to prevent the side effects caused by multiple requests.

Idempotent implementation requires using a unique ID or Token, the general process is to query the only business in DB or cache ID or Token exists, and state whether it is handled, if it is said is repeated requests, so we need to deal with the idempotent, which don’t do any operation, can be returned directly.

When doing the idempotence design need to be aware that not all scenarios to be idempotent, such as user repetitive transfer, withdrawal, etc., because power will make the perception of the external system is called a success, and not block the follow-up process, but in fact we didn’t do any operation system’s interior, similar to the above mentioned scenario, will let users mistook operation has been successful. Therefore, careful distinction should be made between idempotent and non-idempotent business scenarios. For non-idempotent business scenarios, service exceptions should be thrown or specific exception codes returned to block subsequent processes to prevent service problems.

asynchronous

The message queue mentioned above is also a kind of asynchronization. In addition to relying on external middleware, we can also do asynchronization in the application through thread pool and coroutine.

On the realization principle of thread pool, take the Java thread pool model for example, the core is through the task queue and the way of reuse thread to achieve, online about these shared articles are also many. My personal experience with thread pools and coroutines and the like is that there are two things to be aware of:

Compensation is required in key service scenarios

We all know, whether the thread pool, coroutines, are based on memory, if the server is unexpected downtime or restart data in memory will be lost, and the thread pool when insufficient resources might also refused to task and so on some key business scenarios if you use the similar technology such as thread pool, need to use with a compensation, Avoid service impact caused by memory data loss. In the waybill system I maintain, a key business scenario is the receipt of waybills. Simply speaking, it is to receive upstream requests and generate waybills in the system, which is the entrance of the whole logistics performance flow and a particularly key business scenario.

Because the waybill of the whole process longer, dependent on external interface has ten several, so at that time, in pursuit of high performance and throughput, designed in asynchronous mode, which is in the thread pool, at the same time in order to prevent data loss, also made a perfect compensation measures, hungry? This time into the single year this basic no problems, And because of the use of asynchronous design, performance is very good, so how do we do.

The overall process is that after receiving the upstream request, the first step is to put all the request parameters into the library, this step is very critical, if this step fails, the whole request will fail. After a successful drop-off, encapsulate a Task to submit to the thread pool and return success directly upstream. All subsequent processing is done in the thread pool, in addition, there is a timed task will be compensated, the compensation of the data source is the first step in the decline of library data, record each fall library will have a flag field to indicate the processing state, if found to be not processing or treatment failure, by timing task trigger compensation logic again, After the compensation succeeds, update the flag field to processing success.

Do a good job in monitoring

In microservices, such as RPC interface call, MQ message consumption, middleware, infrastructure and other monitoring, these will be targeted to do perfect monitoring, but there is no ready-made monitoring for thread pools, users need to implement their own reporting monitoring, which is easy to be omitted. We know that the implementation of thread pool will have a memory queue, and we generally set a maximum value for the memory queue, if the maximum value is exceeded, the task may be discarded, at this time without monitoring can not find similar problems, so, the use of thread pool must be monitored. What indicators can be monitored in the thread pool? According to my experience, the number of active threads in the thread pool and the number of tasks in the work queue are generally reported. I think these two indicators are the most important, and other indicators can be reported selectively based on specific business scenarios.

preheating

Warm Up. When the system is at a low water level for a long time and the flow suddenly increases, pulling the system directly to a high water level may overwhelm the system instantly. Through the “cold start”, the flow slowly increases to the upper limit of the threshold in a certain period of time, giving the cold system a time to warm up, preventing the cold system from being overwhelmed.

Referring to the definition of the Internet, to put it bluntly, if the service has been at a low water level, then a sudden wave of high concurrent traffic, may suddenly bring the system down. System preheating generally includes JVM preheating, cache preheating, DB preheating, etc. By preheating, the system is first “hot” to prepare for the arrival of high concurrent traffic. There are many scenarios for preheating practical applications. For example, before the big push of e-commerce comes, we can load some hot commodities into the cache in advance to prevent heavy traffic from hitting DB. For example, Java service, due to the dynamic class loading mechanism of JVM, can do a wave of pressure test on the service after startup, and load the classes into memory in advance. It also has the advantage of triggering JIT compilation, Code cache, and more.

There is also a preheat the idea is to use the characteristics of business to do some preload, such as when we were in the maintenance of waybill system made a such optimization, is in a normal delivery business process users after place the order to the trading system to generate orders, then accept the order after payment – > business – > request distribution such a process, Therefore, there is a time difference of seconds to minutes between the user placing an order and the delivery request. We can take advantage of this time difference to load some data in advance by sensing the user’s action of placing an order.

In this way, when the actual request comes, we only need to obtain it from the cache, which greatly improves the performance of some time-consuming operations. Previously, we used this method to improve the interface performance by more than 50%. Of course, one point that needs to be noted is that if some data may change, it may not be suitable for preheating, because the preheated data is stored in the cache, and the interface will not be requested later, which will lead to data inconsistency, which needs special attention.

summary

When designing high-concurrency systems, we always pay special attention to architecture, infrastructure, etc., which is very important, but there are a lot of optimization that can be done at the application level, and the cost is much lower than architecture, infrastructure optimization. A lot of time at the application level optimization requires a combination of specific business scenarios, use a specific business scenarios to make reasonable design, such as the cache, asynchronous, we need to think about what the business scenario can cache, asynchronous, which need to be synchronized, or query the DB, must be combined with business to make a better design and optimization.

specification

This is the last part of my experience sharing on building high-concurrency systems, but I think specifications are no less important than infrastructure, architecture, databases, applications, and probably more important than any of them. According to the 80/20 rule, in the whole life cycle of software, we spend 20% of the time creating the system, but 80% of the time maintaining the system. This reminds me of a sentence that some people say that the code is mainly for people to read, and incidentally for machines to run, which actually reflects the importance of maintainability.

The system did have a good design after we used a high profile architecture and made all the optimizations, but the question was how to prevent architecture corruption during subsequent maintenance, and that’s where the specification came in.

Specifications include code specifications, change specifications, design specifications and so on, of course, I will not introduce how to design these specifications, I would like to say that we must pay attention to specifications, only after the specification, the maintainability of the system can be guaranteed. According to the broken window theory, we try not to have the first broken window in the system through various specifications.

conclusion

Having said so much about design and optimization methods, I would like to share two more points at last.

The first point is that there is a famous saying – “premature optimization is the root of all evil”, I very much agree that all these designs and optimization, I do, are in the system encountered actual problems or bottlenecks, do not be divorced from the actual scene of premature optimization, otherwise it is likely to do useless work or even more than the loss.

Keep it simple, stupid. Simplicity means higher maintainability and less problems. Perhaps this is the truth of simplicity.

All the above are my summary of experience in maintaining high-concurrency system during my work in Ele. me. In view of the length and personal technical level, some parts may not be particularly detailed and in-depth, which is to introduce the jewel. If there is something wrong, we welcome to point it out, and we also welcome exchanges and discussions.