The background,

With the rapid growth of the number of users, the single architecture of VIVO official mall V1.0 has gradually exposed its drawbacks: increasingly bloated modules, low development efficiency, performance bottlenecks and system maintenance difficulties.

The V2.0 architecture upgrade, which was launched in 2017, carries out vertical system physical separation based on service modules. The business lines are separated to perform their respective functions and provide servicification capabilities to jointly support the master service.

The order module is the transaction core of the e-commerce system, and the continuously accumulated data is about to reach the single table storage bottleneck. The system is difficult to support the flow during the new product release and promotion activities, so the transformation of service is imperative.

This paper will introduce the problems and solutions encountered in the construction of vivo mall order system, and share the experience of architecture design.

Second, system architecture

The order module will be separated from the mall and become an order system independently. With an independent database, the order, payment, logistics, after-sales and other standardized services will be provided for the relevant systems of the mall.

The system architecture is shown in the figure below:

Third, technical challenges

3.1 Data volume and high concurrency

The first challenge comes from storage systems:

• Data volume problem

  With the continuous accumulation of historical orders, the amount of order table data in MySQL has reached tens of millions.

  We know that the storage structure of InnoDB storage engine is B+ tree, and the search time complexity is O(log N). Therefore, when the total amount of data n increases, the search speed will inevitably slow down. No matter how to add index or optimize, it cannot be solved, but to reduce the amount of single table data.

  The solutions for large amounts of data include data archiving and table splitting

• High concurrency problem

  Mall business is in a period of rapid development, the order volume has repeatedly set new highs, the business complexity is also improving, the application of MySQL visits more and more.

  The processing capacity of standalone MySQL is limited. When the pressure is too high, the access speed of all requests will decrease, and even the database may break down.

  High concurrency solutions include caching, read/write separation, and separate libraries

These schemes are briefly described below:

• Data archiving

  Order data has time attribute and has hot tail effect. In most cases, it retrieves recent orders, while a large number of old data with low frequency are stored in the order table.

  Storing old and new data separately, moving historical orders into a separate table, and making some changes to the query module in the code can solve the problem of large data volumes.

• Use the cache

  Using Redis as the pre-cache of MySQL can block most query requests and reduce response latency.

  Caching is particularly good for systems that have little relationship with users, such as commodity systems, but for order systems, the order data of each user is different, so the cache hit rate is not high and the effect is not very good.

• Reading and writing separation

  The master library is responsible for executing data update requests, and then synchronizes data changes to all slave libraries in real time, sharing query requests with multiple slave libraries.

  However, there are many update operations of order data, and the pressure of the master database is still not solved when the order peak is placed. In addition, there is a synchronization delay between the master and the slave. Normally, the delay is very small, less than 1ms. However, data inconsistency between the master and the slave may occur at a certain time.

  This would require compatibility for all affected business scenarios, with possible compromises such as switching to an order success page after a successful order is placed so that the user manually clicks to view the order.

• depots

  Sub – library includes vertical sub – library and horizontal sub – library.

  ** The data of the same table can be separated into different databases according to certain rules, and each database can be placed on different servers.

  **② Vertical branch database: ** will be classified according to the business table, distributed to different databases above, each library can be placed on different servers, its core idea is dedicated library dedicated.

• table

  Sub-table also includes vertical sub-table and horizontal sub-table.

  **① Horizontal table: ** In the same database, the data of a table according to certain rules to separate into multiple tables.

  ** Divide a table into multiple tables by field, and each table stores some of the fields.

After considering the cost, effect and impact on the existing business, we decided to use the last option: separate inventory and separate tables

3.2 Selection of sub-database and sub-table technology

The technical selection of sub-database sub-table is mainly considered from the following directions:

1. Open source client SDK solution

2. Middleware Proxy open source solution

3. A self-developed framework provided by the company’s middleware team

4. Build your own wheel

After referring to the previous project experience and communicating with the middleware team of the company, the open-source Sharding-JDBC scheme was adopted. It has now been renamed Sharding-Sphere.

• Github:github.com/sharding-sp…

• Documentation: official documentation is rough, but online information, source code analysis, demo is rich

• Community: Active

• Features: Jar package, belongs to the client fragment, support XA transactions

3.2.1 Strategy of database and table division

Combined with the service characteristics, the user id is selected as the sharding key, and the library table number of the user order data is obtained by calculating the hash value of the user ID and then taking the modulus. Suppose there are n libraries with m tables in each library,

Then the calculation method of library table number is:

– Library id: Hash(userId)/m % n

– Table id: Hash(userId) % m

The routing process is as follows:

3.2.2 Limitations of sub-database sub-table and corresponding solutions

Split table solves the problem of data volume and concurrency, but it will greatly limit the query ability of the database, some of the previous very simple associated query, in the split table after the database may not be able to achieve, it will need to separate these Sharding-JDBC does not support SQL rewrite.

In addition, these challenges were encountered:

(1) Global unique ID design

However, many interactive interfaces between internal systems only have the order number. There is no user identification for the shard key. How can we use the order number to find the corresponding library table?

It turns out that the library table number is implicit in the order number generated. This allows the library table number to be retrieved from the order number in a scenario where there is no user identity.

(2) There is no implied library table information in the historical order number

Use a single form to store the mapping between the historical order number and the user id. Over time, these orders are no longer interacted with the system and are no longer used.

(3) The management background needs to query all orders that meet the conditions in pages according to various screening conditions

Order data is redundantly stored in the search engine Elasticsearch for background queries only.

3.3 How to Synchronize data from MySQL to ES

SQL > select * from Elasticsearch; select * from esql; select * from ESQL;

The considerations here are timeliness and consistency of data synchronization, minimal intrusion to business code, and no impact on the performance of the service itself.

• MQ solutions

  The ES update service, as the consumer, updates the ES after receiving the ORDER change MQ message

• Binlog scheme

  With the help of canal and other open source projects, ES update service disguises itself as a slave node of MySQL, receives Binlog and analyzes and obtains real-time data change information, and then updates ES according to this change information.

Among them, BinLog scheme is more general, but the implementation is also more complex, we finally choose MQ scheme.

Because ES data is only used in the management background, the requirements for data reliability and real-time synchronization are not particularly high.

The ability to manually synchronize ES data under certain conditions was added in the background to compensate for extreme situations such as outages and message loss.

3.4 How Can I Securely Replace the Database

How to migrate data from the original single-instance database to the new database cluster is also a technical challenge

Not only do you want to ensure that the data is correct, but you also want to be able to quickly roll back to the previous step if there is a problem after each step.

We considered two scenarios of downtime migration and non-stop migration:

(1) Non-stop migration scheme:

• Copy the data of the old library to the new library, put on line a synchronization program, use Binlog and other schemes real-time synchronization of the old library data to the new library.

• Online double write order old and new library service, only read and write old library.

• Open double write, stop the synchronization program at the same time, open the contrast compensation program, to ensure that the new library data and the old library consistent.

• Gradually cut read requests to the new library.

• All reads and writes are switched to the new library, and the compensator ensures that the old library data is consistent with the new library.

• Offline old library, offline order double write function, offline synchronization program and comparison compensation program.

(2) Shutdown migration scheme:

• Launch the new order system, perform the migration procedure to synchronize the orders from two months ago to the new database, and audit the data.

• Shut down mall V1 application to ensure that the old database data will not change.

• Perform the migration procedure to synchronize and audit the orders that were not migrated in the first step to the new library.

• Online mall V2 application, start the test and verification, if failed, back to mall V1 application (the new order system has a switch to double write the old library).

Considering that the transformation cost of the non-stop scheme is high, and the business loss of the night shutdown scheme is not big, the final choice is the shutdown migration scheme.

3.5 Distributed transactions

In the transaction process of e-commerce, distributed transaction is a classic problem, such as:

• After the payment is successful, the user needs to notify the delivery system to deliver the goods to the user.

• After users confirm receipt of goods, they need to inform the points system to issue points to users for shopping rewards.

How do we ensure data consistency in the microservices architecture?

Different service scenarios have different requirements on data consistency. In mainstream solutions, two-phase commit (2PC) and three-phase commit (3PC) are used to resolve strong consistency, and TCC, local message, transaction message, and maximum effort notification are used to resolve final consistency.

Instead of describing the above scheme in detail, here is a local message table scheme we are using: asynchronous operations to be performed in a local transaction are recorded in the message table and can be compensated by a scheduled task if the execution fails.

The figure below is an example of giving bonus points after the order is completed.

3.6 System Security and Stability

• Network isolation

  Only a few third-party interfaces can be accessed from the Internet, and all the third-party interfaces verify signatures. Internal systems interact with each other using Intranet domain names and RPC interfaces.

• Concurrent lock

  Any order update operation is restricted by database row-level locking to prevent concurrent updates.

• idempotence

  All interfaces are idempotent, and there is no need to worry about the impact of timeout retries on the other network.

• fusing

  With Hystrix components, fuse protection is added to real-time calls to external systems to prevent the impact of a system failure from spreading to the entire distributed system.

• Monitoring and Alarm

  By configuring the error log alarm of the log platform, the service analysis alarm of the call chain, and the monitoring alarm function of the middleware and basic components of the company, we can find the system exception in the first time.

3.7 Trample pits

MQ consumption is adopted to synchronize the order-related data of the database to ES, and the writing data encountered is not the latest data of the order

The left side of the picture below is the original plan:

In the consumer orders data synchronization of MQ, if thread A prior execution, detected data, this time the order data is updated, thread B began to perform synchronous operation, find out the order during the data prior to thread A step after writing ES, thread A will when performing A write thread B write data coverage, causes the order data in the ES is not up to date.

The solution is to add a row lock when querying the order data, the entire business is executed in a transaction, and the next thread is executed when the execution is complete.

Sharding – JDBC group after sorting page query all data problems

Select a from temp group by a, b order by a desc limit 1,10

Mysql > set 10 to integer. MAX_VALUE; mysql > set 10 to integer. MAX_VALUE; mysql > set 10 to integer. MAX_VALUE;

io.shardingsphere.core.routing.router.sharding.ParsingSQLRouter#processLimit private void processLimit(final List<Object> parameters, final SelectStatement selectStatement, final boolean isSingleRouting) { boolean isNeedFetchAll = (! selectStatement.getGroupByItems().isEmpty() || ! selectStatement.getAggregationSelectItems().isEmpty()) && ! selectStatement.isSameGroupByAndOrderByItems(); selectStatement.getLimit().processParameters(parameters, isNeedFetchAll, databaseType, isSingleRouting); } io.shardingsphere.core.parsing.parser.context.limit.Limit#processParameters /** * Fill parameters for rewrite limit. *  * @param parameters parameters * @param isFetchAll is fetch all data or not * @param databaseType database type * @param isSingleRouting is single routing or not */ public void processParameters(final List<Object> parameters, final boolean isFetchAll, final DatabaseType databaseType, final boolean isSingleRouting) { fill(parameters); rewrite(parameters, isFetchAll, databaseType, isSingleRouting); } private void rewrite(final List<Object> parameters, final boolean isFetchAll, final DatabaseType databaseType, final boolean isSingleRouting) { int rewriteOffset = 0; int rewriteRowCount; if (isFetchAll) { rewriteRowCount = Integer.MAX_VALUE; } else if (isNeedRewriteRowCount(databaseType) && ! isSingleRouting) { rewriteRowCount = null == rowCount ? -1 : getOffsetValue() + rowCount.getValue(); } else { rewriteRowCount = rowCount.getValue(); } if (null ! = offset && offset.getIndex() > -1 && ! isSingleRouting) { parameters.set(offset.getIndex(), rewriteOffset); } if (null ! = rowCount && rowCount.getIndex() > -1) { parameters.set(rowCount.getIndex(), rewriteRowCount); }}Copy the code

Select a from temp group by a desc, b limit 1,10; The version used is sharing-JDBC 3.1.1.

ES paging query if duplicate values sort field there is a unique field as a condition of the second order, avoid the paging query is missing data, find out duplicate data, such as used as the only sort order creation time conditions, at the same time if there is a lot of data, will lead to query order omission or duplication, Either add a unique value as the second sort condition or use the unique value as the sort condition.

Four,

• A one-time online success, stable operation for more than a year

• Core service performance increased by more than 10 times

• The system is decoupled and the iteration efficiency is greatly improved

• Can support the rapid development of the mall for at least five years

Five, the conclusion

We do not blindly pursue cutting-edge technology and ideas in system design, and we do not directly adopt mainstream e-commerce solutions when facing problems, but choose the most appropriate method according to the actual situation of the business.

In my opinion, a good system is not designed by Niu at the very beginning, but must be gradually iterated out with the development and evolution of the business, so as to continuously predict the development direction of the business and formulate the architecture evolution plan in advance. Simply put, it is: go to the front of the business!

Author: Development team of Vivo official website mall