1. The background

Profile 1.1.

As the leader of online ticketing in China, Maoyan has experienced significant development such as brand independence and merger since its establishment. With the independence and merger of brands, accounts, as the cornerstone of business, must consider the independent construction of accounts and the integration of multi-account systems. Up to now, maoyan c-terminal account system has produced a set of account architecture similar to SaaS to support the business product requirements of different channels and systems.

The saas-based system structure determines the data isolation between upper tenants, and the saas-based response of the corresponding account system forms the phenomenon of the same natural person using accounts in different channels. Account isolation meets the differentiated needs of different businesses in the early stage, avoids the impact of historical burden on new businesses, and quickly supports the creation of product matrix. However, as services mature, data isolation of user accounts between different products leads to separation of upper-layer services, assets, and data. For users, the problems of data fragmentation and brand identity occur. For businesses, fragmentation cannot quickly support multi-channel business linkage, and channels cannot effectively use each other’s flow dividends. Currently, maoyan has two major account systems: Maoyan APP account system and small program account system.

The above problems gradually appeared and amplified in the development process of Cat’s Eye, which not only failed to guide the new business well, but even became a stumbling block in the development process of new business. Based on the above background, Maoyan has carried out a special action for the integration of multiple account systems, practicing internal skills, and hoping to completely solve this lingering disease.

Problem of 1.2.

Account integration through is not simple account information data relocation, many problems need to be considered, including:

Flexible opening and integration of differentiated accounts allow users to naturally accept the original fragmented experience integration without perception or weak perception
Supports subjective processes such as user binding change and deregistration, and objective scenarios such as secondary number allocation by mobile operators
Support the demand for accounts in different business scenarios, such as film community focusing on the display of basic user information, fission scene focusing on the display of tripartite information, and e-commerce scene focusing on the unique identification of natural persons
Effectively guide and deal with abnormal accounts in the integration process to avoid large-scale problems
Ensure the consistency and integrity of global 100-million-level accounts and service data, and ensure the overall reliability of services
Reduce business transformation costs and integrate in a less intrusive manner

1.3. Research

When a number of independent product development to a certain stage, product ownership change, product mutual diversion and other reasons for product integration are jumped through the integration of accounts, we first investigated the implementation of the industry in the account transformation. Industry based on their own historical reasons, business considerations have chosen not the same account opening scheme, we respectively on several large factories of the current version of the product account opening ability for research. According to our understanding, the account opening scheme can be roughly divided into three categories:

We try to analyze the similarities and differences behind each product logic:

The same point: did not do to the account system itself integration and other large-scale transformation
Differences: Due to product positioning and service complexity, accounts can be opened by strengthening binding, weakening binding, weakening historical data, and silent binding

Back to the scene of Cat’s Eye, cat’s eye has developed two main account systems, APP and small program, with the following two characteristics:

There is no great difference between small program and APP in business and function. This is different from wechat &QQ, Meituan & Dianping, Taobao & Alipay
Both small programs and APPS have hundreds of millions of users and are highly active. This is different from JD.com

These two points are different from the above industry background. Blindly learning from the experience of big factories cannot solve the pain points of Maoyan’s existing business. Therefore, we have made clear the idea of designing maoyan’s multi-account system integration scheme.

At the same time, in the research process, we also further clarified the three core goals of account integration:

Emphasize your own brand
Integrating business assets
Weakening user influence

2. The cat’s eye practice

2.1. Overall integration scheme

2.1.1. Conditions for getting through – common anchor points

In order to open an account, it is necessary to identify a public anchor (i.e. the unique public identity of a natural person in different channels and platforms) and take the public anchor as the intermediate bridge between different account systems. Common unique identifiers between different account systems mainly include mobile phone numbers and third-party (such as wechat) open platform identifiers. Neither of these two categories can be used seamlessly for Cat’s Eye. The actual situation of cat’s Eye account is as follows:

APP side: Use mobile phone number to register. Registration and login require binding mobile phone number, which is used as the unique identification of different channels
Small program side: use wechat open platform information for registration, registration and login do not require binding mobile phone number, unionId as the unique identifier of different channels

The inability to uniquely identify a natural person has become one of the pain points of maoyan’s various businesses. In addition to the unmet demand for opening accounts, the derived promotional scenarios such as coupon issuance and new and old customer activities may bring additional cost problems.

After analyzing the user group habits, user scale, industry habits and other factors, Maoyan finally decided to take the mobile phone number as the public anchor point.

2.1.2. Landing – writing integration

In the industry, read through is the main way to open multiple accounts. That is, after the account data, user data, and service data are not merged but are displayed and silently bound, data can be selectively merged based on services. The process is roughly as follows:

The differences between cat’s eye and typical business cases have been explained above. If it is realized in the way of reading through, we will face three main problems:

SLA of basic services is difficult. All user operations need to query account connections, which increases the pressure of service requests such as account requests, and may double in extreme scenarios. Maoyan will face great challenges in SLA guarantee under popular activities such as performance seckilling
End-to-end response times are up. To display all assets, you need to query the account relationship before merging the assets of multiple accounts. In addition to the increase in read traffic, the response time of service requests also increases, which directly affects the user experience on the C-terminal
Business iterative development requires compatible reading. In the iterative process of all services, the read through process must be compatible to ensure consistent user experience. Otherwise, the read through process will bring additional troubles to users. In addition, the read through process will bring continuous operation and maintenance costs to services

Based on the above three main reasons, we did not choose the way of read opening, but chose the way of write opening as shown in the picture below: to completely unify the split accounts and relocate the information of users and assets under the accounts. Write through the biggest advantage is once and for all, business write through do not need to consider compatibility through the problem, you can pack light. But at the same time write through in the implementation of the scheme will face greater challenges, including: performance, availability, data consistency and so on, these problems need to do more perfect design and balance in the technical scheme.

2.1.3. Account carrier after integration – target account

The public anchor describes the media problem of account opening, write opening clear way of multi-account opening, and the remaining account carrier problem after opening.

For example, the user has APP account U1 and mini program account U2. After these two accounts are connected, the account bearing account, user and asset information is the target account. There are three options for the optional target account:

We choose APP account as the target account. In addition to the above account and business transformation items, there are also:

Maoyan APP has been online for a long time and has stronger brand recognition, and the number of existing users is larger than the number of existing users of small programs
The transformation of APP end registration and login is complicated, and the processing of stock version needs to be considered, and the realization of client end grayscale scheme is more difficult

At this point, the overall scheme of multi-account system integration is determined: the mobile phone number is taken as the public anchor of the two-side account system, and the small program account is integrated into the APP account through writing integration.

2.2. Account service integration

Before the overall start, let’s first introduce the background information of maoyan mini program account system: There are 200 million mini program accounts in stock. Due to historical reasons, this part of account data is scattered in multiple systems:

HIVE library: Stores basic information about small program accounts, including userIDS and registration information
Mini program account service: it saves the three-party authorization records of mini program users in the wechat system, such as the openId and unionId of mini program and public account
Small program user service: save the small program user phone number authorization, binding records
Wechat service: save the following public account user information, as well as micro credit account like, nickname information

Based on the above background information, the goal of the first stage is to complete the unification of data and systems and integrate the separated account services. In the specific time, we split it into three steps:

Create incremental accounts, modify and read account information. Convergence at the interface and service levels is complete
Relocation of historical accumulated stock data to ensure data integrity and consistency
Registration and login transformation, completely decoupled with historical services and data media

2.2.1. Step 1: Incremental account processing – data double-write

At this stage, we focused on the reconstruction of the entrance of small program registration and login process, completed the unified account registration and login docking facade, and tried to shield the influence of late transformation on the front end. In addition, incremental account data is synchronized at the same time. User data is synchronized between the two systems during registration and login to avoid incremental data omission. In the concrete scheme, we abstract the unified gateway of account service to shield the difference of operation in account life cycle of different account systems. The relationship between the two account systems after transformation is shown as follows:

2.2.2. Step 2: Stock account processing – data relocation

After the first step of registration and login transformation, incremental data can be synchronized to the unified account system, but the stock data cannot be effectively migrated, so it is necessary to synchronize the stock account data.

As mentioned in the background above, the main technical challenges in this phase include:

Scattered data sources increase the complexity of cleaning and make it difficult to ensure data consistency.
Large-scale data and scattered data sources further lead to a long period of synchronization.
Long synchronization periods magnify data inconsistencies

To solve the problems of large-scale data, multi-data sources and intricate data consistency, Maoyan adopts various methods:

In terms of performance, it has designed the ability to support horizontal extension processing. In addition to the conventional multi-process and multi-thread extension, targeted tuning has been carried out, such as TCP parameter tuning for network IO and appropriate adjustment for JVM parameters.
In terms of consistency, multiple rounds of data synchronization, data inspection and data compensation are adopted to ensure the final consistency of data.
During data synchronization, abnormal tasks can be failover. After data synchronization, abnormal data can be acquired for compensation. Extreme scenarios can be degraded or rolled back

The overall architecture of data relocation is shown in the figure below:

2.2.3. Step 3: Historical service decoupling – register from both sides to full switch

After the completion of the second step, the small program account service no longer provides services directly to the outside world. Next, we will completely complete the decoupling of the small program account system. The decoupling of the original applets account system offline requires special consideration of the change of account ID generation logic. There may be various potential risks due to the change of account ID generation logic in the company’s overall business architecture, such as: Conflicts between ID number segments of different account systems may lead to major online accidents, and changes in number segments of function push across product systems may lead to abnormal upper-level services of accounts, such as payment and data statistical analysis. In addition to the multi-account system of APP and mini program, Maoyan also faces the interaction of Meituan and comment business scenes, as well as the identification of account ID by different services of Maoyan and Meituan.

Based on the above reasons, we still decided to adopt a conservative two-side registration strategy to minimize the impact on the business after fully conducting business research and sorting. Specifically, the account generation logic of the mini-program account system will not be directly replaced. When the mini-program account is registered, two different account ids will be generated in the original mini-program account service and the new unified account service, and only the account ID generated by the unified account service will be used in the whole business. The unified account service maintains a mapping relationship between two account ids. If unknown conflicts or faults occur due to account ID changes during the online process, the account service is still capable of cleaning account ids to minimize the impact of potential faults.

After both sides of the registration phase stabilized, we started the final phase of account service integration — complete switching, which means completely no longer relying on the applets account service. So far, we have completed the integration of the two sets of accounts at the system level, and solved the problems of account side entrance and data convergence for the subsequent integration of global assets.

2.2.4. Account service integration summary

The registration and login process in the above three steps roughly experienced the following process:

Double-write: Incremental data synchronization is performed on both sides
Two-way registration: basically decouples the applets account service, keeping only the account ID generation logic
Full switch: Completely shut down the applets account service
Rollback Dual-write: Compatible with both sides of the registration, full switchover phase degradation

In the process of integration, it follows the principle of first increment and then stock. After unified convergence at the interface level, the dependence on the small program account system is gradually reduced and decoupled, and data relocation and cleaning are carried out inside the account service, so as to be unaware of the service as much as possible. Concrete ground practice, given the registered login is a typical writing scene in the release, and roll back the natural complex from general read operation process, login and registration process for all core business starting node, failure will lead to core business directly affected, the industry also frequently appeared many times great business losses caused by account service is not available, We adopted a strategy of small steps in the transformation of registration and login process, and split the transformation of C-end registration and login process into multiple stages, focusing on ensuring that the transformation of each stage can be gray scale, observable, degraded and rolled back.

2.3. Global asset integration

After completing the account service integration scheme, Maoyan began to integrate global assets. Asset integration We split into two phases:

Integration and cohesion. Asset integration is embedded in c-end registration and login. On the basis of registration and login, additional operations such as identity authentication and secondary number allocation authentication required by integration are extended to complete the connection of front and back end interaction processes
Integrate execution. After receiving the consolidation request, global service data is scheduled for consolidation, which is the actual global data consolidation stage

2.3.1. Integration and cohesion

As a necessary condition for global integration, account integration can be embedded into the following positions: core operations within account life cycle and operations within business processes. Considering the low intrusion to the business, Maoyan completed the logic embedding in the registration and login stage of the account to compare the differences before and after the registration and login transformation:

On this basis, the transformation of registration and login still considers interface version rollback and data compatibility after rollback, and can control the actual version used by registration and login to complete negative feedback adjustment according to the actual implementation of integration.

2.3.2. Integrated execution

The user asset data global consolidation process is similar to the Java Virtual machine Full GC process. FGC needs STW to prevent GC from generating new garbage at the same time and leading to incomplete GC. Similarly, user data merge needs a period of time to block user and business operations to avoid incomplete or incomplete data due to new data generated during the merge. How to design global business pause becomes the first problem in asset data merge. Million level user oriented, dozens of assets on a large scale, large areas of data integration of program performance, robustness and challenge, how to effectively complete fusion of scheduling, ensure global data consistency and integrity, the abnormal task, perception, observation data, positioning became the second problem in data integration.

Business pause problem

Pause for global business problem, cat’s eye in the internal combed the various business scenarios, typical C end business scenarios include seat selection, performance, order, payment, UGC, fission activities, etc., in these scenarios of business data integration needs to face different treatment processes, such as typical synchronous scheduling process, many service between asynchronous scheduling process, In order, UGC and other scenarios, serious write amplification needs to be considered, and reliable global data integration after overall evaluation needs minutes or even hours of pause to complete. However, this is seriously inconsistent with the daily usage habits of Maoyan users. Typical usage scenarios of users include ticket purchase and ticket collection. It is assumed that suspending users during ticket purchase and ticket collection will cause serious harm to users.

So the cat’s eye not adopt synchronous blocking the way of consolidation, but took a “asynchronous non-blocking” ways: login with the user to confirm account integration information, record the information, in the morning business slack period began to task scheduling, lock from the account level, to global business data integration, data integration in the global business is completed to unlock the account.

Integrated scheduling problem

For the problem of integration scheduling, we have built an integration scheduling center for the overall governance of integration tasks. Its functions include recording and reporting integration information, scheduling global integration tasks, recording global integration trajectory, and supporting different application scenarios such as testing, operation and maintenance, and customer service.

The overall structure is shown as follows:

In terms of stability, the integration scheduling center embedded registration and login, core business processes need to ensure high availability, and can support millions of calls per day. In the specific design, we focus on four aspects:

Grayscale: ensure that the gradient grayscale and scale are controllable. At the initial stage of online, the gray scale within the internal and controllable range can be exposed and hidden while avoiding external users being affected; Avoid system and service faults caused by data overload
Downgradable: In the downgradable scheme, due to persistent data and uncertainty of subjective operation by users, the specific implementation of the downgradable scheme is compatible with various scenarios, including gray scale, integrated users, reported users, unbinding and cancellation
Observable: data visualization of all dimensions is supported after the launch, and overall data indicators of total volume, business perspectives and stages are supported to ensure visualization and traceability of integrated records
Failure-oriented: The asynchronous scheduling retry mechanism is adopted during the integration process to ensure that users and services are not blocked

In terms of performance, heterogeneous storage architecture is adopted to maximize strengths and circumvent weaknesses to support the data scale of ten billion. Specific:

Use MQ asynchronous, peak clipping and valley filling capabilities to maximize processing speed while avoiding downstream service failures due to overloaded traffic
The database transaction and interval query are used to store and integrate the mapping relationship, and multi-level cache is used to improve the query performance and meet the query requirements of the core process
ES and Hive support retrieval and large-scale data archiving

In addition, theme isolation is carried out for PaaS layer, and speed limiting and horizontal expansion are supported to avoid PaaS unavailability caused by overload peak flow and ensure high reliability and performance of the overall service.

3. Summary

The above are some practices of multi-account system integration of Maoyan account system in the development process. Based on the investigation of industry solutions, we designed the multi-account system integration process of Maoyan based on the product and business background of Maoyan, and based on this process, Maoyan has completed the integration of hundreds of millions of accounts.