preface

In order to deal with the software crisis, software engineering was born in the hope of reducing the cost of software production, improving the quality of software products and improving the level of software productivity. Since the 1960s, from modular, object-oriented design to design patterns, from waterfall models to agile development, dev-OPS, guiding theories and engineering approaches to software production have all evolved, and software productivity has improved dramatically. However, until today, the contradiction between the growth of business demand and the shortage of enterprise development resources still exists widely.

At the same time, in recent years, the concept of no-code/low-code has received more and more attention from domestic and foreign enterprises, and various zero-code and low-code development platforms emerge in an endless stream. According to Gartner research, low-code platforms will be used for 65% of application development by 2024. While it’s not a silver bullet to solve every problem, low code as a trend represents a significant step toward automated coding that could dramatically improve business delivery before AI programming becomes ubiquitous.

Based on the practice of iQiyi App backend in business data synchronization, this paper shares the experience of efficiently delivering business needs and avoiding repeated development based on low code platform.

Part 01

Business background

Firstly, taking mobile terminal as an example, let’s briefly review the general process that business data generally goes through before being presented to users:

· Data production background: Operations personnel or automated programs produce data through the business production background. For example, articles published by editors or users, videos uploaded, or crawlers automatically capture resources on the network, data production background stores these data in the database, and provides reading services for downstream businesses to obtain data. When the data is modified, the downstream is notified to update the data through a message.

· ** Data synchronization: ** business departments convert, aggregate and process the data generated in the production background through data synchronization service, and write it into the database and distributed cache;

· Database & Cache: store all kinds of business data for business back-end interface to read;

· ** back-end interface: ** accepts requests from the front end of App, reads various business data from cache, database and third-party interfaces, and performs various assembly processes according to business needs.

· **App front end: ** requests the back-end interface and parses the returned data, rendering it on the device and presenting it to the user.

For the sake of overall organizational efficiency, the data production department is generally focused on the data production scenario, and there is no need to focus too much on how the data is ultimately used. In practice, the data presented to the user is often rich and varied, which usually means that we need to aggregate the data from different producers. For performance reasons, it is not practical to aggregate the data by leaving it entirely to the back-end interface to access the multi-source interface in real time in response to user requests. User-oriented services tend to have high concurrent traffic. To meet the requirements of high concurrency and high availability, data is often stored in different database middleware to maintain consistency. Based on such background, the data synchronization service take the data from the production side delivered to the consumer side of the bridge role, which makes the production department can focus more on content production iteration, and the consumption side (typically backend interface) to focus on how to quickly deliver business interface and ensure high performance and high availability of the service interface.

Part 02

challenge

In the early stage of business, the data type and amount of data processed by data synchronization are not too high. In this mode, responsibilities of each part are clearly divided and iterations of each business link are efficient. However, with the continuous development of business, demand scenarios continue to enrich, gradually appeared some problems. The main performance is as follows:

** Human resource bottleneck: ** The data synchronization module carries more and more business sources. As far as my team is concerned, there have been 30 + data synchronization business at present, and most of the business needs need to adjust the underlying data, so the development manpower in the data synchronization link has gradually become the bottleneck;

** Due to urgent iteration cycle, project quality is difficult to be guaranteed: ** As business requirements often cannot be intuitively identified on the underlying data, it is easy for product students to omit the analysis of the data layer when delivering the requirements manuscript, and even business development students cannot accurately identify which basic data are dependent on in the early demand assessment stage. This results in requirements dependencies for the underlying data being identified only near release delivery, which means that development time for data synchronization is very tight. At the same time, the schedule of the node test team has been determined, and the test resources are often not fully guaranteed, which brings a certain risk to the project quality. For example, sometimes in order to deliver business requirements quickly, new requirements that are not related to the new business will be directly integrated into the original program, resulting in unnecessary coupling between different businesses, which increases the risk and complexity of late project maintenance.

** Increased management and maintenance costs of storage middleware: ** Data synchronization module is responsible for transferring all kinds of business data to the landing storage middleware, while many downstream business interfaces need to access these middleware to obtain data, which makes the interface need to know the details of data storage. Once the storage scheme needs to be adjusted (for example, virtual machines dependent on middleware need to be taken offline for maintenance, and clusters need to be migrated), in addition to the migration of stock data, data synchronization modules and many business interfaces need to be adjusted. The first step of adjustment is to confirm the workload that needs to be adjusted in dozens of projects. Not to mention then developing and implementing a coordinated migration plan across multiple projects. We developed some basic data interfaces and SDKS that encapsulate data access, which solved the problem to some extent, but on the other hand increased the maintenance costs of the basic data interfaces and associated SDKS.

** There are many scenarios of repeated coding: ** For example, each synchronous service needs to develop the code that listens to the message queue and accesses the production interface, while non-business essential capabilities such as retry, flow limiting, monitoring, etc. need to be implemented in each specific service. For a while, we looked to a generic synchronized template project to solve this problem. However, in practice, there is a contradiction between the versatility of templates and the diversity of businesses, while each business still needs to create project development and test code, and there are still high maintenance costs. Across the company, there are a number of sibling teams that have similar scenarios. For example, PC, H5 and we may all rely on the same upstream producers, and there are a lot of repeated implementations of the same scenarios. How do you avoid repeated development in this situation?

Part 03

Project research

Based on these issues, we want to find solutions that are less expensive to maintain and more efficient to develop. For this reason, we conducted a survey on data synchronization related products, and found that most of them were synchronization oriented to heterogeneous databases, or could only support batch tasks, or could not easily expand access to external interfaces. However, none of them were found to be suitable for our business scenarios. Of course, many features of these products have provided important references for us, such as the plug-in mechanism of dataX and the choreography capability of Spring Cloud Data Flow, which have inspired us a lot.

In the follow-up investigation, the low-code development platform scheme gradually emerged in recent years came into our vision. A low-code development platform is one that generates applications quickly with no code (zero code or no code) or with little code. It allows end users to develop their own applications using easy-to-understand visual tools rather than the traditional way of writing code. Build the capabilities needed for business processes, logical and data models, and add your own code if necessary. After the business logic and functions are built, the application can be delivered and updated with one click.

Combined with the problems our business has encountered, we expect to achieve the following goals at a lower cost through a low-code platform:

**1. Fast delivery ability. ** Enables fast implementation of business logic through visual choreography.

**2. Avoid repetitive development. ** has three meanings:

(1) Functional unit reuse: the same function, no matter access to middleware or access to some business interfaces, only needs to be developed once. If the new business requirements have the same scenario, such as access to the same public access interface, the previous work can be directly reused;

(2) Business scenario reuse: when different business teams have similar business scenarios, it can be quickly transplanted and only need to adjust a few parameters;

(3) CI process reuse: the development and launching of all businesses can reuse the same construction and deployment process, thus reducing maintenance costs.

**3. Flexible scalability. ** For example, the middleware that has not been supported before needs to be easily integrated into the existing functional system and reused in the subsequent business.

**4. High availability ** Stability is the cornerstone of services. For data synchronization, basic capabilities such as exception retry, traffic limiting, monitoring, and alarm are essential.

5. Easy to view the business’s dependence on middleware **. ** For example, you can check which businesses use a group of Redis clusters and which middleware resources are used by a business, which is convenient for later maintenance.

Part 04

Iqiyi Magpie Bridge platform introduction

Based on the background mentioned above, the design idea of magpie Bridge platform is as follows:

· Encapsulate reusable logical nodes and quickly implement business processes through visual arrangement of these logical nodes;

· Through platform reuse of basic capabilities; All business applications benefit from a single development.

For example, message consumption, message entity resolution and specific interface reading can be encapsulated into reusable logical nodes. When implementing business logic, only these logical nodes need to be combined in series. In the run phase, data is processed at each logical node and passed downstream sequentially, and judgment branches can be added according to business needs so that business can be quickly delivered in a way similar to drawing a flow chart.

As the picture above shows, Magpie Bridge is mainly composed of two parts: management background and synchronization engine. Management background for business development students to complete the orchestration and release of business logic. The main functions are:

  1. Simple operation, provides a visual business editor, you can complete business development by way of drag and drop; After service orchestration is complete and published, service description configuration information is generated and stored in the cloud configuration center for subsequent use by the synchronization engine. The following figure shows the editor for business development. On the far left is the list of logical plug-ins, which can be dragged to the middle canvas to form a complete business process. In the property configuration form on the right, you can specify service-related configuration parameters for each logical node, such as traffic limiting configuration, retry configuration, and associated service authorization information.

  1. Provides entity mapping template management. The mapping template describes how to convert a JSON data to the data required by the business. Developers can debug the template in the background. Fields can be converted from mapping templates by entity transformation logical nodes. When the field logic is added and modified in the later stage, it only needs to adjust the template and re-publish it to take effect, and there is no need to pull branches to modify the code, thus completing the requirement delivery more quickly. There have been several cases of delivering business requirements within 10 minutes online.

  1. Provides logical node plug-in management. After an extension implements the agreed programming interface and imports it in the background, it can be used in the business editor. You need to specify the coordinates of the repository where the plug-in resides, the full path name of the logical implementation class, and you can define the plug-in property configuration form when typing. General data synchronization business is the backend development students to complete, the backend is generally familiar with the business logic, a relatively complete plug-in backstage threshold logic extension does not exist, but because of the need to logic plugins and visual editor for integration, involves the development of the front page, this is often the back-end students unfamiliar field. In order to avoid the cost of learning the front end for students at the back end, we made the generation of the attribute form into drag-and-drop, which can quickly complete the development of the form without the foundation of the front end technology.

  1. Management of middleware resources. You can import the middleware resources required by services into the background, and select them from the drop-down list of the corresponding logical node property configuration during service development. At the same time, the platform automatically maintains the business and middleware dependencies.

  2. The management background is connected with the company’s related basic support platform to avoid repetitive manual processes to the greatest extent. For example, get through the application operation and maintenance platform to achieve one-click deployment; Connect with the log platform to automatically complete the delivery of service logs; After a service application is created, it automatically registers with the alarm monitoring platform.

The synchronization engine completes processing of the data. Firstly, in the application construction stage, the list of logical node plug-ins that need to be used is analyzed based on the business configuration and the plug-ins in the list are packaged together with the core module of the engine. After the application is deployed, the engine loads the service configuration of the configuration center, initializes access to middleware resources, and initializes logical nodes. This step is to load the configuration parameters set for each logical node instance in the service configuration, and execute the initialization logic implemented by the plug-in. After normal initialization, the engine will enter the run phase and start processing online data. After the data enters the business process from the start node, each logical node is executed in turn. During the running of the engine, the heartbeat is periodically reported to the monitoring service. If the heartbeat times out, an alarm is triggered to notify the business development students to handle it in time. Monitoring data for business metrics (such as TPS, success rate, time taken) are delivered to the monitoring system.

As mentioned above, the administrative backend provides the developer with business development and maintenance capabilities, while the synchronization engine interprets and executes the choreographed business logic. Business students no longer need to follow the conventional process of “pull branch A, modify code A, test A and go online” for each business requirement. They only need simple drag and configuration, which greatly improves the efficiency of business delivery. At the same time, basic capabilities related to stability have been deposited and reused through platformization, which reduces maintenance costs while ensuring service stability.

Since its launch, iQiyi Magpie Bridge platform has carried nearly 20 data synchronization services, 30 + application instances and integrated 30 + business logic nodes. It provides stable support for the rapid iteration of related businesses. Later, we will transplant the synchronous business of the stock to this platform to reduce the maintenance cost.

Summary & Prospect

This paper mainly introduces the main functions of magpie Bridge platform. At present, Magpie Bridge reduces the traditional development time from hours to days to minutes, which greatly improves the development efficiency. At the same time, out of the box high availability to ensure business stability.

Subsequently, we considered to optimize the iteration in the following directions:

  1. Further provide automatic generation of data access layer service code, further reduce the development cost;

  2. Enabling deployment on private cloud platforms enables more business teams;

  3. Combined with the platform to form the company’s internal business data mart, to avoid repeated development between different business teams.

References:

**1.**The Rise of No/Low Code Software Development — No Experience Needed?

**2.**zhuanlan.zhihu.com/p/128581398

3. * * * * baike.baidu.com/item/%E8%BD…