[Architecture 101] Thinking shift from business to platform

One requirement: a lottery system

If you receive a development request: Develop a backend system for a lottery. The first reaction is to open IDEA, create a new project and enter the project name: Lottery. And then you start designing your database, your API, and then you develop it, which is a very common development process.

Some time later, you receive a similar request: develop another backend system for the lottery, but some of the business logic and process processing is not quite the same, and then copy the previous backend system and prepare to modify it.

It’s time to stop this seemingly efficient but actually stupid approach to development! This article will show you how to turn your mind around and give you a chance to transform your business, your systems, and even yourself. I would use a bit of back-end thinking, as much graphics as possible, as little code as possible, and as many examples as possible to elevate your level from a skilled business tamer to a beginner architecture hunter.

Reflection: What did I do wrong

The use of copy, paste, modify method to develop a similar new business system, from the development of a single system in terms of development efficiency is indeed quite high, but there are still the following problems:

• Low code reuse. There may be more and more similar business systems, and copy-and-paste leads to code with too much repetition and too little reusability
• The test and deployment efficiency is low. As a new background service, a set of testing, building, deploying, publishing process needs to be created and implemented. Some common business logic has been repeatedly tested. The building and publishing process also needs to be managed separately
• Collecting data is difficult. Data of similar service systems are scattered in different databases and logs. It is difficult to collect and analyze common service logs
• Business systems are too private to provide a SAAS equivalent that allows anyone to create their own sweepstakes

In short, mice have poor eyesight and cannot see things at a distance, but we need to avoid duplication of short-sighted development in order to reuse business systems in the future.

Therefore, it is necessary to abstract the business system into a business platform to support the general business process, so that the business platform can provide a flexible and changeable basic business implementation scheme for the similar business system in the future. As for the special business process, a special business system can be developed separately to provide complete services by combining the two.

In addition, the users of the platform should be SAAS like, with the ability to serve everyone, providing each user and even each organization with the ability to create their own unique environment and build their own individual services through simple configuration on the platform.

As shown in the figure, business system and business platform serve users together. A business platform contains a common process for a business that might just meet all the needs of the business, so there is no need to develop another business system. But if there are some more specific business processes or must create a business alone and convenient management control process, then can be used to combine with the business system and business platform, business system provides customizable interface, business platform provides a common interface, a business system can also be through authorized call interface to invoke the business platform access, modify the data.

General business

First of all, we need to think about how the design business logic can be more common from a platform perspective. How to distinguish whether the service logic is common and split the service?

generality

Purpose: To solve N possibilities of business implementation. Support as much common business logic as possible on the business platform.

How to solve: Taking the long view, what business logic does our business system need to support right now? What needs to be supported in the future? Then make selective trade-offs without thinking too much, or the system will be designed to be too complex and use too few parts. A simple and effective way to do this is to extract common logic from previous systems and current systems that need to be implemented.

A few examples:

• Business process considerations. For example, the last activity was only held for one day, and the prizes were the same, but this activity needs to be held for two times, and each prize may be different. Therefore, we can abstract out: a lucky draw may have multiple events, and each event may have different awards and time, which need to be modified at any time. Therefore, the activity is divided into two levels: the event and the event. Therefore, a field needs to be added to correspond to the dimension of the field in the database design
• Considerations on database table fields. The exchange of prizes may be through express delivery of physical prizes and direct display of exchange codes, so the types of table fields should be considered more, such as the need to save the user’s delivery address and contact information. The attribute selection of a single field should also consider some extensibility, such as field length and field type

Boundary processing, business separation

The boundary here refers to the boundary of the business split, for example, what belongs to the platform common logic? You can put that general logic into the platform. What is business specific logic? This logic does not need to be added to the platform.

For example, for the lottery, the allocation of prizes, distribution of prizes is the general logic of the lottery platform, but the user leaderboard is not universal, except for game-related scenes, rarely need to use the function of the leaderboard, then the function of the leaderboard should belong to the business specific logic.

In the whole business process of the lottery, there is another business process that when the number of lotteries is used up, the number of lotteries can be increased by sharing. Therefore, the general logic of lottery business can still be split. The sharing function does not belong to the general logic of lottery platform, so the sharing function can be separately extracted into a sharing platform, so that it can also serve other platforms or business systems.

It seems that the current situation has been quite clear, but it is not over yet, because this is only for the process of [sharing success -> adding a lucky draw opportunity], what if you need to share many times to add a chance? What if you add a different chance to draw each time you share? Who is responsible for maintaining the reward system? This requires an engine, process control center to deal with, better called task scheduling platform

The core business process of task scheduling platform is “completing the task – triggering the reward”. As for who completes the task and triggering the reward, it is not important to it. As long as it can maintain the corresponding relationship between the task trigger and the triggered, the trigger condition, and the way of invoking the reward interface of the triggered in the internal, when it detects that the trigger has reached the trigger condition, it can invoke the corresponding reward

The data processing

Large amount of data storage

Since it is a platform, because the user will use a long time, the amount of data will be more than the general specific business system, in the design of the need to take into account in advance for the book data volume of some table horizontal sub-table design, post a simple sub-table example:

Select * from shard key; select * from shard key; select * from shard key;

func GetTableNameByActivityId(activityId int64) string {
	if activityId > 0 {
		return "activity_prize_redeem_" + strconv.FormatInt(activityId % 16.10)}return "activity_prize_redeem"
}
Copy the code

When dividing tables, it is also necessary to consider which field (shard key) is used to divide tables. Data should be evenly distributed among multiple tables without affecting normal queries. In this way, data can be evenly distributed among different tables by dividing tables. However, if you use another field to query data, you may need to traverse all tables to query the data. Therefore, the choice of shard key is related to service query requirements and uniform data distribution.

After table splitting, the most direct impact is the need for dynamic adjustment of SQL statements during development. Some ORM frameworks do not support the development efficiency, but the code is not changed, once and for all.

High-performance Query

For a platform, concurrent traffic may also be large, so caching, queues, ES are essential.

For caching, it’s basically using Redis, and the clustering modes available are master/slave, sentinel, and cluster. The strategies used also include lazy loading, direct writing, and some breakdown and failure cache issues, as can be seen here

For message queues, it is a very important middleware in terms of business decoupling and traffic peak shaving, as you can see here

If full-text Search is required by the service, you can use this module together with EFK. For details about how to use this module, please refer to the following scenarios.

Operation burying point, report

Data is so important! To analyze user behavior, predict market trend, or be a reference index for future system design, we, as technical developers, also need to think for the operation personnel, we need to calculate some data that the operation personnel are interested in, and it is better to directly ask them in advance which data they need. In order to save some key data in some fields when we design the database.

Such as in the sweepstakes, operating personnel need to know the web platform, android, IOS participate in lucky draw one respectively how many respectively, but the design of database fields, in the code to print out the log can be does not exist or is difficult to through the existing design, then you need to separate the statistical requirements of design.

Log and operation record form

Logging in code is a must, often printing success or failure messages on access apis, key logic in an API, and key data.

In addition, some business scenarios require strict statistics on the situation before and after data operation, operation type, operator, and operation time. Although you can also output logs, but not standardized, query statistics difficult, easy to lose. Therefore, a separate operation flow table needs to be designed to persist important information.

After each significant operation that needs to be counted, use message queues or another thread to insert an operation record into the table, as shown below

go service.Record(&models.SysOperateRecord{
	BizType:     dao.BizType_Customer,
	OperType:    dao.OperType_INSERT,
	OperContent: fmt.Sprintf("batch insert one new customer: %s", customer),
	Operator:    userinfo.Id,
	CreateTime:  currentTime,
	UpdateTime:  currentTime,
})
Copy the code

Platform management

Saas-like independent environment

How can users have their own independent environment? So they can create their own apps after a simple configuration on the platform?

In fact, it is very simple, in only consider users and not tenants of the namespace, the system level and database design level can be considered, for example, add two fields on the main table/master entity of the database: namespace_id, app_id.

• Namespace_id represents a unique namespace owned by a user that is isolated from other users. All service systems and data created by a user are under the namespace_id. Therefore, the data of all service systems created by the user can be queried using namespace_id.
• App_id indicates the unique ID of a service system (application). So the data created by the user for a particular business system can be queried with app_id.

Unified management of multiple platforms

After the establishment of multiple platforms (lottery, sharing, task platform), a background management platform is needed to unify the management of these platforms, so as to provide unified configuration for users.

Take creating a service system as an example. If the management platform is used for configuration, users only need to fill in the corresponding service configurations of different platforms on the management platform.

1. The management platform creates a unified namespace_id and app_id
2. Write namespace_id, app_id, and the corresponding configurations of different platforms to the database configuration table of the corresponding platform through internal calls

You can modify or delete platform configurations and collect statistics on service data through the management platform configuration adjustment and statistics interface

Technology selection

Micro service

Why is it appropriate to develop a platform with a microservices architecture?

• Expansion of platform business. In the direction of the platform, the business logic will be more and more complex, the amount of code will be more and more huge, for the independent deployment of the service test, adjustment, service separation is inevitable.
• Unified platform management. Multiple services can be centrally managed through components such as an API gateway, a configuration center, a service discovery registry, and fuses. For example, add an authentication module to the gateway, collect all request logs, centrally manage service configuration files in the configuration center, and manage and configure different services in fuses. This is all about unifying disparate services through related components.

Business decision Architecture

How does the business determine architecture?

• Concurrency and stability of services. If the amount of concurrency is a business requirement, a highly available architecture that can support a large number of requests is required. For example, the multi-node load balancing of each service node, the choice of using MySQL or MongoDB for database, the separation of master and slave copy read and write for database architecture, the choice of middleware, front-end cache and CDN technology
• Third party support for business. For example, it is difficult to implement push and search functions by themselves, and third-party services are generally used. The combination of third-party services and their own services is also reflected in the architecture
• Internal interactions triggered by business requirements. Such as the general application of the registration function, will use the message queue middleware to write database behavior and other activities (SMS, email) decoupled; It is also possible to design timed tasks that periodically fetch updates from some data sources to another data source

Here is another example article that reflects the idea of business decision architecture

performance

If performance is the only consideration, then microservices architecture should not be chosen because individual applications completely overwhelm microservices architecture in terms of performance. If you still have to choose microservices for global reasons, how can you maximize access performance while using microservices?

The first step is to test performance bottlenecks by pressure testing. This article may help.

At the very least, you need to ensure that performance problems can be resolved immediately after they occur, so at the very least, the service can be versioned back at any time and dynamically scaled to more machines. We then address performance issues by optimizing code, using caches, queues, adding servers, and improving the middleware architecture.

Deployment way

For an architect, the code needs to be taken into account from delivery to live, and even the DevOps architecture needs to be built for the team. To put it simply, the automation pipeline was used to automate code submission, testing, construction and deployment, and to assist the development and test operation and maintenance in code version switching and deployment practices during the project launch.

Container technology such as Docker and container choreography platform such as K8S must be used to achieve blue-green deployment, rollback and dynamic scaling of services in the future. It is not necessary to master the specific construction and maintenance details, but it is important to know the deployment and service maintenance modes and their advantages and disadvantages.

Safety considerations

Business security

For a lottery system, the prize is tied directly to money because it has real value. In order to prevent malicious users from jerking off, the possible existence of the lottery back door must be blocked.

The biggest backdoor to a lottery system is the winning interface. How do you tell if the user actually played the game and made it all the way to the end to win the reward? For the Web platform is really a more difficult problem, because there is no real-time access to user data like the game for detailed judgment, so we can only increase the verification rules of the winning interface to the maximum extent. As follows:

• Game logic validation. According to the front end design, first test out the fastest time to win the prize, and then add judgment in the background: if the time is less than the fastest time, block. In addition, it can be judged according to the business logic of the game: the winning interface can be successfully invoked only by clicking [Agree agreement] and [Start game] buttons (access these two interfaces).
• Verification code. To prevent malicious users from automatically brushing scripts, add verification code to the winning interface. But this greatly affects the use of the experience, use caution.
• Blacklist control. For a user that calls the interface repeatedly, the user ID is added to the blacklist and the interface access is prohibited for a period of time.

Site safety

These are some of the more general Web security issues that are generally resolved within the framework, but you also need to verify that the relevant security mechanisms are enabled.

• SQL injection. You don’t need to worry about this if you are using the ORM framework, because the framework already uses escape to handle special characters that affect SQL statements. If complex SQL manual concatenation is involved, you must concatenate strings using placeholders at development time to solve the problem
• CSRF. For example, Beego framework of Go language, Ruby On Rails framework of Ruby language, And Spring Security of Java language have provided solutions, but they are not enabled by default and need unified configuration at the front and back ends to take effect. After the back-end configuration, A CSRF token is stored in the session and waited for the next request to be passed to the front end for verification. Therefore, CSRF defense needs to be added for certain operations with high security
• XSS. There may be no ready-made solutions in the framework, so it is necessary to take precautions at the front and back ends. For example, in the front-end JS, it is necessary to avoid the execution of untrusted data as code, and in the back end, special filters need to be configured. In the filter, escape tools will affect the execution of JS special characters to escape
• HTTPS. Prevent man-in-the-middle attacks, improve site security, search weight, everyone knows, no more details
• Permission control. To prevent common users from having similar permissions to administrators, you need to restrict the permissions on the backend
• Password management. Even if the database is cracked, you can’t let others see your password in clear text. For the password that does not need to be decrypted, such as the user login password, use bcrypt algorithm or MD5 salt encryption. Use the AES or symmetric encryption algorithm to encrypt and save the passwords to be decrypted. Save the keys to be decrypted separately

And so on, website security problems are in too many, can’t write out, but the above problems are more common, need to pay attention to in the development, although these will not be shown on the architecture design, but will be an existing architecture hidden trouble

conclusion

We comprehensively consider how to design a simple business into a platform from the aspects of business, data, technology and security, focusing on long-term benefits and improving our own capabilities. As a developer, you may be tired of writing business code, so challenge yourself by thinking differently.

Welcome to follow the program meow kadun wechat official account: program meow Kadun, get the latest free information, consultation ~