Engaged in the service end work, has been a few years, from ignorant chicken, grow into a bald eagle can fly freely, those lost youth and hair witness their growth

Perhaps, this is the master should look like it

Here, relatively common parts of similar problems/business scenarios will be extracted and sorted out as experience for mutual encouragement

Power etc.

Business scenarios:

If the user clicks the button for several times, or if there is no response after clicking the button due to the performance of the device or the connected network, the user will continue to try to click the button, triggering multiple request submission

Solution:

Anti-key click: The client can only click once to prevent key click. The control button cannot be clicked again until the response result is returned

Server:

1. The table constraints

Unique constraints on table design fields, such as: INSERT log-in record, then UPDATE integral parallel execution, must have only one INSERT success, all the other failure, will eventually only accumulate one integral

2. Distributed lock

Distributed lock constraints can be implemented by taking advantage of redis INCR atomic operations

Before operating services, obtain the INCR of the user ID. If the value =1 is obtained, the lock is successfully obtained. Perform atomic operations, and then execute the service logic

If the value is greater than 1, the execution fails, indicating that the execution is not complete and the lock is not released. Therefore, the execution fails

To avoid network jitter or service execution errors, you need to run the INCr command to obtain the lock and obtain the TTL value. If the TTL is not set, you need to set the TTL for the key. After the TTL is set, the key automatically expires. Lead to a deadlock

3. The token mechanism

Obtain the token before the operation. The token can be used only once. Before executing the service logic, you need to update the token usage status

You can use the mysql token table +redis list as a token bucket, the required services from the queue pop token, update the status of the token table

Master-slave delay

Business scenarios:

Users reported that they could not see the data just submitted or did not update successfully, or triggered the logic that could be understood by the normal process. After investigation, the data was found to be normal

Solution:

Speaking of master-slave delay, we should be familiar with, as long as the database (mysql, Redis) deployment is master-slave separation, more or less will encounter this problem

In low-probability scenarios, data is written/updated to the master database, but the slave database is not synchronized in time due to network jitter and other reasons, and then the slave database is queried, resulting in dirty data. In this case, the server environment can only be stable

In fact, this problem is more likely to occur after the insert/update to the master database, immediately query data, this time may not be synchronized to the slave database, although the master/slave synchronization will be faster, but there is still a certain delay

In this case, the query needs to be assigned to the master library for operation, which can avoid the problem of master/slave delay and not being able to query the latest data

concurrent

Business scenarios:

Users can click the button quickly or write scripts to send concurrent requests to multiple servers. Multiple servers receive the requests at the same time. Multiple/concurrent requests may be executed in parallel, causing problems beyond the normal logical range

Often in this case, there will be a lot of abnormal data, such as: multiple check-ins on the same day, and multiple accumulative bonus points

Professional fleecers use tools or scripts to maliciously initiate concurrent request interfaces, double profits and cash out, and profit from the vulnerability

Solution:

Table constraints are idempotent solutions to ↑

Security privacy

Business scenarios:

When it comes to user privacy data or some sensitive commercial data business, the interface does not desensitize the data when it delivers the data, exposing the user’s privacy data nakedly, such as the user’s mobile phone number, ID card number and other important information directly in plaintext and the interface output

The user ID as a picture name, can easily traverse the user uploaded pictures, ID photos, etc., used for illegal purposes

Solution:

Data desensitization:

Under the condition of not violating the system rules, the real data can be transformed to desensitize the data

  • Modify sensitive data output according to rules
    • In the middle of star
    • truncation
    • replace
    • .
  • Sensitive data transfer, encryption processing
    • Aes encryption
    • Hashids are short, unpredictable, and unique numeric ids
    • .

CND media Address security:

Most CDN platforms support it

  • The URL to
    • Prevents address construction by rules
  • Preventing hotlinking
    • The address is inaccessible after the expiration time
  • Limit access to
    • Referer hotlinking prevention
    • UserAgent Blacklist and whitelist
    • IP blacklist and whitelist
    • Etc. (see the third support)

MQ business decoupling wizard

Asynchronous service decoupling

Business scenarios:

For example, after the order is placed and settled successfully, push notifications will be sent, coupons will be issued, asynchronous business tasks will be operated, and users will be notified to claim, etc

Similar to this non-business main process, the main process can immediately return a response to the user upon completion, and other successful additional operations are queued to MQ for asynchronous processing

MQ can also be used for cross-process, cross-language messaging

Multiple subscriptions facilitate business development, ACK mechanism to ensure the completion of execution, dead letter queue, fault tolerance processing

Different MQ middleware support slightly different features and advantages. You can choose an appropriate middleware based on your own requirements

  • rabbitmq
  • kafka
  • rocketmq
  • .

Buffer solution

In high concurrency scenarios, hot data can be cached to reduce DB pressure and improve response speed

Caches can be divided into server caches and client caches

Server side cache:

Redis is currently the most widely used distributed memory cache database, which can meet most of the requirements combined with the supported data types and features, coupled with the creativity of development

But in the process of use will also encounter some improper use of the problem, here is a list of common problems:

1. Cache updates

For some private data, it is generally used to del cache during data update, and then obtain the subsequent data from the cache. If the data does not exist, it is then obtained from the DB ->cache, and finally output to the user

However, due to network jitter, del cache may fail in a low probability. Therefore, we usually add an expiration time to the cache to make dirty data invalid in a short period of time. In this way, we can also expiate some infrequently queried data

For some common thermal data, such as commodity list, etc., operating personnel by the background configuration items, the configuration is completed, the cache update operation, this time need to smooth over the cache update, can’t delete the key first, then write to the cache, this operation will lead to a user in the cache update in before, can’t get goods in a short time interval

We have done similar requirements before. The solution is to design the version number rule in the created cache key, and then after the cache is created successfully, replace the version number that can be displayed, and set the expiration time of the old version number data

Data of an earlier version cannot be deleted immediately. A proper expiration time is set because data of an earlier version will be used within a short period of time. For example, if a user queries data of an earlier version and continues to perform paging query, the expiration time can be set to clear up unused data within a reasonable period of time

Data acquisition is to obtain the current version number to be displayed first, and then obtain the corresponding data of this number

A similar, more complex cache update scenario was written earlier, with high concurrency business interface development ideas

2. Through

There is no data in the cache or db. After reading the cache, there is no data in the DB

In general, null data problems are caused by null data problems. The solution is to cache null data to avoid penetrating DB data. If there are many NULL data, bitmaps Blom filter can be used to identify null data to save storage space

3. Breakdown, avalanche

A large amount of cache data expires at the same time, causing a large number of requests to be sent to db at the same time

For the cache of hot data of high concurrent services, the expiration time cannot be deleted or set, and the update can only be carried out smoothly, similar to the scheme mentioned in the cache update above

4. Compress data, and the data expires

Redis cache uses memory space, so it is relatively scarce. Even if there are many machines with deep pockets distributed, they can not afford random ho-ho

Do not store unused fields or data in the cache. Sometimes, for convenience, the entire object is serialized directly with JSON and cached directly

For the user’s private cache or cache with low popularity, it is necessary to set the expiration time of the cache to avoid the accumulation of garbage data that is not queried for a long time, occupying space, and then the bottleneck encountered later

Client cache:

1. Cache version data

The client caches data + data version number and uploads the data version number parameter every time it obtains data. The server verifies whether the data is the latest. If the data is the latest, the server does not deliver the data and the client can continue to use the local data

2. Incremental pull update

When the server interface returns data, it returns the current timestamp, and the client caches the data and the pull timestamp. When the client requests to bring the timestamp, the server matches the data with the update time > the timestamp time and delivers the data to realize the incremental/modified update of the client data

Redis use opportunely

  • A distributed lock
    • Write a mention
  • Limit frequency
    • Redis actual combat limited operating frequency
  • Timer queue
    • Redis actual combat to achieve a timed execution of the task

Log/Monitoring

About logs:

When online users report problems, we need to troubleshoot the problems. Sometimes it is difficult to troubleshoot the fundamental problems based on the user’s descriptions and APP screenshots

At this time, if you can provide the user’s request log track, it can help to troubleshoot

We currently support the log piece has two, one is nginx request log, through elK to build a log system, log collection and display

At the same time, The System backs up a long time request log. If the request log history is too long, you can query the table in the system

Common error logs can also be reported to the ELK to create an independent Err group for easy query

  • ELK
  • Number plus history request log

About monitoring:

Monitoring can be divided into server monitoring and service function monitoring

Online server stability is an important factor determining the stability of business functions, which is mainly guaranteed by operation and maintenance

Under the business functions of monitoring, in addition to the occasional error log, repair the abnormal situation, also need for some business functions of monitoring, such as: some regular service, timing of delivery, need to remind users not recorded the hour every day to push, framed the user’s need to ensure efficiency and speed of delivery, security content push out within the prescribed time

With the growth of business and the continuous increase of data, the execution that was completed in an hour may be continuously extended, and finally may be completed in a whole day. For this kind of business, it is necessary to get the problem before the user feedback, and then optimize and improve it

At this time, it is necessary to have a monitoring function to monitor the business functions and give early warning notification beyond the early warning to improve the problem as soon as possible

conclusion

During the years of server-side development, I have participated in several projects related to e-commerce and tools, etc. Due to the technical background of the project and the need for technical improvement, I have also dabled in several development languages, including.NET (project), Java (project), NodeJS (project), Python (acquisition, crawler), etc. PHP (job transfer project), Golang (micro services), not to mention how proficient each language is, the general business development is not much problem

In fact, language is a tool to realize business requirements, just like hoe and pick, sickle and firewood knife, kitchen knife and knife. The basic use mode is similar, but the advantages are different in different demand scenarios, and appropriate tools are used in appropriate scenarios

Working on so many projects and languages, you will find that the experience on the server side is universal, independent of language and project, just the ideas and solutions to solve some problems and business scenarios

It is possible to sort out and summarize the solutions of the businesses/problems I have participated in in a certain period of time, so as to extract the solutions of common scenarios into my own experience, otherwise I will forget much of the content I have done after a long time

First published on Github: 🌱 “Big Talk WEB development”, WEB development related experience summary share, welcome everyone Star wave, follow-up want to see not lost 🌈