This article is created by Li Qiankun from Ximalaya technology team. The original title is “Push system Practice”. Thanks for the author’s selfless sharing.

1, the introduction

1.1 What is Offline Notification Push

Offline notification push is a familiar requirement for IM developers, as shown in the following figure.

1.2 Offline push on The Android terminal is really difficult

Offline notification push on mobile devices involves only two terminals — iOS and Android. There is nothing to say on iOS, and APNs is the only option.

Android terminal is quite strange (mainly refers to the domestic mobile phone), in order to achieve offline push, all kinds of live black technology emerge in an endless stream, with the continuous upgrade of live difficulty, can use live means is less and less, you can read the following articles I organized, feel (the article is in chronological order, With the increase of the difficulty of keeping The Android system alive, continuously advanced).

• “Application survival (1) : dual-process daemon survival practices under Android6.0”
• Android6.0 and above (process prevention)
• Android6.0 and above survival practices (killed and resurrected)
• Android P is coming soon: The Real Nightmare of Backend Apps and Notifications
• A Comprehensive Review of the real Operation Effect of the Current Android Background Preservation Scheme (by 2019)
• In 2020, Android backstage survival is still a play? See how I elegantly implement!
• “The most powerful Android Survival ideas: In-depth Analysis of Tencent TIM Process Immortality Technology”
• “Android Process Immortality Technology Ultimate Reveal: Process killing basic principles, APP Coping skills”
• “Android from Start to Quit: Guide users to add white list (with 7 big models add white example)”

These are just a few of the articles I’ve put together on this topic, but pay special attention to the last one: How to whitelist Android from Starter to Quit. Yes, the current Android system has almost zero tolerance for APP self-preservation, so almost all of those existing methods of preservation will be disabled in the new version of the system.

It’s no longer possible to do your own job, so you still have to do offline notifications. How to do? According to current best practice, this is a room-level push channel for mobile phone manufacturers. I won’t go into details here, but if you’re interested, you can read more about Android P coming soon: The Real Nightmare of Backend App survival and Push notifications.

In the era of self-maintenance and self-built push channel (here of course, it refers to the Android terminal), the architecture design of offline message push system is relatively simple, nothing more than each terminal calculates a deviceID, and the server carries out message transparent transmission through the self-built channel, that’s all.

And in the self-built channel dead, can only rely on manufacturers push channel now, Xiaomi, Huawei, Meizu, OPPO, Vivo (just a few mainstream) and so on, there are too many mobile phone models, each push API, design specifications are different (don’t tell me what unified push alliance, I’ve been waiting for him for three years — see the “Unified Push Alliance” for details), which led to previous offline push system architectures having to be redesigned to fit the new age of push technology.

1.3 How to design reasonably

So, for ROOM level push channels of different manufacturers, how should we design the background push architecture reasonably?

The offline message push system design shared in this paper is not specifically for IM products, but no matter how different the business layer is, the general technical ideas are the same. I hope this sharing by Ximalaya can bring some inspiration to you who are designing offline message push with a large number of users.

*** Recommended reading: ** Another article “Long Connection Gateway Technology Topic (5) : Himalaya self-developed 100 million level API Gateway Technology Practice” shared by Himalaya technical team, you can also read it together.

2. Technical background

First of all, I would like to introduce the role of push system in Ximalaya APP. The picture below is the push/notification of a news business.

Offline push is mainly a means to reach users when they do not open the APP, so as to maintain the presence of the APP and improve the daily activity of the APP.

At present, our main businesses using push include:

• 1) Anchor broadcast: The company has live broadcast business, and the anchor will send a push broadcast reminder to all the fans of the anchor when opening live broadcast
• 2) Album update: There are a lot of albums on the platform, and a series of specific sounds are below the album. For example, a novel is an album, and the novel has many chapters. When the chapter of the novel is updated, a reminder will be sent to all users who subscribe to this album:
• 3) Personalization, news business, etc.

If you want to send an offline push to a user, the system needs to have a channel with that user’s device.

** who has done this knows that ** self-built push channel requires App resident in the background (i.e. the application “survival” mentioned in the introduction), while mobile phone manufacturers generally adopt “radical” background process management strategy due to power saving and other reasons, resulting in poor quality of self-built channel. Currently, channels are generally maintained by “push service providers”, which means that the company’s push system does not directly send notifications to users (as mentioned in this article from the previous section: Android P official release coming soon: App survival in the background, real Nightmare of push).

In this case, the offline push flow is as follows:

Major domestic manufacturers (millet,huawei,meizu,OPPO,vivoEtc.) have their own official push channel, but each interface is different, so some manufacturers such as Xiaomi, Getui provide integrated interface. When sending, the system is pushed to the integrator, and then the integrator sends the push channel to the specific manufacturer according to the specific equipment, and finally sends it to the user.

When sending a push to a device, it’s important to specify what you’re sending: title, message/body, and which device you’re sending it to.

A token is used to identify a device. In different scenarios, the meaning of a token is different. A device is usually identified by a UID or deviceId inside the company. Be responsible for converting UID and deviceId to integrator token.

3. Overall architecture design

As shown in the figure above, the push system as a whole is a queue-based streaming system.

** On the right side of the figure above: ** is the main link. Each business side sends push to the push system through the push interface. The push interface will send data to a queue for consumption by conversion and filtering services. Conversion is the uid/deviceId to token conversion described above. Filtering will be discussed below. After conversion and filtering, it is sent to the sending module and finally to the integrator interface.

**App startup: ** will send a binding request to the server and report the binding relationship between uid/deviceId and token. When the token is invalid due to uninstallation/reinstallation of the App, the integrator notifies the push system through HTTP callback. Each component sends a stream through Kafka to the company’s XStream real-time stream processing cluster, aggregates the data and dumps it to mysql, where grafana provides a presentation of reports.

4. Design of service filtering mechanism

Each business side can mindlessly send push to users, but the push system should be restrained, so the business message should be selectively filtered.

The design of the filtering mechanism includes the following (in order of support) :

• 1) User switch: The App supports user switch configuration. If the user turns off push, push will not be sent to the user’s device.
• 2) Copy scheduling: a user cannot receive duplicate copy, which is used to prevent the upstream business from sending logic errors;
• 3) Frequency control: each business corresponds to one MSg_type, setting the maximum number of XX pushes within XX time;
• 4) Silent time: no push will be sent to users from XX to XX every day, so as not to disturb the rest of users.
• 5) Hierarchical management: hierarchical control is carried out from the user and message dimensions.

For point 5, specifically:

• 1) Each MSG /msg_type has a level to provide more opportunities for important/higher-level services to be sent;
• 2) When users receive XX push messages a day, non-important messages are no longer sent to these users.

5. Multi-dimensional query problems under sub-database sub-table

Most of the time, design is based on theory and experience, but in practice, there will always be a variety of specific problems.

Ximalaya now has 600 million + users, and the device table of the corresponding push system (which records the mapping of UID /deviceId to token) also has a similar level of magnitude. Therefore, the device table is divided into database and sub-table, with deviceId as sub-table.

** But in reality: ** often has a query requirement based on UID /token, so you also need to establish a mapping relationship based on UID /token to deviceId. Because uid query scenarios are also frequent, the secondary UID table also has the same fields as the primary table.

Because there are global push once or twice a day, and special push for silent users (i.e. those who do not use the APP often), there is no “hot spot” in storage. Although caching is used, its effect is limited and it takes up a lot of space.

Multiple tables and caches result in three or four copies of data, different logic uses different copies, frequent inconsistencies (pursuing consistency affects performance), and very complex query code with low performance.

Finally, we chose to store the device data on TIDB, which greatly simplified the code on the premise of sufficient performance.

6. Timeliness of special business

6.1 Basic Concepts

Push systems are queue-based, “first come, first push”. Most businesses do not require high real-time performance, but the live broadcast business requires delivery in half an hour, and the news business is “unsatisfied”, the faster the better.

** If a huge amount of “album update” push is waiting to be processed in the queue for news push, the album update service will seriously interfere with the delivery of news service.

6.2 Is this an isolation problem?

** Initially we thought of this as an isolation issue: ** For example, 10 consumer nodes, 3 dedicated to time-sensitive services and 7 dedicated to general services. The queues were using RabbitMQ and spring-Rabbit was adapted to support routing messages to specific nodes based on mSYType.

The scheme has the following disadvantages:

• 1) Some machines are always busy while others are “sitting on their hands”;
• 2) When new services are added, the mapping relationship between msgType and consumer node needs to be configured, which costs a lot to maintain.
• 3) RabbitMQ is based on memory, which takes up a large amount of memory at peak push times, resulting in rabbitMQ instability.

6.3 is really a matter of priorities

Later we realized that this was a priority issue: high-priority businesses/messages could cut the queue, so encapsulation Kafka supports priority, which is a better solution to the problem of isolated solutions. The implementation is to create multiple topics, each representing a priority, and encapsulate Kafka primarily to encapsulate the logic on the consumer side (that is, to construct a PriorityConsumer).

Consumer. Poll (num); consumer. Poll (num); consumer.

There are three options for the PriorityConsumer implementation, described below.

1) Poll to memory to reorder:

Java has an existing memory-based PriorityQueue, PriorityQueue or PriorityBlockingQueue, which kafka consumers consume normally and push polled data back to the PriorityQueue.

• 1.1) If a bounded queue is used, after the queue is full, the following messages cannot be put into the queue no matter how high their priorities are, and the “queue jumping” effect is lost.
• 1.2) If you use unbounded queues, messages that should be heaped on Kafka will be heaped into memory. OOM risk is very high.

2) First pull the data of high-priority topics:

As long as there is consumption, until there is no data consumption of a lower level of topic. In the process of consuming lower-level topics, if a higher-level topic message is detected, the higher-priority message will be consumed.

The implementation of this solution is complicated, and low-priority services may completely lose the opportunity to push in the busy time period such as the evening rush hour.

3) From high priority to low, data is pulled in a cycle:

The logic of a loop is:

consumer-1.poll(topic1-num);

cosumer-i.poll(topic-i-num);

consumer-max.priority.poll(topic-max.priority-num)

If topic1-num=topic-i-num= topic-max-priority-num, the scheme has no priority effect. Topic1-num can be regarded as the weight, we agreed: topic-high-num =2 * topic-low-num, all topics will be consumed at the same time, through the number of consumption in a way to achieve the “queue-jumping effect”. In detail, the “sliding window” strategy is also used to optimize the overall consumption performance of a priority topic when there is no message for a long time.

We can see that the timeliness problem is first understood as an isolation problem, then regarded as a priority problem, and finally transformed into a weight problem.

7. Storage and performance problems of filtering mechanism

In our architecture, the speed of push sending is mainly affected by tiDB query and filtering logic, and filtering mechanism is divided into storage and performance issues.

Here, we take xx service frequency control limit of “maximum one transmission per hour” as an example for analysis.

** First version implementation: ** REDis KV structure is <deviceId_msgtype, number of sent push >.

Frequency control logic is as follows:

• 1) When sending, incr key, the number of sending increases by 1;
• 2) If the limit is exceeded (incr command return value > the upper limit of sending times), it will not be pushed;
• 3) If the limit is not exceeded and the return value is 1, it means that the expiration time (equal to the frequency control period) needs to be set by the expire key for sending messages to the deviceId for the first time within the MSgType frequency control period.

The above scheme has the following disadvantages:

• 1) At present, the company has 60+ push services, 600 million + deviceId, a total of 600 million *60 keys, occupying a huge space;
• 2) Many times, processing a deviceId requires two directives: INCR + EXPIRE.

To this end, our solution is:

• 1) Replace REDIS with PIKA (disk-based REDis), and the disk space can meet the storage requirements;
• 2) The entrusted system architecture group extends the Redis protocol to support the new structure EHash.

Ehash is a two-level map <key,field,value> based on redis Hash. In addition to keys,field also supports validity periods and conditional validity periods.

The storage structure of frequency control data changes from <deviceId_msgtype,value> to <deviceId, msgType,value>. In this way, for multiple MSgTypes, deviceId can be saved only once, saving space.

Incr and EXPIRE are combined into one instruction: INCR (Key,filed,expire), which reduces network communication.

• 1) When field does not set the validity period, the validity period is set for it;
• 2) If field has not expired, the validity period parameter is ignored.

Since the push system heavily uses INCR instruction, it can be regarded as a write instruction. Pipeline is also used in most scenes to achieve the effect of batch write. We entrusted our partner in the system architecture team to optimize the write performance of PIKA, supporting “write mode” (optimizing related parameters in the write scene), and the QPS reached more than 10W.

The ehash structure also plays an important role in stream logging, such as

, where 100001002 is an example value of the data format we agreed on, The first, middle, and last three parts (each with three bits) represent the details about the sending, receiving, and clicking of a message (msgId) against deviceId. For example, the first three bits 100 indicate that the message fails to be sent because it is in the silent period. (This article is simultaneously published at: www.52im.net/thread-3621…)
,msgid,100001002>