(Hornet’s Nest Technology original content, public ID: MFWtech)

Mobile Internet technology has changed the world of travel, reducing the previously heavy cost of information distribution. The communication paths between users and service providers and between users are gradually opening up, and the communication scenarios are also constantly expanding. This forces all mobile app developers to take a user’s perspective and better meet their needs.

In the hornet’s nest era, the communication form between users is relatively simple, mainly simple reply and so on. In order to meet the needs of users quickly with a small cost, non-real-time message scheme was adopted to realize the message delivery between users.

With the development of the industry and the company, Hornet’s Nest has established a unique business model of “content + transaction”. Under the background of increasing user scale and changing business form, providing stable and reliable pre-sales and after-sales technical support for users and businesses has become a top priority for e-commerce mobile business lines.

I. Design ideas and overall architecture

We combined with B2C, C2B, C2C different business scenarios design to achieve hornet’s nest travel mobile terminal private message, user consultation, user feedback and other instant messaging services; At the same time, in order to better empower cooperative merchants, the mobile terminal of hornet’s Nest merchants is added with session-related consulting user management, customer service management, operation resource statistics and other functions.

Currently, IM involves the following services:

In order to realize the integration reuse and UI customization of Hornet’s Nest tourism App, IM business logic of merchants and public resources, the problem is broken down into the following parts to solve:

1. IM data channel and abnormal reconnection mechanism to ensure real-time message delivery and stability of different services;

2. IM real-time message subscription and distribution mechanism solves message directional sending and business subscription consumption, avoiding unnecessary request resource waste;

3. IM session list UI drawing universal solution to solve the complex problems of rapid iterative development and management of different message types;

The overall implementation structure is divided into four parts for encapsulation, which are data management, message registration and distribution management, general UI encapsulation and business management as shown in the figure below.

Two, the technical principle and realization process

2.1 Universal Data Channel

For the acquisition of routine business display data, the client needs to initiate the request actively. The process of request and response is one-way, and the real-time requirement is not high. However, FOR IM messages, both receiving and sending operations must be supported and real-time requirements are high. To meet this requirement, a stable data channel must be created between the client and server to provide bidirectional data communication between the client and server.

2.1.1 Basic Interaction principles of Data Channels

In order to improve the scalability of service support by data channels, we encapsulate all communication data into packets with the same outer structure, so that data of multiple service types can use the same data channel for communication and unified distribution and processing, thus reducing the number of channels to be created and the maintenance cost of data channels.

The common client-server data interaction depends on the HTTP request response process, and only the client initiates the request can get the response result. In combination with the specific business scenario of hornet’s nest, we hope to establish a reliable message channel to ensure that the server informs the client actively and realizes the transmission of business data. At present, it is implemented in the form of HTTP long-link polling. All business data message types only need to follow the agreed general data structure to be sent to the client through the data channel. The data channel does not care about the specific content of the data, only needs to be concerned with receiving and sending.

2.1.2 Implementation principles of client Data Channel

The core of client data channel management is to maintain a service scenario request stack, and to push different service scenario parameter data in the process of switching between different service scenarios. Each HTTP long link request uses the top of the stack request data, which can simulate different processing in specific business scenarios, such as with different user messages. The data correlation processing is encapsulated in the data channel management. The business layer only needs to register the corresponding receiving processing in the data channel management to get the required business message data.

2.2 Message subscription and distribution

In software systems, subscription distribution is essentially a messaging pattern. The party that does not directly deliver the message is called the “publisher,” and the party that receives the message is called the “subscriber.” The publisher classifies different messages and distributes them to subscribers of the corresponding type to complete message delivery. The advantage of subscription distribution is that different interceptors can be added to handle message parsing, message filtering, exception handling, and data collection for unified management.

2.2.1 Message Subscription

The business layer focuses only on message processing and does not care about the process of receiving and distributing messages. The significance of subscription is to better understand the decoupling of business processes and data channels. The business layer only needs to subscribe to the message type of interest and passively wait to receive the message.

The business layer subscription will automatically monitor the life cycle of the current page after registration, and delete the corresponding message subscription after the page is destroyed, so as to avoid manually writing pairs of subscriptions and unsubscriptions, reduce the coupling of the business layer, and simplify the call logic. Subscription distribution management maintains the distribution operations of subscriber queues for message reception based on the business type.

2.2.2 Message Distribution

The core of the data channel is to maintain a set of subscribers corresponding to multiple message types and distribute parsed messages to the business layer.

The data channel is shared by multiple service messages. After each request receives a new message list, it is re-divided into multiple message lists according to their respective service types, which are distributed to the subscription processor corresponding to each service type and finally transmitted to the service layer and handed over to the corresponding page for processing and display.

2.3 Drawing session Message list

Based on different scenarios, such as private messages based on social interaction, consultation and feedback based on user service, etc., all need the display form of session list; However, different scenarios are different. Therefore, you need to analyze the commonality of the current session list and the parts that can be encapsulated and reused to better support the expansion of subsequent services.

2.3.1 Composition structure of messages displayed in the list

The IM message list is characterized by multiple message types and diversified UI displays. Therefore, you need to establish a mapping relationship between each type of message and its layout. After receiving the message, you need to match the corresponding layout according to the message type and add the message to the corresponding message list.

2.3.2 Message types and display layout management principles

For different message types and presentations, the core of the problem is to establish the mapping between message types, message data structures, and message presentation layout management. The above three are maintained through the establishment of mapping management tables in the implementation process, respectively, the establishment of list storage message type/message body encapsulation structure/message display layout management, and the setting of corresponding relationship to associate three lists to complete the search.

2.3.3 UI drawing process of sending and receiving Messages once

Each type of message has a different content presentation, but the overall session message presentation style can be divided into three types: receive message, send message, and middle of the page. The difference is only in the internal message style. So the drawing of the message UI can be broken down into two steps, first creating a generic presentation container, and then filling in the specific presentation styles for each message.

The purpose of the split is to focus only on unique data for each type of message UI processing. Common messages such as profile picture, name, message time, whether to report, read or unread status, and send failure/retry status can be processed in a unified manner, reducing the cost of modification and maintenance. In addition, the UI processing logic of each message is less and clearer, facilitating the expansion management of new types.

After receive and send to the message, according to message type judgment is “send and receive type” or “center display type”, find the layout of the outer style, according to the specific message types find UI style, splicing in outer layout, get the complete message card, and then set the corresponding data rendering to the list, complete the message map.

Third, details optimization & trample pit experience

In the process of implementing the above IM system, we encountered a lot of problems and did a lot of detail optimization. Here’s a summary of some implementation considerations for your reference.

3.1 Message deduplication

In the previous schema, we used msg_id to mark each message in the message list, which is generated by storing data uploaded by the client.

After client A requests the IM server, the MSg_ID is generated and then distributed to client A and client B using the request return and Polling. When the process is established, clients A and B perform local de-duplication using the msg_id distributed by the server. However, this scheme has the following problems:

When client A fails to receive the message request due to A network fault, the retransmission mechanism is triggered. At this time, although the IM server has received the message sending request from client A once, it cannot determine whether the two requests are from the same original message, so it has to accept the request again, resulting in the generation of repeated messages. The solution is to introduce client message id. Because we’ve done a lot of work with the old msg_id, we don’t want the client message ID to take over the msg_id function, so redefine a random_id.

Random_id = random + time_stamp. Random_id identifies the unique body of the message, which is generated by a random number and the timestamp used to generate the body. When a retry is triggered, the random_id of both requests will be the same, and the server can de-iterate the message based on this field.

3.2 Localized Push

When we are in the context of a session page or list page, we can intuitively observe the changes in the interface to receive new messages and update the unread. However, after exiting from the session or list page, this information cannot be retrieved from the interface, and other mechanisms are needed to keep the user informed of the current message status.

System push and third-party push are a viable option, but push is essentially a long link based service. In order to remedy the instability and risk of push, we adopt the form of data channel + local notification to perfect the message notification mechanism. If the message delivered through the data channel needs to achieve the prompt effect of Push, it carries the corresponding Push display data. At the same time, the system determines the current page to avoid repeated reminders of the message content on the current page.

The mechanism of data channel + local notification display can improve message arrival rate, reduce the dependence on remote push, reduce the pressure on push system, and improve user experience during the running time of the application.

3.3 Abnormal Reconnection mechanism of data Channels

The current data channel is implemented using HTTP long-link Polling. The following figure shows the impact of Polling in different business scenarios:

As the network request states of users’ mobile phones are different, they sometimes encounter network interruption or server abnormalities, thus terminating the request for Polling. To enable users to continue session services after the network is recovered, the reconnection mechanism needs to be introduced.

In version 1.0 of the retry mechanism, the delayed retry limit of five consecutive error reports within 60 seconds was added to the case that many retry requests might occur. The specific process is as follows:

The following problems were found in practice:

• If a server exception persists for more than 1 minute, the client starts the retry mechanism and resends the reconnection request every 1 minute. This is the equivalent of a short, concentrated “attack” on the server, and can even bring it down.

• It does not make sense to try again immediately after the client is disconnected from the network, because it takes a certain amount of time for the user to recover the network, during which the reconnection request is meaningless.

Based on the above analysis and improvement, we design the second version of retry mechanism. This time, the delay time of the following 5 request errors is changed to 5-20 seconds random retry, and the client retry requests are scattered at multiple time points to avoid the instantaneous pressure of the simultaneous requests to the server. At the same time, delayed retry is performed when the client is disconnected from the network.

After the modification of Polling mechanism, the volume of requests is divided, which is more evenly distributed than before, and the problem of centralized requests no longer appears.

3.4 Unique Session ID

3.4.1 Why is the Message Line ID imported

Different message lines represent sessions of different objects. From the DB level, a table is required to store this relationship. Uid + object_id + busi_type = message line ID

In the early IM implementation, session configuration parameters (including service source and session parameters) were used to identify the session ID, which has three functions:

• Find the merchant ID, get the source of advice, and make the steward assignment

• Find an existing message line

• Judge the status of the client page and decide whether to send a push or not

There are two problems with this approach:

• The corresponding merchant ID is resolved by business source and session parameters. If one of the two parameters is missing, the merchant ID will be resolved incorrectly, and the merchant ID can only be obtained by various query databases, affecting efficiency.

• The interface for switching session types identifies the current session type. The switching page frequently triggers network requests. If an accident occurs to the request interface, the problem of message content error may occur, which depends heavily on the robustness of the client

It is inevitable to use business source and session parameters to help us with steward assignment, but we can replace the role of business source and session parameters in finding message lines by introducing the message line ID to bind the message line. In addition, the local push notification mechanism described above has been used to solve the problem of sending push notifications.

3.4.2 When to Create a Message Line

• When entering the session page to send messages, check whether the corresponding message line exists in the DB. If not, the message ID is used as the message line ID. If the message id exists, the message is reused.

• When entering the session, check whether the corresponding message line exists in the DB according to the user ID and service type ID. If the message line does not exist, create the message line. If the message line exists, the message line is reused.

3.4.3 Importing the message Line destination

• Reduces the cost of server querying message lines.

• Removing interface requests related to old version state changes indirectly improves push reach rate.

• Reduce the complexity of user message matching on the mobile end.

Fourth, prospects and recent optimization

4.1 Upgrade the data channel implementation mode to Websocket

WebSocket is a protocol for full duplex communication over a single TCP connection. WebSocket makes it easier to exchange data between the client and the server, allowing the server to actively push data to the client. In the WebSocket API, the browser and server only need to complete a handshake to create a persistent connection and two-way data transfer.

Compared with the current HTTP polling implementation mechanism, Websocket has the following advantages:

• Less control overhead. When data is exchanged between the server and client after the connection is created, the packet headers used for protocol control are relatively small. Without extensions, this header size is only 2 to 10 bytes (depending on packet length) for server-to-client content; For client-to-server content, an additional 4-byte mask is required for this header. This is significantly less overhead than carrying a full header each time an HTTP request is made.

• More real-time. Because the protocol is full-duplex, the server can proactively send data to the client at any time. Compared with HTTP, which requires waiting for the client to initiate a request before the server can respond, the latency is significantly less; Even compared to long polling like Comet, data can be delivered more times in a shorter time.

• Keep the connection state. Unlike HTTP, Websocket needs to create a connection first, which makes it a stateful protocol that can omit some of the state information for subsequent communication. HTTP requests, on the other hand, may need to carry status information (such as authentication) with each request.

• Better binary support. Websocket defines binary frames, making it easier to process binary content than HTTP.

• Support for extensions. Websocket defines extensions. Users can extend protocols and implement some customized sub-protocols, such as some browsers that support compression.

• Better compression. Compared to HTTP compression, Websocket, with appropriate extension support, can use the context of the previous content, which can significantly improve compression when passing similar data.

In order to further optimize our data channel design, we explored and verified the feasibility of Websocket, and conducted research and design:

In the near future, the HTTP polling implementation will be replaced to further optimize the efficiency of the data channel.

4.2 Extension of Service Functions

It is planned to make the functional module of IM mobile terminal into a general instant messaging component, which can more easily give each business IM capability, so that each business can quickly add the chat function on its own product line, and reduce the cost and difficulty of IM development. At present, IM functions are mainly composed of two components, which are the common data channel and UI components.

With the development of hornet’s cell business, there are still many ways to build and upgrade the existing IM system. For example, in the support of message types, the support for short videos, voice messages and quick message replies is extended to improve the convenience and interest of social communication. For multi-player scenes, we hope to add support for groups, interest channels, multi-player audio and video communication and other scenes.

We believe that in the future, through the expansion of more business functions and the exploration of application scenarios, CELLULAR mobile IM will better enhance user experience, and continue to empower businesses.

Author: Research and development team of IM mobile terminal of Cellular e-commerce business.

(Hornet’s Nest Technology original content)