The author of this article is Xiao Fuge, the original title “Using DDD+Netty, the development of a distributed IM(instant messaging) system”. In order to improve the reading experience, there are many revisions and changes, thanks to the original author.
1. Introduction to this paper
The study of computer programming, can acquire knowledge hand, pay attention to hands-on practice. In most of the articles I’ve written, the results of code validation are the core of the article.
I have been fond of hands-on activities since CHILDHOOD. Taking an instant communication project as an example, I have realized it for 5 or 6 times based on different technical solutions. Just to practice the technology, the screenshots are as follows.
As shown above:
-
1) Some are just after learning Socket and Swing, want to try these technologies can write a QQ out;
-
2) Some projects need to be completed because of internship training, but after having some basic knowledge, you can write all the functions in a week;
-
3) Although these projects look ugly at the moment, the code logic may not be perfect either. However, each implementation in the learning phase can bring a lot of technical growth for myself.
So, this opportunity to take advantage of this article, IM practice opportunities for you, I hope you can use.
Next, I will introduce you how to develop an IM in all aspects, including system architecture, communication protocol, single chat group chat, emoticon sending, UI event driven, etc., as well as a full set of practical source code so that you can start to learn.
Note: The source code can be downloaded in the attachment of the section “4. Source Code of this article”.
2. Knowledge preparation
* Important note: This article is not an instant messaging theory article, it is all organized by actual code, if you know too little about instant messaging (IM) technology theory, it is recommended to read more: “Getting started: Developing Mobile IM from Scratch”.
For those of you who don’t know what Netty is, here’s a quick introduction:
Netty is a Java open source framework. Netty provides an asynchronous, event-driven network application framework and tools to rapidly develop high-performance and reliable network server and client programs.
In other words, Netty is a niO-based client/server programming framework that allows you to quickly and easily develop a network application, such as a client/server application that implements a certain protocol.
Netty simplifies and streamlines the programming and development of network applications, such as TCP and UDP Socket services.
Here are a few introductory articles about Netty that are worth reading:
- Getting Started: The most Thorough Analysis of Netty high-performance Principles and Frameworks so far
- For Beginners: Learning Methods and Advanced Strategies for Netty, the Java High-performance NIO Framework
- The most popular Netty framework in the history of long article: Basic introduction, environment building, hands-on combat
If you don’t even know what Java NIO is, the following article is recommended as a first read:
- The Difference between Java NIO and Classic IO in one Minute
- The greatest Introduction to Java NIO ever: For those worried about getting started and giving up, read this!
- Java BIO and NIO are difficult to understand?
Netty API and Netty API
- 1) Netty-4.1.x
- 2) Netty-4.0.x
- 3) NetTY-4.1. x API Documentation (online version) (* recommended)
- 4) NetTY-4.0.x API Documentation (online version)
3. Operation effect
Before we start learning, we will demonstrate the operation effect of the accompanying source code (the source code can be downloaded in the attachment of the section “4, source code of this article”).
Chat page:
Add a friend:
Message alert:
4. The source code of this article
Download the complete code attachment:
(Download from the synchronous publishing link: www.52im.net/thread-3789…)
The directory structure of the source code is shown below:
This IM code is divided into three groups of modules: UI, client and server.
The reason for this split is to isolate THE UI presentation from the business logic, using events and interfaces to drive the code hierarchy cleaner and easier to extend and maintain.
The functions of each module are explained as follows:
5. System design
In this SET of IM, the server uses DDD domain-driven design mode to build. The function of Netty to SpringBoot for start and stop control, at the same time in the server to build a console can be very convenient operation of communication system, user and communication management. UI separation is adopted in the construction of the client to ensure the separation of business code and UI display, so that it is very easy to expand the control.
In addition, the function realization includes: perfect imitation of wechat desktop client, login, search to add friends, user communication, group communication, emoticons and other core functions. If there is a function that needs to be used, it can be extended according to this system framework.
Explain:
-
1) UI development: use JavaFx and Maven to build UI desktop project, and gradually explain login box, chat box, dialog box, friend bar and other UI display and operation events;
-
2) Architecture design: using the four-tier model structure of DDD domain driven design combined with Netty to construct a reasonable layered framework (design of corresponding library table functions);
-
3) Function realization: including; Login, add friends, dialogue notification, message sending, disconnected reconnect and other functions.
6. UI development
6.1 Function Division
Chat form, compared with the login form, chat form content will be more, but also relatively complex.
The following is a sketch of the chat form’s function definition:
As shown above:
-
1) The first is the definition of our entire main form of chat, which is a blank panel with the default border button (minimize, exit, etc.) removed;
-
2) After that comes our left sidebar, which we call Bar, the implementation of function area;
-
3) Finally add a form event to change the fill information in the content panel when the button is clicked.
6.2 Chat Interface
Dialog box selected content area display, that is, between users to send and display information.
On the whole, this is a linkage process. Click the user in the dialog box on the left, and the corresponding content will be filled on the right. So the list of conversations to be filled on the right side of the ListView needs to be associated with each conversation user. When clicking on the chat user, it is filled by repeatedly switching. The effect is shown below.
See the image above, let me explain:
-
1) Click each dialog box on the left, and the filling content of the right chat box will change accordingly (and the corresponding dialog name will also change);
-
2) In the dialog box, the left side shows the information sent by friends, and the right side shows the information sent by individuals (at the same time, the message content will increase the height and width as the content increases);
-
3) At the bottom is the text input field. In the later implementation, the text input field is designed in a common way. Of course, you can also design it for individual use.
6.3 Friend List
We often use wechat on the PC side, and we can see that there are several sections of content in the friends bar, including: new friends, public accounts, groups and friends at the bottom (functions divided as shown in the following figure).
See the image above, let me explain:
-
1) The content of the top search box remains unchanged, the same as before. The current approach we use is FXML design. For example, this part is generic and can be abstracted into code as a component element class.
-
2) After our analysis, based on the development of JavaFx components, this part is a nested ListView, that is, the bottom panel is a ListView, friends and groups are each a ListView, after this processing, we will be very convenient for data filling;
-
3) and the structure of the main benefit in the process of our program is running, if you add a friend, so we need to refresh the friend information to friends bar, and then fill the data, in order to more convenient and efficient, so we design the nested ListView (if not special understanding, can have the answer from the following code).
* define
In desktop UI development, in order to keep the UI isolated from the business logic, we need to package the UI to provide the interface to display the effects of the interface and interface action event abstract classes.
Then it can be understood as follows:
The above interfaces are all the behavior interfaces provided by our UI for the external at present. A link description of these interfaces is: open a window, search for friends, add friends, open a dialog box, send messages.
7. Communication design
7.1 System Architecture
Earlier I said that the more appropriate architecture is the one that best fits your current needs.
So how do you design the required architecture?
The reason for this design is that there are several premises in this system:
-
1) The system should have web pages on the server to manage communication users and control and monitor the server;
-
2) the object class of the database, do not be external pollution, to have isolation (for example: your database class is exposed to the external show class use, then now need to add a field, and this field is not your database attributes. The database class has been contaminated.
-
3) Because Netty communication is currently implemented in the Java language, both the server and the client need to use the protocol definition and parsing in the communication process. Then we need to pull this layer out and provide the Jar package externally (good for reuse, otherwise the client and server copy the same code maintenance, too gross);
-
4) Interfaces, business processing, underlying services and communication interactions should be clearly distinguished and implemented to avoid confusion and difficulty in maintenance.
Combined with our four premises, what model structure do you have in mind? Is there a plan for the technology stack selection?
Next I’ll introduce two models of architectural design, MVC, which you’re familiar with, and DDD domain-driven design, which you’ve probably heard of.
7.2 Communication Protocols
According to the drawing, we need to add a “frame identifier” in the transmission packet to determine which object the current business object is when transferring objects, which can make our business clearer and avoid using a lot of IF statements to judge.
Framework of agreement:
agreement
└ ─ ─ the SRC
├ ─ ─ the main
│ ├ ─ ─ Java
│ │ └ ─ ─ org. Itstack. Naive. Chat
│ │ ├ ─ ─ the codec
│ │ ├─ ObjDecoder.java
│ ├ ─ ├ ─ garbage
│ │ ├ ─ ─ protocol
│ │ ├─ Demo
│ │ ├─ Command
│ ├ ─ ├ ─ garbage
│ │ └ ─ ─ util
│ │ └ ─ ─ SerializationUtil. Java
│ ├ ─ ─ the resources
│ │ └ ─ ─ application. Yml
│ └ ─ ─ a webapp
│ └ ─ ─ chat
│ └ ─ ─ res
│ └ ─ ─ index. HTML
└ ─ ─ the test
└ ─ ─ Java
└ ─ ─ org. Itstack. Demo. Test
└ ─ ─ ApiTest. Java
Protocol package:
public abstract class Packet {
private final static Map<Byte, Class<? extendsPacket>> packetType = new ConcurrentHashMap<>();
static{
packetType.put(Command.LoginRequest, LoginRequest.class);
packetType.put(Command.LoginResponse, LoginResponse.class);
packetType.put(Command.MsgRequest, MsgRequest.class);
packetType.put(Command.MsgResponse, MsgResponse.class);
packetType.put(Command.TalkNoticeRequest, TalkNoticeRequest.class);
packetType.put(Command.TalkNoticeResponse, TalkNoticeResponse.class);
packetType.put(Command.SearchFriendRequest, SearchFriendRequest.class);
packetType.put(Command.SearchFriendResponse, SearchFriendResponse.class);
packetType.put(Command.AddFriendRequest, AddFriendRequest.class);
packetType.put(Command.AddFriendResponse, AddFriendResponse.class);
packetType.put(Command.DelTalkRequest, DelTalkRequest.class);
packetType.put(Command.MsgGroupRequest, MsgGroupRequest.class);
packetType.put(Command.MsgGroupResponse, MsgGroupResponse.class);
packetType.put(Command.ReconnectRequest, ReconnectRequest.class);
}
public static Class<? extends Packet> get(Byte command) {
return packetType.get(command);
}
/ * *
* Get protocol directives
*
* @return Returns the instruction value
* /
public abstract Byte getCommand();
}
7.3 Adding a Friend
As you can see from the flowchart above, there are two parts: Searching for friends and adding friends.
When adding a friend is complete, the friend will appear in our friends bar.
And here we use unilateral consent to add friends, that is, when you add a friend, the other side also has your friend information.
If you need to add friends in your business and agree with them, you can add a status message to request them to add friends. Two users can become friends and communicate with each other only after the other agrees.
Example code for adding a friend:
public class AddFriendHandler extends MyBizHandler {
public AddFriendHandler(UserService userService) {
super(userService);
}
@Override
public void channelRead(Channel channel, AddFriendRequest msg) {
// 1. Add A friend to database [A->B B->A]
List userFriendList = newArrayList<>();
userFriendList.add(newUserFriend(msg.getUserId(), msg.getFriendId()));
userFriendList.add(newUserFriend(msg.getFriendId(), msg.getUserId()));
userService.addUserFriend(userFriendList);
// 2. Push the friend to add A
UserInfo userInfo = userService.queryUserInfo(msg.getFriendId());
channel.writeAndFlush(newAddFriendResponse(userInfo.getUserId(), userInfo.getUserNickName(), userInfo.getUserHead()));
// 3. Push friend completion B
Channel friendChannel = SocketChannelUtil.getChannel(msg.getFriendId());
if(null== friendChannel) return;
UserInfo friendInfo = userService.queryUserInfo(msg.getUserId());
friendChannel.writeAndFlush(newAddFriendResponse(friendInfo.getUserId(), friendInfo.getUserNickName(), friendInfo.getUserHead()));
}
}
7.4 Message Response
As can be seen from the overall process: When a user initiates a friend or group communication, an event is triggered, and then the client sends a request for a conversation with a friend to the server.
After the server receives a chat request: For a friend chat, the server needs to save the communication information with the friend to the dialog box. At the same time notify friends, I and you are going to communicate. You added me to your list of dialogs.
If it is group communication: such notification is not necessary, because it is impossible to notify all the group users who are not online (they have not logged in), so this part only needs to create a dialog box to the list after the user goes online and receives the information. You can think about it carefully, but you can also think about other ways to implement it.
Sample message response code:
public class MsgHandler extends MyBizHandler {
public MsgHandler(UserService userService) {
super(userService);
}
@Override
public void channelRead(Channel channel, MsgRequest msg) {
Logger. info(” Message handler: {}”, json.tojsonString (MSG));
// Asynchronous write library
userService.asyncAppendChatRecord(newChatRecordInfo(msg.getUserId(), msg.getFriendId(), msg.getMsgText(), msg.getMsgType(), msg.getMsgDate()));
// Add dialog [add if the other party doesn’t have your dialog]
userService.addTalkBoxInfo(msg.getFriendId(), msg.getUserId(), Constants.TalkType.Friend.getCode());
// Get the buddy communication channel
Channel friendChannel = SocketChannelUtil.getChannel(msg.getFriendId());
if(null== friendChannel) {
Logger. info(” User ID: {} not logged in!” , msg.getFriendId());
return;
}
// Send a message
friendChannel.writeAndFlush(newMsgResponse(msg.getUserId(), msg.getMsgText(), msg.getMsgType(), msg.getMsgDate()));
}
}
7.5 Reconnection after a Cable Break
As we can see from the above flow, when the network connection is disconnected, a request is sent to the server for reconnection. So the process of initiating a link is different from the initial link of the system. Disconnection is to send the user ID information to the server so that the server can update the binding relationship between the user and the communication Channel.
In addition, you need to update the reconnection information in the group and add the reconnection information of users to the group mapping. At this point, the user can resume communication with friends and groups.
Sample message response code:
// Channel status periodic inspection; The command is executed every 5 seconds after 3 seconds
scheduledExecutorService.scheduleAtFixedRate(() -> {while(! Nettyclient.isactive ()) {system.out.println (” NettyClient.isActive () “);
Try {system.out. println(” Pipeline inspection: Reconnection [Begin]”);
Channel freshChannel = executorService.submit(nettyClient).get();
if(null== CacheUtil.userId) continue;
freshChannel.writeAndFlush(newReconnectRequest(CacheUtil.userId));
} the catch (InterruptedException | ExecutionException e) {System. Out. Println (” communications pipeline inspection: break line reconnection [Error] “); }
}
}, 3, 5, TimeUnit.SECONDS);
Related articles to learn:
- Why Does TCP – based MOBILE IM Still Need heartbeat Keepalive mechanism?
- Understanding the Mechanism of Network Heartbeat Packet in Instant Messaging Applications: Functions, Principles, Implementation ideas, etc.
- “Sharing of Rongyun Technology: Practice of Network Link Preservation Technology of Rongyun Android IM Products”
- “Correctly understand the IM long connection heartbeat and reconnection mechanism, and start to implement (complete IM source code)”
- Discussion on the design and Implementation of an Android IM Intelligent Heartbeat Algorithm (with sample code)
- “Hand to hand teach you to use Netty to realize the heartbeat mechanism of network communication program, disconnection reconnection mechanism”
- Instant Messaging practice on the Web: How to Make your WebSocket Down and Reconnect Faster?
7.6 Cluster Communication
As shown above, I implement IM cluster communication like this:
-
1) Use Redis publish and subscribe to deliver messages across services. If you are a large service, you can use ZooKeeper.
-
2) When sending A message to user B, user A needs to send user B’s channeId for the server to check whether channeId belongs to its own service.
-
3) A single machine can also start multiple Netty services, the program will automatically find available ports.
8. Summary of this paper
There are many contents of technology stack involved in this IM system: Netty4.x, SpringBoot, Mybatis, Mysql, JavaFx, LayUI and other technology stack, as well as the whole system framework structure adopts DDD four-layer architecture +Socket module to build, all THE UI are designed in the way of back-end separation event drive. In this process as long as you can stick to learning down, so will harvest a lot of content. Enough boasting!
The learning process for any new technology stack will include running HelloWorld, mastering the API, working on projects, and finally deep source code mining. So when hearing such a requirement, Java programmers must think of a number of technical knowledge points to fill the modules in our project (such as: interface using JavaFx, Swing, etc., communication Socket or know Netty framework, server control using MVC model plus SpringBoot, etc.). However, it is the most important part in the process of learning, practicing and growing that how to set up our system properly.
Well, IM development actually involves knowledge dimension is very much, limited to space is not here verbose more, you readers must be on the source code synchronization to learn, so that the effect will be better (source code in this article “4, this article source code” section of the attachment can be downloaded).
9. Series of articles
“Follow the source code to learn IM(a) : handhold teach you to use Netty to achieve heartbeat mechanism, disconnection mechanism” “follow the source code to learn IM(two) : their own IM development is difficult? Handhold teach you a Android version of IM” follow the source code to learn IM(three) : Based on Netty, from zero development of an IM server “follow the source to learn IM(four) : pick up the keyboard is dry, teach you freehand development of a set of distributed IM system” follow the source to learn IM(five) : the correct understanding of IM long connection, heartbeat and reconnection mechanism, and begin to achieve “follow the source to learn IM(six) : Hand-hold teach you to use Go to quickly build high performance, extensible IM system “follow the source to learn IM(seven) : Hand-hold teach you to use WebSocket to create Web IM chat” follow the source to learn IM(eight) : ten thousand words, hand-hold teach you to use Netty to build IM chat “follow the source to learn IM(nine) : A Distributed IM System Based on Netty
10. Reference materials
[1] Beginner: By far the most thorough Netty high-performance principle and framework architecture analysis
[2] For beginners: Learning methods and Advanced Strategies for Java High performance NIO Framework Netty
[3] The most powerful Introduction to Java NIO in history: From getting started to giving up, please read this article!
[4] Are Java BIO and NIO difficult to understand? Use code practice to show you, do not understand me to change!
[5] The most popular Netty framework introduction long article: Basic introduction, environment building, hands-on practice
[6] Theory and Practice: a description of a typical IM communication protocol design
[7] Discussion on IM system architecture design
[8] Briefly describe the pits of mobile IM development: architecture, communication protocols, and clients
[9] A set of mobile IM Architecture Design practice sharing of massive Online Users (with detailed pictures and texts)
[10] A theoretical architecture scheme for original distributed instant messaging (IM) systems
[11] A set of high availability, easy to scale, high concurrency IM group chat, single chat architecture design practice
[12] A set of 100 million user IM architecture technology Dry product (Part 1) : overall architecture, service separation, etc
[13] A set of 100 million user IM architecture technology, Part 2: Reliability, order, weak network optimization
[14] From novice to Expert: How to design a distributed IM system with 100 million messages
[15] Based on Practice: Summary of technical points of a million message small scale IM system
(This article has been simultaneously published at: www.52im.net/thread-3789…)