Now that we use the Internet almost every day, we’ve learned how to program Go, but how do we get our programs to talk to each other over the network? In this chapter, we will learn network programming in the Go language. Network programming is a huge field, and this article has simply demonstrated how to use NET packages for TCP and UDP communication. For more detailed network programming please search and read professional materials.
Introduction to Internet Protocols
At the heart of the Internet is a set of protocols, collectively known as the Internet Protocol Suite, that govern how computers are connected and networked. When we understand these protocols, we understand how the Internet works. Because these protocols are too large and complex, there is no way to cover all of them here. We can only introduce a few protocols that we deal with a lot in our daily development.
Internet Hierarchical model
The logical implementation of the Internet is divided into several layers. Each floor has its own function, and like a building, each floor is supported by the next. The user only touches the top layer and never feels the layers below. Understanding the Internet requires a bottom-up understanding of what each layer implements.
As shown in the figure above, the Internet is divided into different layers according to different models, but no matter what model is divided, the higher the layer is, the closer the user is, and the lower the layer is, the closer the hardware is. The model we use most often in software development is the one shown above that divides the Internet into five layers.
Let’s take a look at each layer, layer by layer, from the bottom up.
The physical layer
To communicate with the Internet outside our computer, we need to connect the computer to the network first, we can use twisted pair, optical fiber, radio waves and other ways. This is called the real physical layer, and it’s the physics that connects computers. It mainly defines some electrical characteristics of the network, the role is responsible for the transmission of 0 and 1 electrical signals.
Data link layer
Zeros and ones alone don’t make any sense, so we users assign them specific meanings that dictate how to interpret electrical signals: for example, how many electrical signals are in a group? What does each signal bit mean? This is the function of the “data link layer”, which, above the “physical layer”, determines how the 0s and 1s transmitted at the physical layer are grouped and what they represent. In the early days, each company had its own way of grouping electrical signals. Gradually, a protocol called Ethernet came to dominate.
Ethernet states that a group of electrical signals constitutes a packet, called a Frame. Each frame is divided into two parts: Head and Data. The “header” contains some description of the packet, such as sender, receiver, data type, etc. “Data” is the specific content of the packet. The length of the header, fixed to 18 bytes. The minimum length of data is 46 bytes and the maximum length is 1500 bytes. Therefore, the entire “frame” is 64 bytes minimum and 1518 bytes maximum. If the data is long, it must be split into multiple frames and sent.
So how are senders and receivers identified? Ethernet stipulates that all devices connected to the network must have a “nic” interface. Packets have to go from one network card to another. The address of the nic is the address for sending and receiving packets, which is called the MAC address. Each NIC is delivered with a MAC address unique in the world. The length of the ADDRESS is 48 bits, usually represented by 12 hexadecimal numbers. The first six hexadecimal numbers are manufacturer ids, and the last six are nic serial numbers of the manufacturer. With a MAC address, you can locate the path of the network card and the packet.
We use ARP to obtain the MAC address of the receiving party. After obtaining the MAC address, how can we accurately send data to the receiving party? Actually here Ethernet USES a very “primitive” way, it is not the packets sent to the receiver accurately, but to all the computer to send, in this network to each computer reads this package “header”, find the MAC address of the receiver, and then compared with its own MAC address, if both are the same, just accept the package, do further processing, Otherwise, the package is discarded. This mode of transmission is called broadcasting.
The network layer
According to the rules of the Ethernet protocol we can rely on the MAC address to send out data. Theory on MAC address, your computer’s network card can be found in another corner of the world of a computer network card, but this kind of practice is a major defect is Ethernet USES the broadcast mode to send data packets, all members of a “package” hand, not only low efficiency, and sending data can only be confined to the sender’s own subnet. That is, if two computers are not in the same subnetwork, the broadcast will not be transmitted. This design is reasonable and necessary because it is unrealistic for every computer on the Internet to receive every packet sent and received over the Internet.
Therefore, a way must be found to tell which MAC addresses belong to the same subnetwork and which do not. If the packets are from the same subnetwork, the packets are sent in broadcast mode; otherwise, the packets are sent in routing mode. This led to the creation of the “network layer”. What it does is introduce a new set of addresses that allow us to distinguish between different computers that belong to the same subnetwork. This set of addresses is called “network addresses”, or “web addresses” for short.
With the advent of the “network layer,” every computer has two types of address, one is a MAC address, the other is a network address. The MAC address is bound to the nic, and the network address is assigned by the network administrator. Network addresses help us determine which subnetwork the computer is on, and MAC addresses send packets to the destination network card in that subnetwork. Therefore, it is logical to assume that the network address must be processed first and then the MAC address.
The protocol for specifying network addresses is called the IP protocol. The address it defines is called an IP address. At present, the IP protocol version 4, abbreviated IPv4, is widely used. IPv4 defines that a network address consists of 32 binary bits. An IP address is represented by four decimal digits, ranging from 0.0.0.0 to 255.255.255.255.
Data sent over IP is called AN IP packet. IP packets are also divided into header and data. The header contains the version, length, and IP address, and the data contains the specific content of IP packets. The length of the “header” part of an IP packet ranges from 20 to 60 bytes, and the total length of the entire packet is up to 65535 bytes.
The transport layer
With MAC addresses and IP addresses, we can already establish communication between any two hosts on the Internet. But the problem is that there will be many programs on the same host need to use the network to send and receive data, such as QQ and browser these two programs need to connect to the Internet and send and receive data, how do we distinguish a packet is under which program? That is, we also need a parameter to indicate which program (process) is using the packet. This parameter is called a port, and it is the number of each program that uses the network card. Each packet is sent to a specific port on the host, so different programs can get the data they need.
“Port” is an integer between 0 and 65535, with exactly 16 binary bits. Ports 0 to 1023 are occupied by the system. You can select only ports larger than 1023. With IP and port, we can realize the unique determination of a program on the Internet, and then realize the program communication between networks.
We have to include port information in packets, which requires new protocols. The simplest implementation is called UDP, and the format is almost nothing more than a port number in front of the data. A UDP packet consists of header and data. The header defines the sending and receiving ports, and the data defines the specific content. UDP packets are very simple. The “header” section is only 8 bytes long, and the total length is no more than 65,535 bytes, which fits into an IP packet.
The ADVANTAGE of UDP is that it is simple and easy to implement. However, the disadvantage is that the reliability is poor. Once a packet is sent, you cannot know whether the packet has been received. In order to solve this problem, improve network reliability, TCP protocol was born. TCP ensures that data is not lost. Its disadvantages are complicated process, difficult implementation and consuming more resources. The length of a TCP packet is unlimited. However, to ensure network efficiency, the length of a TCP packet does not exceed that of an IP packet, so that a single TCP packet does not need to be split.
The application layer
The application receives data from the “transport layer” and then unpacks the data. “Because the Internet is an open architecture and data comes from so many different sources, the data format for communication must be defined in advance, otherwise the recipient will not be able to get the actual content of the data sent.” The application layer defines the data format used by applications. For example, protocols such as Email, HTTP, and FTP are common on top of TCP. These protocols constitute the Application layer of Internet protocols.
As shown in the figure below, during the transmission of HTTP data from the sender over the Internet, the header information of each layer protocol will be added successively. After receiving the data packet, the receiver will unpack the data according to the protocol in turn.
Socket programming
A Socket is a process communication mechanism of BSD UNIX. It is also called a Socket. It describes an IP address and port and is a handle to a communication chain. Socket can be understood as the TCP/IP network API. It defines many functions or routines that programmers can use to develop applications over the TCP/IP network. Applications running on a computer usually make or respond to network requests through sockets.
Diagram of the socket
Socket is an intermediate software abstraction layer for communication between application layer and TCP/IP protocol family. In the design mode, Socket is actually a facade mode, it hides the complex TCP/IP protocol family behind the Socket, for users only need to call the relevant functions specified by the Socket, let the Socket to organize in line with the specified protocol data and then communicate.
Go language to achieve TCP communication
TCP protocol
Transmission Control Protocol/Internet Protocol (TCP/IP) is a connection-oriented and reliable Transport layer communication Protocol based on byte streams. Because it is a connection-oriented protocol, data flows like water, causing sticky packets.
TCP server
A TCP server can connect to many clients at the same time. For example, users from all over the world access Taobao.com by using browsers on their computers. Because it’s easy and efficient to create multiple Goroutine implementations concurrently in Go, we can create a Goroutine for each link we make.
TCP server program processing flow:
- Listen on port
- Receive a client request to establish a link
- Create a Goroutine processing link.
We use the Go language net package to achieve the TCP server code is as follows:
// TCP /server/main.go // TCP server/ / func process(conn net.conn) {defer conn.close () // Close connection for {reader := NewReader(conn) var buf[128] Byte n, err := read. Read(buf[:]) // Read data if err! = nil { fmt.Println("read from client failed, err:", Err) break} recvStr := string(buf[:n]) fmt.Println(" Received data from client: ", recvStr) conn.Write([]byte(recvStr)) // send data}} func main() {listen, err := net. listen (" TCP ", "127.0.0.1:20000") if err! Println("listen failed, err:", err) return} for {conn, err: = list.accept () // Establish a connection if err! Println(" Accept failed, err:", err) continue} go process(conn) // Start a goroutine process connection}} copy codeCopy the codeSave the above code and compile it into a server or server.exe executable.
TCP client
The TCP communication flow of a TCP client is as follows:
- Establish a link to the server
- Data receiving and receiving
- Close links
The TCP client code using the Go language NET package is as follows:
// TCP /client/main.go // client func main() {conn, err := net.Dial(" TCP ", "127.0.0.1:20000") if err! = nil { fmt.Println("err :", Err) return} defer conn.close () // Close the connection inputReader := bufio.newreader (os.stdin) for {input, _ := inputReader.ReadString('\n') // inputInfo := strings.Trim(input, "\r\n") if strings.ToUpper(inputInfo) == "Q" {// If Q is entered, return} _, Err = conn.Write([]byte(inputInfo)) // Send data if err! = nil { return } buf := [512]byte{} n, err := conn.Read(buf[:]) if err ! = nil {fmt.Println("recv failed, err:", err) return} fmt.Println(string(buf[:n]))}} Copy codeCopy the codeCompile the above code into client or client.exe executable file, start the server and then start the client, enter any content in the client, you can see the data sent by the client on the server, so as to achieve TCP communication.
Glue the TCP packet
Stick pack sample
The server code is as follows:
// socket_stick/server/main.go func process(conn net.Conn) { defer conn.Close() reader := bufio.NewReader(conn) var buf [1024]byte for { n, err := reader.Read(buf[:]) if err == io.EOF { break } if err ! Println("read from client failed, err:", err) break} recvStr := string(buf[:n]) fmt.Println(" Received data from client: ", recvStr)}} func main() {listen, err := net. listen (" TCP ", "127.0.0.1:30000") if err! = nil { fmt.Println("listen failed, err:", err) return } defer listen.Close() for { conn, err := listen.Accept() if err ! = nil {fmt.Println(" Accept failed, err:", err) continue} go process(conn)}} Copy codeCopy the codeThe client code is as follows:
// socket_stick/client/main.go func main() {conn, err := net.Dial(" TCP ", "127.0.0.1:30000") if err! = nil { fmt.Println("dial failed, err", err) return } defer conn.Close() for i := 0; i < 20; i++ { msg := `Hello, Hello. How are you? 'conn.write ([]byte(MSG))}} Copies the codeCopy the codeSave the above code and compile it separately. Start the server first and then the client. You can see the output of the server as follows:
Received data from client: Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Received data from client: Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Received data from client: Hello, Hello. How are you? Hello, Hello. How are you? Received data from client: Hello, Hello. How are you? Hello, Hello. How are you? Hello, Hello. How are you? Received data from client: Hello, Hello. How are you? Hello, Hello. How are you? Copy the codeCopy the codeThe client sends 10 times of data, but the server does not successfully output 10 times of data, but multiple pieces of data “stick” together.
Why do sticky bags appear
The main reason is that THE TCP data transfer mode is stream mode, and multiple incoming and outgoing packets can be sent while maintaining a long connection.
Sticky packets can occur at both the sender and the receiver:
- Sticky packets at the sender caused by Nagle algorithm: Nagle algorithm is an algorithm to improve the efficiency of network transmission. In simple terms, when we submit a piece of data to TCP, TCP does not immediately send the piece of data. Instead, TCP waits for a short period of time to see if there is any more data to be sent. If there is, TCP sends both pieces of data at once.
- Sticky packets on the receiving end caused by delayed reception: TCP stores the received data in its own buffer and notifies the application layer to retrieve the data. When the application layer fails to retrieve TCP data in time for some reason, several pieces of data are stored in the TCP buffer.
The solution
The key to sticky packets is that the receiver is not sure about the size of the packets to be transmitted, so we can pack and unpack the packets.
Packet encapsulation: A packet is a header added to a piece of data. In this way, the packet is divided into the header and the packet body (the “packet tail” is added to the packet when illegal packets are filtered). The length of the packet header part is fixed, and it stores the length of the packet body. According to the fixed length of the packet header and the variable containing the packet body length, a complete packet can be correctly split.
We can define a protocol by ourselves, for example, the first 4 bytes of the packet is the packet header, which stores the length of the data to be sent.
// socket_stick/proto/proto.go package proto import ("bufio" "bytes" "encoding/binary") // Encode Encode message func Encode(message string) ([]byte, error) { Var length = int32(len(message)) var PKG = new(bytes.buffer) // Write to the message header err := binary.Write(PKG, binary.LittleEndian, length) if err ! = nil {return nil, err} // Write the message entity err = binary.Write(PKG, binary.LittleEndian, []byte(message)) if err! = nil {return nil, err} return pkg.bytes (), nil} // Decode Decode messages func Decode(reader * bufio.reader) (string, Error) {// Read the length of the message lengthByte, _ := reader.peek (4) // Read the first 4 bytes of data lengthBuff := bytes.NewBuffer(lengthByte) var Length int32 err := binary.Read(lengthBuff, binary.LittleEndian, &length) if err ! = nil {return "", err} // Buffered Returns the number of bytes available in the buffer that can be read. If int32(reader.buffered ()) < length+4 {return "", err} pack := make([]byte, int(4+length)) _, err = reader.Read(pack) if err ! = nil {return "", err} return string(pack[4:]), nil} copy codeCopy the codeThe data is then processed on the server side and the client side using the Decode and Encode functions of the Proto package defined above, respectively.
The server code is as follows:
// socket_stick/server2/main.go func process(conn net.Conn) { defer conn.Close() reader := bufio.NewReader(conn) for { msg, err := proto.Decode(reader) if err == io.EOF { return } if err ! = nil {FMT.Println("decode MSG failed, err:", err) return} FMT.Println(" Received data from client: ", MSG)}} func main() {listen, err := net. listen (" TCP ", "127.0.0.1:30000") if err! = nil { fmt.Println("listen failed, err:", err) return } defer listen.Close() for { conn, err := listen.Accept() if err ! = nil {fmt.Println(" Accept failed, err:", err) continue} go process(conn)}} Copy codeCopy the codeThe client code is as follows:
// socket_stick/client2/main.go func main() {conn, err := net.Dial(" TCP ", "127.0.0.1:30000") if err! = nil { fmt.Println("dial failed, err", err) return } defer conn.Close() for i := 0; i < 20; i++ { msg := `Hello, Hello. How are you? ` data, err := proto.Encode(msg) if err ! = nil {fmt.Println("encode MSG failed, err:", err) return} conn.write (data)}} Copy codeCopy the codeGo language to achieve UDP communication
UDP protocol.
The User Datagram Protocol (UDP) is a connectionless transport Protocol in the Open System Interconnection (OSI) reference model. Data can be sent and received directly without establishing a connection, which is not reliable and has no timing communication. However, UDP protocol has good real-time performance and is usually used in fields related to live video broadcasting.
UDP server.
Net package using Go language to achieve the UDP server code is as follows:
// UDP/server/main.go // UDP server func main() {listen, err := net.listenudp (" UDP ", &net.udpaddr {IP: net.IPv4(0, 0, 0, 0), Port: 30000, }) if err ! = nil { fmt.Println("listen failed, err:", err) return } defer listen.Close() for { var data [1024]byte n, addr, ReadFromUDP(data[:]) // Receive data if err! = nil { fmt.Println("read udp failed, err:", err) continue } fmt.Printf("data:%v addr:%v count:%v\n", string(data[:n]), Addr, n) _, err = list. WriteToUDP(data[:n], addr) // Send data if err! = nil {fmt.Println("write to UDP failed, err:", err) continue}}} Copy codeCopy the codeUDP client
Net package using Go language to achieve the UDP client code is as follows:
// UDP client func main() {socket, err := net.dialudp (" UDP ", nil, &net.udpaddr {IP: net.ipv4 (0, 0, 0, 0), Port: 30000, }) if err ! = nil {fmt.Println(" Failed to connect to server, err:", err) return } defer socket.Close() sendData := []byte("Hello server") _, Err = socket.Write(sendData) // sendData if err! Println(" Failed to send data, Err :", err) return} data := make([]byte, 4096) n, remoteAddr, err: = socket.ReadFromUDP(data) // Receive data if err! Println(" Failed to receive data, Err :", err) return} FMT.Printf("recv:%v addr:%v count:%v\n", string(data[:n]), remoteAddr, n)} Copy codeCopy the codeAbove is about Go language network programming introduction, if you want to know more, welcome to “Go keyboard man” public number, exchange and learn together, ^_^!
Recommended articles:
Performance Tuning for Pprof in Go