The socket socket

This is the 8th day of my participation in the More text Challenge. For details, see more text Challenge

Wechat public number search [program yuan xiaozhuang], concerned about halfway out of the program yuan how to rely on Python development to make a living ~

preface

Two applications if you need to communicate one of the most basic premise is to be able to show a process, only we know the IP address of the IP layer host only labels, and TCP protocol and port number identifying the host can be the only one process, so that we can use the IP address + port number + agreement only a process identified in the network. Once processes on the network can be uniquely identified, they can communicate using sockets, which are described in this article.

What is a socket

What is a socket? Socket is an abstraction layer between the application layer and the transport layer. It abstracts the complex operation of TCP/IP layer into several simple interfaces for the application layer to call to realize the process communication in the network.

The purpose of learning network programming is to develop software based on Internet communication, whether BS architecture or CS architecture. We develop the Internet communication software is in the TCP/IP five layer protocol application layer, when the data needs to be transmitted through the Internet, it needs to use the socket abstraction layer. This socket abstraction layer is not part of the TCP/IP layer 5, but is an abstraction that helps us encapsulate the other layers below the transport layer. It hides the complex TCP/IP protocol family behind the Socket interface. For the user, a simple set of interfaces is all, allowing the Socket to organize data in accordance with the specified protocol. In the development, only need to follow the provisions of socket code, written out of the program to follow the TCP/UDP protocol.

History and classification of sockets

Sockets originated in the 1970s in the Uc Berkeley version of Unix, known as BSD Unix. For this reason, sockets are sometimes referred to as “Berkeley sockets” or “BSD sockets.” Originally, sockets were designed to communicate between multiple applications on the same host. This is also known as interprocess communication, or IPC. There are two kinds of sockets (or there are two races), one is file based and the other is network based.

Family of sockets based on file type

The name of the socket family: AF_UNIX. In Unix, everything is a file. A file-based socket calls the underlying file system to fetch data.

Family of sockets based on network type

Socket family names: AF_INET, as well as AF_INET6, are used for ipv6, and there are a number of other address families that are either platform specific, obsolete, rarely used, or not implemented at all. AF_INET is the most widely used of all address families. Python supports many address families, but since we only care about network programming, we use only AF_INET most of the time.

The socket module

A socket enables two processes in a network to communicate. We can compare the working flow of a socket from an example in daily life. For example, if you want to call a friend, dial first, and the friend picks up the phone after hearing the phone ring. When the conversation is over, hang up and end the conversation. A real-life scenario explains how sockets work.

Server:

1 Initialize the socket object

2 Binding the Socket object to the IP address and port of the server (bind)

3 Listening on a Port

Client:

1 Initialize the socket object

2 Connecting to the Server (connect)

3 If the connection is successful, the connection between the client and server is established

After successful connection:

1 The client sends a data request

2 The server receives and processes the request

The response data is sent to the client, and the client reads the data

4 Close the connection. The interaction is complete

The Socket module in Python encapsulates the socket abstraction layer and provides some simple interfaces to help us do this. The following code describes the basic use of the socket module:

Add: The python way to view the source code: hold down CTRL, hover the mouse over the object that you want to view the source code, click, and you can see the source code.

# socket_family can be EITHER AF_UNIX or AF_INET. Socket_type can be SOCK_STREAM or SOCK_DGRAM. The default value of socket_family is -1, indicating AF_INET. The default value of socket_type is also -1, indicating SOCK_STREAM. Can be viewed through the source code
socket.socket(socket_family, socket_type, protocal=0)

# Get TCP/IP socket
tcp_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcp_sock = socket.socket()  The default value is -1

Get the UDP/IP socket
udpSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
Copy the code

Socket module provides different methods for client program and server program to send and receive data, and also provides some common methods. Know the socket module usage, we can according to the socket module based on TCP/UDP protocol for the development of client and server side small program. Let’s introduce them in turn:

First, the socket module provides a method for the server:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  # Get the socket object
s.bind()   Bind host and port number to socket
s.listen()  Start TCP listening
s.accept()  # Passively accept TCP client connections and wait for a connection to arrive
Copy the code

Here’s what the socket module provides for clients:

import socket

s = socket.socket()  # socket object
s.connect()  # Initializing the connection to the TCP server
s.connect_ex()  An extended version of the connect() function that returns an error code instead of throwing an exception
Copy the code

Finally, there are some common methods that some sockets provide for clients and servers:

s.recv()   # Receive TCP data
s.send()   # Send TCP data (when the amount of data to be sent is greater than the remaining space in the cache, the data will be lost and will not be sent out)
s.sendall()   # Send complete TCP data (essentially, call sendall in a circular way. When the amount of data to be sent is greater than the remaining space in the cache on its own side, data will not be lost, and call Sendall in a circular way until the data is sent out)
s.recvfrom()   # Receive UDP data
s.sendto()   Send UDP data
s.getpeername()   # Address of the remote end to connect to the current socket
s.getsockname()   # The address of the current socket
s.getsockopt()   # Returns the arguments for the specified socket
s.setsockopt()   Set the parameters for the specified socket
s.close()   Close the socket connection
Copy the code

TCP based sockets

TCP is based on bidirectional connections, so you must first start the server and then start the client to connect to the server.

Start with the server: the server initializes the Socket, then binds to the port, listens to the port, blocks with the Accept call, and waits for the client to connect.

If a client initiates a Socket and then connects to the server (connect), if the connection succeeds, the connection between the client and the server is established.

The client sends the data request, the server receives the request and processes the request, and then sends the response data to the client, the client reads the data, and finally closes the connection, the end of an interaction.

The above is a simple process based on TCP communication, according to the principle of the call comparison, the specific code is as follows:

Simple VERSION of TCP communication

Server file:

# sever. Py server file
Take the mobile phone for example
import socket

# 1 Buy a phone -- get the object the server sends and receives data from
phone = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

# 2 Mobile phone card - bind IP+ port to server
phone.bind(('127.0.0.1'.6666))

# 3 Startup -- The server is listening
phone.listen(3)  Half connection pool can only hold three connection requests to be confirmed

# 4 Wait for the call - the server and the client to establish a connection, after the establishment of the connection can get TCP connection channel information, and the client IP and interface
conn,client_addr = phone.accept()
print(conn)
print('Client IP and interface',client_addr)

# 5 The phone is connected, and two people have a pleasant chat - sending and receiving data
data = conn.recv(1024)  # The maximum amount of data received is 1024bytes
print('Client message', data.decode('utf-8'))
conn.send(data.upper())

# 6 Hang up -- Disconnect from the client
conn.close()

# 7 Mobile phone shutdown - Server shutdown
phone.close()
Copy the code

Client file

# client.py
import socket

phone = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

phone.connect(('127.0.0.1'.6666))

phone.send('show information'.encode('utf-8'))

data = phone.recv(1024)

print('Server',data.decode('utf-8'))

phone.close()
Copy the code

TCP circular communication

The above simple version of the communication code, there is a problem, is a great effort to connect successfully, the result of a message to the connection is disconnected, just like a phone call, a call to say a word, hang up the phone, if there is not finished, but also need to call….

To solve this problem, we can use a loop to continuously receive or send data – loop communication

Server file:

# sever.py
import socket

# 1 Buy a phone -- get an object on the server
phone = socket.socket()

# 2 Mobile phone card - Determine the IP and port of the server
phone.bind(('127.0.0.1'.7890))

# 3 Mobile phone boot - the server enters the listening state
phone.listen(3)

# 4 Wait to answer the call and get the IP and port of the TCP channel and client

conn,client_addr = phone.accept()

# 5 Call -- send and receive data
while True:
    # Exception handling: prevent server crash when the client suddenly shuts down
    try:
        data = conn.recv(1024)
        if not data:break
        print('Client',data.decode('utf-8'))

        conn.send(data.upper())
    except Exception:
        break

# 6 Hang up -- the server is disconnected from the client channel
conn.close()

# 7 Shutdown -- The server is shutdown
phone.close()
Copy the code

Client file:

# client.py
import socket

phone = socket.socket()
phone.connect(('127.0.0.1'.7890))
while True:
    info = input('> >').strip()
    # When the length of the sent data is 0, both the server and client will enter the blocking stage of waiting for the data to be received. Therefore, judge whether the length of the information entered by the user is 0 after being processed by the strip
    if len(info) == 0:continue
    if info == 'q': break
    phone.send(info.encode('utf-8'))
    data = phone.recv(1024)
    print('Server', data.decode('utf-8'))

phone.close()
Copy the code

TCP communication is not disabled on the server

The above code scheme also has problems. As a server, it should meet two conditions: one is to provide services to the client all the time; the other is to provide services concurrently. When the client is disconnected from the above circular communication, the server will also terminate, and it can no longer provide services to other clients. So we can keep the server in the open state. The following is the optimized server code:

# sever.py
import socket

phone = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

phone.bind(('127.0.0.1'.65447))

phone.listen(5)
while True:
    conn, client_addr = phone.accept()
    while True:
        try:
            data = conn.recv(1024)
            if not data:break
            print('Client',data.decode('utf-8'))
            conn.send(data.upper())
        except Exception:
            break

    conn.close()  Disconnect the client from the server
Copy the code

The server can only serve one client at a time. If you want the server to serve multiple servers, you need to use the knowledge we will learn later – concurrent programming. I’m not going to go into that here.

When using TCP communication, clients can also have problems if they send empty messages. If the message sent by the client is empty, the message is not sent. Instead, the application sends the message to the operating system. After receiving the message, the operating system finds that it is empty and does not send the message out. So the client sent empty into recV and other state state, and at this time the server is not received any message will not reply, so in the awkward state of both sides. The solution is to determine whether the message sent by the client is empty. If the message is empty, skip to the next loop and prevent the empty message from being sent to the operating system.

UDP based sockets

Udp is not linked, the first to start which end will not be an error. Therefore, it is much simpler to use than TCP-based sockets.

UDP is a datagram protocol. When sending a null message, the server can also receive the null message.

Udp-based sockets are not connected. Both the client and the server can start the socket communication first, and both can end the communication in advance. That is, the client leaves without affecting the normal operation of the server.

Server code

# server.py
import socket

ip_port = ('127.0.0.1'.8080)
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind(ip_port)
print('The server starts listening on...... ')

while True:
    get_msg, client_addr = server.recvfrom(1024)
    print('from client:', get_msg)
    server.sendto(get_msg.upper(), client_addr)

# server.close()
Copy the code

Client code

import socket

ip_port = ('127.0.0.1'.8080)
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

while 1:
    msg = input('> > > :').strip()
    if msg == 'q':
        break
    if not msg:		In udp sockets, the client can send null, and the server can also receive null
        continue
    client.sendto(msg.encode('utf-8'), ip_port)
    get_msg, server_addr = client.recvfrom(1024)
    print(get_msg.decode('utf-8'))                     # Close the client socket
Copy the code

The TCP protocol is sticky. Procedure

Stick package problem

First of all, TCP has the problem of sticking packets, UDP does not have the problem of sticking packets.

The sender can send 1KB of data, and the receiving application can carry 2KB of data, but it is also possible to carry 3K or 6K of data at a time, or only a few bytes at a time, which means that the application sees the data as a whole, or as a stream. The number of bytes in a message is not visible to the application, so TCP is a flow-oriented protocol, which is why it is prone to sticky packets. While UDP is a message-oriented protocol, each UDP segment is a message, and the application must extract data as a message, not any byte of data at a time, which is very different from TCP.

TCP is connection-oriented and stream-oriented and provides high reliability services. There must be a one-to-one pair of sockets at both ends (client and server). Therefore, the sender uses an optimization method (Nagle algorithm) to combine multiple packets with small intervals and small data volume into a large data block and then packet. This makes it difficult for the receiver to tell the difference and must provide a scientific unpacking mechanism. That is, flow-oriented communication has no message protection boundary.

UDP is connectionless, message-oriented, and provides efficient services. Because UDP supports one-to-many, the skbuff(socket buffer) on the receiving end uses a chain structure to record each incoming UDP packet, and within each UDP packet is the message header (source address, port, etc.). In this way, for the receiver, it is easy to distinguish processing. That is, message-oriented communication has message protection boundaries.

** The problem of sticking packets based on TCP protocol is mainly caused by the receiver not knowing the boundaries between messages and how many bytes of data should be extracted each time. ** To sum up, there are two situations in which the sticky bag problem can occur:

• The data with small intervals and small data volume is merged into a large data block and then encapsulated, resulting in sticky packets
• The client sends a piece of data, the server receives only a small part, the server receives the next time or from the buffer to take the last data, resulting in sticky packets

Solve the sticky bag problem

The root of the problem is that the receiver does not know the length of the byte stream to be transmitted by the sender, so the solution for sticking packets is to let the sender know the total size of the byte stream to be sent before sending data, and then the receiver receives all the data in a cycle. After each message is received, leave no residue.

The method is as follows: Before sending data, the sender sends the length of the data to be sent to the receiver. Send the data length in a fixed length byte to the receiver; The receiving end first receives the fixed-length data header, learns the length of the data to be received from the data header, and gets ready for circular reception.

With the help of struct module, any data type can be converted into bytes of fixed length. Therefore, with the help of this module, the total size of data sent by the sender can be packaged into bytes of fixed length through the struct module. After receiving, the receiver can obtain the total size of data sent by the sender and use circular receiving.

Make the header into a dictionary, the dictionary contains the details of the real data to be sent, then serialize the JSON, and then use struck to pack the serialized data into 4 bytes.

When sending: send the header length first, then encode the header content and send, and finally send the real content;

When receiving: first receive the length of the header, use struct to fetch it, receive the content of the header according to the length of the fetched header, and then decode and deserialize it. From the deserialization result, the detailed information of the data to be fetched is retrieved, and then the real data content is fetched.

Server code:

import subprocess
import struct
import json
import socket

server=socket(AF_INET,SOCK_STREAM)
server.bind(('127.0.0.1'.8083))
server.listen(5)

The server should do two things
# First thing: loop out the link request from the board connection pool, establish a two-way link with it, and get the link object
while True:
    conn,client_addr=server.accept()

    # 2: Get the link object and communicate with it
    while True:
        try:
            cmd=conn.recv(1024)
            if len(cmd) == 0:break
            obj=subprocess.Popen(cmd.decode('utf-8'),
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE
                             )

            stdout_res=obj.stdout.read()
            stderr_res=obj.stderr.read()
            total_size=len(stdout_res)+len(stderr_res)

            # 1. Make the head
            header_dic={
                "filename":"a.txt"."total_size":total_size,
                "md5":"1111111111"
            }

            json_str = json.dumps(header_dic)
            json_str_bytes = json_str.encode('utf-8')


            # 2. Send the length of your head first
            x=struct.pack('i'.len(json_str_bytes))
            conn.send(x)

            # 3. Hair information
            conn.send(json_str_bytes)
            # 4. Send real data again
            conn.send(stdout_res)
            conn.send(stderr_res)

        except Exception:
            break
    conn.close()
Copy the code

Client code:

import struct
import json
from socket import *

client=socket(AF_INET,SOCK_STREAM)
client.connect(('127.0.0.1'.8083))

while True:
    cmd=input(' 'CMD > >.).strip()
    if len(cmd) == 0: continue
    client.send(cmd.encode('utf-8'))

    # the receiving end
    # 1, collect 4 bytes and extract the length of the next header
    x = client.recv(4)
    header_len=struct.unpack('i',x)[0]

    # 2, receive header and parse
    json_str_bytes=client.recv(header_len)
    json_str=json_str_bytes.decode('utf-8')
    header_dic=json.loads(json_str)
    print(header_dic)
    total_size=header_dic["total_size"]

    # 3. Receive real data
    recv_size = 0
    while recv_size < total_size:
        recv_data=client.recv(1024)
        recv_size+=len(recv_data)
        print(recv_data.decode('utf-8'),end=' ')
    else:
        print(a)Copy the code

conclusion

The article was first published in the wechat official account program Yuan Xiaozhuang, and was synchronously published in Digijin and Zhihu.

Please explain where it came from. If you pass by, please put out your cute little finger and click like before you go (╹▽╹)