Welcome to github.com/hsfxuebao/j… , hope to help you, if you feel ok, please click on the Star
From: www.cnblogs.com/jixp/p/9811…
KafkaServer is the main class of Kafka server. The network layer related service components in KafkaServer include SocketServer, KafkaApis, and KafkaRequestHandlerPool, both of which use requestChannels exposed by SocketServer ) to handle network requests. SocketServer**** focuses on the communication protocol at the network layer. The specific service processing logic **** is entrusted to KafkaRequestHandler and KafkaApis****. SocketServer and these two components work together to complete a request processing in the following steps.
(1) The request sent by the client is forwarded to the processor for processing by the Acceptor.
(2) The handler places the request in the global request queue of the RequestChannel.
(3) KafkaRequestHandler fetches the client request in the request channel.
(4) Call KafkaApis for business logic processing.
(5) The KafkaApis send the response results to the response queue corresponding to the processor in the request channel.
(6) The processor retrieves the response result from the corresponding response queue.
(7) The processor returns the response result to the client, and the client finishes processing the request.
The SocketServer is a NIO service that starts one Acceptor thread and multiple processors. The NIO service uses a receiver thread to receive all client connection requests and distribute the received requests to different processors. This is a typical Reacto pattern because the server receives connections and requests from multiple clients. The Reactor pattern is designed to process connections and requests on separate threads so that the processing of requests does not block incoming connections. Otherwise, maintaining a connection for each client consumes server resources and degrades server performance. Using a Reactor pattern and a selector to manage multiple client connections reduces context switching and resource overhead.
Server resources are limited, the number of processors is not very large, and client connections are in the tens of thousands, so one processor can manage multiple client connections. The receiver’s job is simple: it accepts the client’s connection request and creates a SocketChannel that communicates with the client. Any read or write operations that occur on this SocketChannel are independent of the receiver, because it has passed the created SocketChannel to the processor, which is responsible for all operations on the channel.
The receiver thread registers the OP_ACCEPT event when it starts, and the receiver thread’s selector listens for the OP_ACCEPT event. The server handles the OP_ACCEPT event by getting the ServerSocketChannel bound to the select key, calling its Accept () method, and generating a network channel on the server to connect to the client.
As shown in the figure, after the server accepts the client connection, it creates a corresponding SocketChannel to communicate with the client. The Kafka channel through which a client connects to a server is actually a SocketChannel that communicates with the server. Both the client and server use selector mode for each channel. The steps from establishing a connection from the client to accepting a connection from the server are as follows.
(1) The ServerSocketChannel on the server registers OP_ACCEPT events with the selector. (2) The client registers the OP_CONNECT event with the selector and calls socketChannel.connect () to connect to the server. (3) The server selector listens to the connection event of the client and accepts the connection of the client. (4) the server using the ServerSocketChannel. The accept () to create a SocketChannel and client communication.
Other events (such as reads and writes) on both the client and server sides are similar in that the corresponding event must be registered before the selector can listen for certain types of events. The client selector handles connection/read/write events, while the receiver handles only connection events, which are handed over to the processor. In general, the client and server events are corresponding. When the client connects to OP_CONNECT, the server accepts OP_ACCEPT, when the client writes OP_WRITE, the server reads OP_READ, and when the server writes OP_WRITE, the client reads OP_READ.
The processor uses the selector’s polling to handle network requests. The receiver allocates the client’s Socketchannels to multiple processors in a round-robin fashion. Each processor has multiple Socketchannels, and the processor also uses a selector to manage multiple client connections. When the processor accepts a new SocketChannel connection, it puts it on a blocking queue and then wakes up the selector thread to work. A client creates a Kafka channel when it connects to a server, where the server processor creates a Kafka channel for SocketChannel. The configueNewConnections () method registers read events for SocketChannel, creates a Kafka channel, and binds the Kafka channel to the SocketChannel select key. The NetworkClient on the client and the processor on the server send the request and receive the response, both triggered by the selector polling. Before polling, the client needs to prepare requests to be sent; The server needs to prepare a response to send and poll for the completed send and receive.
The interaction between the client and the server is driven by round-robin selector polling. The steps to string together a complete request and response process are as follows, combining polling on the client and server side with the selector. (1) The client completes sending the request, and the server polls the request sent by the client. (2) The server receives the request sent by the client, processes the service, and prepares the response result for sending. (3) The server completes sending the response, and the client polls the response sent by the server. (4) The client receives the response sent by the server
Request channels Request queues and response queues In KafkaServer, SocketServer request channels are sent to the Kafka request processing thread (Kafka request Processing thread) and KafkaApis. The request channel is where the processor exchanges data with the request processing thread and KafkaApis. If the processor adds a request to the request channel, both the request processing thread and KafkaApis can retrieve the request from the request channel. If the request processing thread and the KafkaApis add a response to the request channel, the processor can also get the response from the request channel.
Because the request channel holds both request and response types of queues, its various methods receive and send requests and responses in order: send request → receive request → send response → receive response.
(1) sendRequest () : the processor puts the request into the request queue after receiving the request from the client. ReceiveRequest () : The request processing thread gets the request from the queue and submits it to KafkaApis for processing. (3) sendResponse () : after KafkaApis are processed, put the response result into the response queue. (4) receiveResponse () : The processor obtains the response result from the response queue and sends it to the client.
The above is only a request and response in the request channel call sequence, to the server side processing multiple client requests as an example, and combined with other related components, to illustrate the process of the processor to put the request into the request channel, until the response from the request channel. As shown, because a SocketServer has multiple processors, each processor is responsible for a portion of the client’s requests. If request 1 is sent to processor 1, then the response corresponding to request 1 can only be sent to processor 1, so each processor has a response queue. The request queue is shared globally by all processors, although multiple request-processing threads process client requests in the request queue at the same time. Suppose processor 3 has two client requests, which may be processed by different request-processing threads after they enter the global request queue. Finally KafkaApis**** puts the responses of both requests into the corresponding response queue of processor 3.
The request processing thread KafkaServer creates a request processing thread pool (KafkaRequestHandlerPool) in which multiple request processing threads (KafkaRequestHandler) are created and started. The global request channel in SocketServer is passed to each request processing thread. Thus each request processing thread collectively consumes all client requests in the same request channel. Each request-processing thread receives a request and submits it to the unified KafkaApis for processing. Note: a KafkaServer has multiple request processing threads, but only one KafkaApis.
To summarize the components related to the server and network layer: One Acceptor thread, one processor, one RequestChannel, one RequestQueue, and one response sequeue ), a connection pool of request processing threads (KafkaRequestHandlerPool), multiple request processing threads (KafkaRequestHandler), a server request portal (KafkaApis).
Request processing entry on the server Client requests are sent by the request processor to KafkaApis, which are responsible for specific service logic processing. KafkaApis receive the request, execute the service logic, and send the response result corresponding to the request to the response queue of the request channel. The KafkaApis.handle () method is an entry point for the server to process various requests. Not only clients (such as producers or consumers), communication between nodes on the Kafka server also goes through this unified portal (such as pull requests for backup copies, and other internal requests).
After receiving the NetworkReceive object from the client, the processor parses the NetworkReceive content, adds the current **** manager number, and installs the package **** into a RequestChannel. Request object. Requestchannel.request is then placed in the Request queue for the Request channel. The request object sent by the client is Cli****entRequest, but it is packaged as Send object ** when it is sent to the server over the network. The server reads the client’s request by polling the selector in the processor and gets the NetworkReceive object. ** The server needs to parse NetworkReceive to create a RequestChannel.Request object. ** The request processing thread gets the request from the request channel and hands it to KafkaApis for processing. After processing the request, KafkaApis**** creates a Response object (requestchannel.response) and places it in the RequestChannel. The response object also holds the reference **** of the request object. Because the request object has a handler number, the response object can get the handler number from the request object, ensuring that both the request and response are processed in the same handler