Abstract: Kafka network module Server, introduces the Server start, receive requests and processing requests process.

This article is shared by Huawei cloud community “Kafka network module -Server side”, the original author: middleware brother.

SocketServer is a module on the Kafka server that processes requests. It is created, initialized, and started during Kafka startup.

SocketServer startup process:

  • Initialize acceptors according to the endpoint order, one Acceptor for each endpoint, and create processors for each Acceptor (the number of acceptors is determined by the num.net work.Threads configuration item). When an Acceptor is started, it listens for a connection using a selector and sends the new connection to the Processor.

  • Start all Processors

  • Acceptor connection process:

  • After an Acceptor is started, a serverSocketChannel is created, listens on the endpoint corresponding to the Acceptor, registers OP_ACCEPT with the selector, and an infinite loop is created. Get the ready key (the previously registered serverSocketChannel) by selector, indicating that a connection has arrived, and then create a socketChannel corresponding to that connection by accept(). We then poll one of the processors that this acceptor is responsible for, and pass the socketChannel to the processor that we want to process and the connection to.

  • An Acceptor sends a connection to the Processor by adding a socketChannel to the Processor’s connection queue newConnection, which the Processor continuously obtains and processes in the run method.

  • After the Processor obtains the socketChannel from newConnection, it registers OP_READ with the selector, and creates the corresponding KafkaChannel.

The Server receives the request and processes it:

  • After the Processor receives the OP_READ event, it checks and tries to complete the SSL handshake and SASL verification. The handshake may not be complete. Therefore, after the Processor receives the OP_READ event, it checks to ensure that the handshake is complete. SSL/SASL reference section 9.4)

  • After the SSL handshake and SASL verification is complete, the data is read from the channel, a NetworkReceive object is constructed, and merged into stagedReceives

  • Extract the lead element from stagedReceives (removed) and add complete Receives

  • Remove elements from completedReceives (not removed), construct the Request object, add to the requestQueue, remove event registration to OP_READ, and assign the corresponding KafkaChannel to a fraternal, Buy for MUTED_AND_RESPONSE_PENDING again

  • The KafkaRequestHandler takes the element from the requestQueue (removed) and gives it to the KafkaApi module to process the request

  • After processing the request, the KafkaApi puts the response into the responseQueue and inflightResponses of the corresponding processor and wakes up its selector

  • The Processor removes the response from the responseQueue. If the response needs to be sent back to the client, it assigns the send value to KafkaChannel and registers the OP_WRITE event

  • When a channel is ready to write, send is written to the channel’s buffer. After sending is finished, remove the registration of the OP_WRITE event and add send to the completedSends

  • Remove the corresponding response from inflightResponses, perform the response callback, switch KafkaChannel to a fraternal, then to a not_energy-intensive, and re-add the OP_READ event register

Click to follow, the first time to learn about Huawei cloud fresh technology ~