1. The background

As the previous big data environment is built by Docker, no matter in learning, testing, migration is undoubtedly the fastest way, especially when you want to use the same set of environment in multiple environments,github.com/hulichao/do… , build ideas, you can refer to this warehouse, in general, as long as the attention than the virtual machine to do a port mapping, port mapping is also the network, in fact, is relatively redundant, so in the establishment of Kafka environment when just encountered the following problems, Connection to node 0 Colud not be establised. Broker may not be avaliable

Kafka does not connect to the broker, which is fine on the machine of the virtual machine, but the local use of IDEA connections is problematic.

2. The train of thought

Is written after the blog, so at that time some screenshots of troubleshooting problems, not a post, just briefly describe the process and the final solution, and elicits Kafka on the internal and external network isolation when the solution

2.1 First, troubleshoot various network problems

For example, for the virtual machine network, ping www.baidu.com is used to check the port, Telnet Linux121 9092 is used to check the port, and no problem is found. In addition, Docker’s virtual network adapter is checked, which can refer to another article, blog.csdn.net/hu_lichao/a… And found no problem

2.2 Then start thinking about Kafka

I didn’t think in this direction, but finally I had to use Google to input the keyword Kafka Docker idea and failed to connect to the Broker, as shown in the figure below

Then refer to # 1 and read the key message: Kafka’s clients (producer and consumer) need to go through a layer of authentication to connect to the Broker, without which they cannot connect! A client connection to Kafka requires two layers of authentication: the connection and port configured by advertised. Listeners The catch here is that the PLAINTEXT content must be compared to the Java application as it is, regardless of how you got the kafka connection in the virtual machine Docker. The content of the bootstrap.servers that you send should be consistent with the PLAINTEXT. This requires configuring the hosts of Linux121 to map to the actual IP and port.

Java code

Zk gets broker information

Configuration of Kafka’s server.properties

3. Extranet and Extranet parameters of Kafka

According to the troubleshooting above, there are two important parameters, listeners and advertise. What are these two parameters used for? The official translation is taken here.

Specifies the address and port at which the current Broker publishes services. With advertised. Listeners, used to do the internal and external network isolation.

Here are some related parameters:

listener.security.protocol.map

Extranet isolation configuration: Configures the mapping between listener names and security protocols. For example, you can isolate extranets and extranets, even if they both use SSL. Listener. Security. Protocol. The map = INTERNAL: SSL, EXTERNAL: SSL note that the name of each listener can only appear in the map again. inter.broker.listener.name

Name of the listener used to configure communication between brokers, which must be in the Advertised.Listeners list. Intel, for example, the broker. The listener. Name = EXTERNAL listeners

This is used to configure the list of URIs and listener names that the broker listens to. Use commas to separate multiple URIs and listener names. If the listener name representative is not a secure protocol, you must configure the listener. The security. The protocol. The map. Each listener must use a different network port. Advertised the address to The ZooKeeper client for use. If the configuration of the address used by the client is different from that of the listeners. It can be found in ZooKeeper’s get /brokers/ids/

.

In an IaaS environment, the network interface of this entry must be different from the network interface bound to the broker. If this entry is not set, the configuration for Listeners is used. Unlike Listeners, this entry cannot use 0.0.0.0 network port. The address must be configured in or part of the Listeners configuration.

Note: The function of INTERNAL and EXTERNAL is to alias the PLAINTEXT protocol. By default, there is only one protocol with the same name in the map, so write it separately

Here is a typical common configuration:

The above is written with Wolai notes. You can use my invitation code DJED4WI to register. Seats are limited.

4. Reference

  • www.cnblogs.com/hellxz/p/wh…