preface

This is an article I wanted to write for a long time. Recently, I happened to see a friend in the group sharing an article about Dubbo connection, which reminded me of this topic again. Today I want to talk to you about connection control in Dubbo. For those of you who may not be aware of connection control, the following configuration may be familiar:

<dubbo:reference interface="com.foo.BarService" connections="10" />
Copy the code

If you still don’t understand the usage of Dubbo connection control, you can refer to the official document: dubbo.apache.org/zh/docs/adv… Recently, Dubbo’s official documents have undergone a major change, and many familiar documents have almost failed to find where Orz is.

As we all know, duBBo protocol communication is long connections by default, and the connection configuration function is used to determine the number of long connections established between the consumer and provider. However, the official documentation only explains how to use this feature, but not when to configure connection control. This article will focus on this topic.

This article will also cover some topics related to long links.

use

To start with a simple demo built by Dubbo, start a consumer (192.168.4.226) and a provider (192.168.4.224) and configure their direct connection.

Consumer:

<dubbo:reference id="userService" check="false"
   interface="org.apache.dubbo.benchmark.service.UserService"
   url="Dubbo: / / 192.168.4.224:20880"/>
Copy the code

Provider:

<dubbo:service interface="org.apache.dubbo.benchmark.service.UserService" ref="userService" />
<bean id="userService" class="org.apache.dubbo.benchmark.service.UserServiceServerImpl"/>
Copy the code

Long connections are invisible things, and we need an observational job to “see” them. After starting the provider and consumer, you can use the following command to view TCP connections

  • For Mac:lsof -i:20880
  • Available under Linux:netstat -ano | grep 20880

Provider:

/ root ~ # netstat - ano | grep tcp6 20880 0 0 192.168.4.224:20880: : : * LISTEN off (0) / 0.00/0 tcp6 2502 0 192.168.4.216:20880 192.168.4.218:59100 ESTABLISHED OFF (0.00/0/0)Copy the code

Consumer:

/ root @ ~ # netstat - ano | grep tcp6 192.168.4.226:720 320 20880 59110, 20880:192.168.4.224 ESTABLISHED on (0.00/0/0)Copy the code

From the above observation we can discover several facts.

Just by starting the provider and consumer, the ABOVE TCP connection already exists, knowing that I did not trigger the call. That is, Dubbo’s default policy for building connections is at address discovery, not at invocation. Of course, you can modify this behavior by lazy-loading lazy=”true”, which delays the linking until it is called.

<dubbo:reference id="userService" check="false"
    interface="org.apache.dubbo.benchmark.service.UserService"
    url="Dubbo: / / 192.168.4.224:20880"
    lazy="true"/>
Copy the code

In addition, it can also be found that there is only one long connection between the consumer and the provider. 20880 is the default open port of the Dubbo provider, just like 8080 is the default open port of Tomcat, while 59110 is a randomly generated port of the consumer. (I have communicated with some friends before, and found that many people do not know that consumers also need to occupy a port)

Today’s main character “connection control” can control the number of long connections, for example, we can do the following configuration

<dubbo:reference id="userService" check="false"
    interface="org.apache.dubbo.benchmark.service.UserService"
    url="Dubbo: / / 192.168.4.224:20880"
    connections="2" />
Copy the code

Start the consumer again and observe the long connection

Provider:

/ root @ ~ # netstat - ano | grep tcp6 20880 0 0 192.168.4.224:20880: : : * LISTEN off (0) / 0.00/0 tcp6, 2508, 96 192.168.4.24:20880 192.168.4.26:59436 ESTABLISHED on (0.00/0/0) TCP6 5016 256 192.168.4.24:20880 192.168.4.26:59434 ESTABLISHED on (0.00/0/0)Copy the code

Consumer:

/ root @ ~ # netstat - ano | grep tcp6 20880 0 2520 192.168.4.226:59436 192.168.4.224:20880 ESTABLISHED on tcp6/0 (0.00/0) 48 1680 192.168.4.226:59434 192.168.4.216:20880 ESTABLISHED on (0.00/0/0)Copy the code

And you can see that there are two long connections here.

When do I need to configure multiple Long Links

Now we know how to do connection control, but when should we configure how many long connections? At this time, I can tell you that the specific production depends on the situation, but if you often read my public account, you will know that this is not my style, what is my style? Benchmark!

Before writing this article, I had a brief discussion on this topic with several colleagues and netizens. In fact, there was no conclusion, but the argument about the throughput difference between single connection and multi-connection. Refer to previous Dubbo Github issues, for example: github.com/apache/dubb… To be fair, I was skeptical of the PR discussion, and my view at the time was that multiple connections would not necessarily improve the throughput of the service (which was conservative, not so absolute).

Next, let’s use Benchmark. Test engineering is our old friend. Use Dubbo – Benchmark.

  • Test engineering address: github.com/apache/dubb…
  • Test environment: 2 Sets of Aliyun Linux 4C8G ECS

The test engineering is described in the previous article, but the test scheme is very simple. Two rounds of benchmark test connections=1 and connections=2 respectively, and observe the throughput of the test method.

Just do it, skip a bunch of testing steps, and just give the results.

connections=1

Benchmark Mode Cnt Score Error Units Client.createUser THRPT 3 22265.286 ± 3060.319 OPS /s client.existUser THRPT 3 33129.331 ± 1488.404 OPS/Client. THRPT 3 19916.133 ± 1745.249 OPS/Client. THRPT 3 3523.905 ± 590.250 ops/sCopy the code

connections=2

Benchmark Mode Cnt Score Error Units Client.createUser THRPT 3 31111.698 ± 3039.052 OPS /s client.existUser THRPT 3 THRPT 3 30647.173 ± 2551.448 OPS /s client.listuser THRPT 3 6581.876 ± 469.831 ops/sCopy the code

From the test results, it seems that the difference between single connection and multiple connection is very large, almost can be seen as 2 times! It looks like connection control works really well, but is it really?

After the first test of this scheme, I was not convinced by the results, because I had tested too many connections in other ways, and I had participated in the 3rd Middleware Challenge, so MY understanding of long connections was that most of the time, single connections tend to give the best performance. Even for hardware reasons, the gap should not be twice that. With this in mind, I began to research, is there something wrong with my test scenario?

Find problems with the test scheme

After a discussion with the Flash, his words finally helped me locate the problem.

I don’t know if you saw my conversation with the Flash and immediately identified the problem.

The biggest problem with the previous test scheme was that variables were not well controlled. As everyone knows, the number of IO threads actually used changed while the number of connections changed.

Dubbo uses Netty to implement long connection communication. When it comes to the relationship between long connection and IO thread, Netty’s connection model will be introduced here. In a word, Netty’s IO worker thread and channel are bound to one to many, that is, after a channel is connected, one IO thread will be responsible for all IO operations. Dubbo sets NettyClient and NettyServer worker thread groups:

The client org.apache.dubbo.remoting.transport.netty4.Net tyClient:

    private static final EventLoopGroup NIO_EVENT_LOOP_GROUP = eventLoopGroup(Constants.DEFAULT_IO_THREADS, "NettyClientWorker");
    
    @Override
    protected void doOpen(a) throws Throwable {
        final NettyClientHandler nettyClientHandler = new NettyClientHandler(getUrl(), this);
        bootstrap = new Bootstrap();
        bootstrap.group(NIO_EVENT_LOOP_GROUP)
                .option(ChannelOption.SO_KEEPALIVE, true)
                .option(ChannelOption.TCP_NODELAY, true) .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT); . }Copy the code

The DEFAULT_IO_THREADS in org. Apache. Dubbo. Remoting. The Constants are written in dead

int DEFAULT_IO_THREADS = Math.min(Runtime.getRuntime().availableProcessors() + 1.32);
Copy the code

On my 4C8G machine, the default is 5.

The service side org.apache.dubbo.remoting.transport.netty4.Net tyServer:

protected void doOpen(a) throws Throwable {
        bootstrap = new ServerBootstrap();

        bossGroup = NettyEventLoopFactory.eventLoopGroup(1."NettyServerBoss");
        workerGroup = NettyEventLoopFactory.eventLoopGroup(
                getUrl().getPositiveParameter(IO_THREADS_KEY, Constants.DEFAULT_IO_THREADS),
                "NettyServerWorker");

        final NettyServerHandler nettyServerHandler = new NettyServerHandler(getUrl(), this);
        channels = nettyServerHandler.getChannels();

        ServerBootstrap serverBootstrap = bootstrap.group(bossGroup, workerGroup)
        				.channel(NettyEventLoopFactory.serverSocketChannelClass());
								.option(ChannelOption.SO_REUSEADDR, Boolean.TRUE)
                .childOption(ChannelOption.TCP_NODELAY, Boolean.TRUE)
                .childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
                
 }
Copy the code

For example, we can use protocol to control the number of I/O threads on the server:

<dubbo:protocol name="dubbo" host="${server.host}" server="netty4" port="${server.port}" iothreads="5"/>
Copy the code

If this is not set, it is the same as the client logic: core + 1 thread.

Well, here’s the problem. Since I haven’t set up any IO threads, both the client and server have 5 IO threads enabled by default. If connections=1, Netty will bind channel1 to an IO thread. If connections=2, Netty will bind channel1 to an IO thread. Netty will bind channel1 and Channel2 to nettyWorkerThread-1 and NettyWorkerThread-2 in order, so that there will be two I/O threads working.

In actual production, the number of connections must be greater than the number of I/O threads in most distributed scenarios. Therefore, the number of channels in the test scenario is rarely less than the number of I/O threads.

The solution is simple, we need to control the variables, keep the number of I/O threads consistent, and just watch how the number of connections affects throughput. For servers, ioThreads =1 can be configured on the Protocol layer. For the client, because the source code is written dead, here I can only modify the source code, a local package, so that the client I/O thread number can also be specified by the -d parameter.

After the transformation, we got the following test results:

1 IO thread 1 Connection

Benchmark Mode Cnt Score Error Units Client.createUser THRPT 3 22265.286 ± 3060.319 OPS /s client.existUser THRPT 3 33129.331 ± 1488.404 OPS/Client. THRPT 3 19916.133 ± 1745.249 OPS/Client. THRPT 3 3523.905 ± 590.250 ops/sCopy the code

1 I/O thread 2 Connection

Benchmark Mode Cnt Score Error Units Client.createUser THRPT 3 21776.436 ± 1888.845 OPS /s client.existUser THRPT 3 31826.320 ± 1350.434 OPS /s client.getuser THRPT 3 19354.470 ± 369.486 OPS /s client.listuser THRPT 3 3506.714 ± 18.924 ops/sCopy the code

It can be found that simply increasing the number of connections does not improve the throughput of the service, and such test results are more consistent with my cognitive expectations.

conclusion

From the results of the above tests, some configuration parameters do not mean that the larger the better, I have also analyzed similar examples in multi-threaded file writing scenarios, only theoretical analysis + practical testing can draw a convincing conclusion. Of course, personal testing can also cause errors due to the omission of local key information. For example, if I do not find an implicit correlation between the number of I/O threads and the number of connections, it is easy to conclude that the number of connections is proportional to throughput. Of course, it does not necessarily mean that the final conclusion of this paper is reliable, perhaps not perfect, but also welcome everyone to leave comments and suggestions.

Finally, back to the original question, when should we configure connection control for Dubbo? According to my personal experience, most of the time, the number of connections is very much production environment, you can select a host, online through the netstat – ano | grep | wc -l to statistics, about 20880 is generally far in excess of the IO number of threads, there is no need to configure more multiplied the number of connections, The connection count and throughput do not grow linearly.

Having this capability in the Dubbo framework and actually needing it are two very different things, and I’m sure most readers are past the point where technical novelty drives projects? If one day you need to control the number of connections for a particular purpose, you’ll be amazed that Dubbo has this extension point.

Is It true that Dubbo’s connection controls are completely useless? Not really, my test scenarios are very limited and may have different effects on different hardware. For example, IN the 3rd Middleware Performance Challenge, I achieved the best result with 2 connections instead of single connections.

Finally, if you’re just using Dubbo to maintain your microservices architecture, most of the time you don’t need to worry about connection control. Take the time to move the bricks.