Hi, I’m crooked.
Yes, I recently saw this article “ICBC distributed Service C10K Scenario Solution”.
Why is that?
Because this article at the beginning of the release of the time I read, at that time feel very good to write, the universe (industrial and Commercial Bank) is really diao look.
But once I read it, I didn’t go to the detail.
When I saw it this time, IT happened to be on the way off work, so I carefully watched it again.
Well, often read often new, or very productive.
So write an article to report what I learned after reading it again.
The paper feed
I know many students have not read this article, so I put a link first, “ICBC Distributed Service C10K Scenario solution”.
First to refine the content of the article, but if you have time, you can also read this article carefully first, feel the strength of the universe.
The article goes something like this.
In the cosmic architecture, as the business evolves, there will be a scenario in which providers serve thousands or even tens of thousands of consumers for the foreseeable future.
Under such a high load, if the server program is not well designed, the network service may be inefficient or even completely broken when handling tens of thousands of client connections, which is known as THE C10K problem.
The C10K problem will not expand to talk about, look up on the Internet, very famous program related problems, but the problem has become history.
The universal RPC framework uses Dubbo, so their article is based on this question:
Can a Distributed service platform based on Dubbo handle complex C10K scenarios?
To do this, they set up a large-scale connection environment, simulated service invocation, and conducted a series of explorations and validations.
First they use Dubbo version 2.5.9. The version is indeed a little low, but the bank, as we all know, architecture upgrade can not move, stable operation is king.
In this version, they built a server, the logic of the server was sleep 100ms, simulated business calls, and deployed on an 8C16G server.
The corresponding consumer sets the service timeout time to 5s, then deploys the consumer on hundreds of 8C16G servers (my goodness, hundreds of 8C16G servers, it is all money, it is good to have money), deploys 7000 service consumers in container mode.
Each consumer invokes the service once a minute after it is started.
Then they customized two test scenarios:
In scenario 2, the exception is inevitable because there is only one provider and the consumer is still making requests during the restart, which is bound to be cold.
But scene 1 is not supposed to.
You can imagine that the consumer has configured a timeout of 5s while the provider’s business logic only processes 100ms. Time is enough to say the least.
It should be added that this article also focuses only on scenario 1.
But, my friends, but ah.
Although the frequency of callers sending a request once a minute is not very high, but let’s say there are 7000 callers. These 7000 callers, this is the legendary burst of traffic, but the “burst” is once a minute.
So, the occasional timeout is also understandable, after all, server processing capacity is limited, there are tasks in the queue for a little time out.
I can write a small example to illustrate this. It looks like this:
I’m just going to have a thread pool with 200 threads. I then submitted 7000 tasks that took 100ms each, simulating concurrency with CountDownLatch, and ran in 3.8s on my 12-core machine.
So if, in the Dubbo scenario, each request is combined with a little bit of network time, a little bit of in-frame consumption, and that little bit of time is multiplied by 7000, the number of tasks executed could theoretically exceed 5s.
So occasional timeouts are understandable.
But, my friends, here we go again.
I’ve been talking about theory, but practice is the only way to test the truth.
Take a look at the cosmic test:
First of all, we can see that the consumer is very fast in both initiating the request and processing the response, but the bottleneck is between the time the server receives the request and the time it processes it.
After packet capture analysis, they concluded that the cause of transaction timeout was not on the consumer side, but on the provider side.
This conclusion is also easy to understand, because the pressure is on the side of the service provider, so the blocking should be on its side.
In fact, at this point we can almost confirm that there must be some operation in the Dubbo framework that is causing the increase in time.
The hard part is figuring out, what is the operation?
Through a series of operations and careful analysis, a conclusion has been reached:
The netty worker thread is busy due to the heavy heartbeat, which increases the transaction time.
Which is the point in the conclusion:
With a conclusion, it’s easy to find the lesion, and the right medicine.
As mentioned earlier, this article focuses only on scenario 1, so let’s look at the proposed solution for scenario 1 universe:
It’s all about optimizing the heartbeat, and the results are as follows:
The most effective operation is “Heartbeat Bypass serialization.”
The average processing time difference between the consumer and provider decreased from 27ms to 3m, an increase of 89%.
The time spent on the top 99% of transactions decreased from 191ms to 133ms, an increase of 30%.
Well, writing this is almost a retelling of something I read in that article at the time, which is not very nutritious.
It’s just that I remember when I first read the article, I went something like this:
I think it’s pretty cool that a small heartbeat in the C10K scenario turns into a performance problem.
I had to look into it. By the way, the most important solution was “heartbeat bypass serialization”. I also had to look into how Dubbo implemented this feature.
But…
I can’t remember why I didn’t watch it, but it doesn’t matter, I remember now, and I’ll start working on it.
How does a heartbeat bypass serialization
How do I do that?
Directly into the source code?
Yes, is to the source code inside the rush.
But before rushing, I did a quick trip to Dubb’s Github:
Github.com/apache/dubb…
I did a first search for “Heartbeat” in the Pull Request and found some good things:
My eyes lit up when I saw the two PR’s.
Boy, I was just going to look around, but I didn’t realize I had a direct location on what I was studying.
I only need to look at these two PR’s to see how the “heartbeat bypass serialization” works, which immediately saves me a lot of twists and turns.
First look at this:
Github.com/apache/dubb…
As you can tell from this description, I’ve found the right place. From his description, we know that “heartbeat skip serialization” is used to replace the serialization process with NULL.
At the same time, this PR also explains its transformation ideas:
I’ll take you through the submitted code.
What do you think?
You can see the file corresponding to this commit on Git:
To the source code to find the corresponding place, this is also a way to find the source code.
I’m familiar with the Dubbo framework, and I probably know where to find the code without looking at the PR. But what about a different framework that I’m not familiar with?
Git is actually a good place to start.
A small tip to read the source code, for you.
It doesn’t matter if you don’t know the Dubbo framework, we’ll just focus on one point: how heartbeats skip serialization. As to who and how and when the heartbeat is initiated, this section is left out for the moment.
Next, we start with this class:
org.apache.dubbo.rpc.protocol.dubbo.DubboCodec
We can see from the commit record that there are two main changes, and the code for both changes is exactly the same, both in the decodeBody method, but one in the if branch and one in the else branch:
What does this code do?
If you think of an RPC call, it must involve encode and decode of the message, so this is mainly to decode the request and response message.
One heartbeat, back and forth, one request, one response, so there are two changes.
So I’ll just show you how to handle the request packet:
You can see that after the code modification, a special judgment is made on the heartbeat packet.
There are two methods involved in the special handling of heartbeat events, both of which are new methods for this submission.
The first method is like this:
org.apache.dubbo.remoting.transport.CodecSupport#getPayload
The InputStream stream is replaced by an array of bytes, which is then passed as an input parameter to the second method.
The second method looks like this:
org.apache.dubbo.remoting.transport.CodecSupport#isHeartBeat
You also know from the method name that this is to determine whether the request is a heartbeat packet.
How to determine it is a heartbeat packet?
The first thing you have to look at is where the heartbeat starts:
org.apache.dubbo.remoting.exchange.support.header.HeartbeatTimerTask#doTask
From where the heartbeat originated, we know that what it sent out was null.
So where the packet is received, check whether its contents are null. If so, it is a heartbeat packet.
With these two simple methods, the serialization of heartbeat skips is done and performance is improved.
The above two methods are in this class, so the core changes are still in this class, but there are not many changes:
org.apache.dubbo.remoting.transport.CodecSupport
There are two small details in this class that I can take you through again.
First up here:
This map caches null values for different serialization methods, and this code does exactly what the author says:
Another detail is to look at the class’s commit record:
There was another optimized commit, and this time it looked like this.
We define a ThreadLocal and initialize it to 1024 bytes:
So what is this ThreadLocal for?
When reading InputStream, we need to create an array of bytes. To avoid frequent creation and destruction of this byte, we need to create a ThreadLocal:
ThreadLocal does not call the remove method.
No, my friends, the NIO thread that executes this in Dubbo is reusable and contains only a 1024 byte array with no dirty data, so it doesn’t need to be removed and reused.
It is the reuse that improves performance.
That’s the details. The devil is in the details.
This detail is another PR mentioned earlier:
Github.com/apache/dubb…
Here we can see how the universe can make the heartbeat skip serialization. There is no complicated code, just a few dozen lines of code.
But, my friends, it’s buts again.
As I write this, I suddenly feel something is wrong.
Because I wrote this article before, the Dubbo deal shit.
In this article, there is a picture like this:
This was taken from the official website at the time.
In the protocol, the event identifier field is preceded by 0 and 1.
But now it’s different, because if you look at the code, it’s going to expand the range of 1, which doesn’t necessarily mean the heartbeat, because there’s an if-else in there
So, I went to look at the description of the agreement on the official website.
Dubbo.apache.org/zh/docs/v3….
Sure enough, something changed:
This is not to say that 1 is a heartbeat packet, but to say that 1 May be a heartbeat packet.
Rigor, that’s rigor.
Therefore, open source projects are not just finished code changes, but also take into account the maintenance of some peripheral information.
A variety of designs for heartbeat
I also found one while studying Dubbo’s heartbeat.
Github.com/apache/dubb…
Here’s the headline:
This translates to the suggestion to use IdleStateHandler instead of using Timer to send heartbeat.
I fixed my eyes to see, good opportunity, this is not 95 old xu, old acquaintance.
Let’s have a look at what Lao Xu said. His suggestions are as follows:
Several Dubbo leaders exchanged many ideas in this PR, and I benefited a lot after careful reading.
You can also click in to have a look, I report to you here their harvest.
First of all, a few elder brothers in the real-time heartbeat of a battle.
All in all, we know that Dubbo’s heartbeat detection has a certain delay, because it is based on the time wheel, which is quite a timed task, and the timeliness of the trigger cannot guarantee the real-time trigger.
It’s like if you have a timed task that executes every 60 seconds, at 0 seconds the task starts, at 1 second there’s a data ready, but it’s not processed until the next task triggers. Therefore, your maximum delay for processing data should be 60 seconds.
That should make sense to you.
As a side note, the result of the discussion above is “heartbeat latency is currently 1/4”, but I looked at the source code for the latest master branch and it felt like 1/3 latency:
The HEARTBEAT_CHECK_TICK parameter is 3 when calculating the time. So I understand it’s 1/3 delay.
But it doesn’t matter, it doesn’t matter, you know there’s a delay.
Kexianjun believes that if the Netty-based IdleStateHandler does this, each detection timeout will be recalculated to the next detection time, so it can be relatively timely to check the timeout.
This is an optimization in real time.
Xu felt that, in addition to the real-time consideration, in fact, IdleStateHandler is an elegant design for heartbeat. However, since it is based on Netty, there is no way to change the communication framework when it is not using Netty, so we can keep the design of Timer to deal with this situation.
Soon after, Carryxyh gave constructive advice:
Dubbo supports multiple communication frameworks.
By “multiple,” I forgot to mention that in addition to Netty, it also supports Girzzly and Mina, two underlying communication frameworks, as well as customization.
But I think it’s 2021. Are Girzzly and Mina still available?
They can also be found in the source code:
org.apache.dubbo.remoting.transport.AbstractEndpoint
Girzzly, Mina, and Netty each have their own servers and clients.
There are two versions of Netty, because Netty4 is a bit too big to be compatible with previous versions, so it’s better to create an implementation.
But no matter how it changes, it’s still called Netty.
All right, back to the constructive comments.
If you use IdleStateHandler for the heartbeat, and other communication frameworks keep Timer mode, you’ll get code like this:
if transport == netty {
don't start heartbeat timer
}
Copy the codeThis is something that shouldn’t happen in an open source framework because it adds complexity to the code.
Therefore, his suggestion is that it is better to use the same method for heartbeat detection, that is, using the Timer mode.
Just when I thought what he said was reasonable, I read Xu’s answer, and suddenly I felt what he said was also reasonable:
I don’t think I need to explain, just read it and think about it.
And here’s What Carryxyh had to say:
This is when the opposite appears.
Lao Xu’s point of view is that heartbeat is definitely necessary, but he thinks different communication frameworks need not keep the same way of implementation (now they are all based on the way of Timer time wheel). He does not think it is an elegant design for Timer to abstract into a unified concept to realize connection preservation.
In Dubbo we mainly use Netty, and Netty’s IdleStateHandler mechanism is naturally used for heartbeat.
So, PERSONALLY, I think he thought using IdleStateHandler was a more elegant way to implement it first, and then improved timeliness.
However, Carryxyh thinks that the Timer is abstract, which is a very good design. Because of its existence, we can care not about whether the bottom layer is Netty or MINA, but only about the concrete implementation.
For the IdleStateHandler solution, he still thinks it has an advantage in terms of timeliness. But PERSONALLY, I think his idea is that if it really has an advantage, we can refer to its implementation method and assign other communication frameworks with the function of “Idle”, so as to achieve great unification.
Looking at this, I think the two brothers battle point is like this.
The first premise is all around the “heartbeat” function.
One thinks that “heartbeat” is better implemented when using Netty, and that Netty is Dubbo’s main communication framework, so it should be possible to just change the Implementation of Netty.
An implementation of “heartbeat” should be unified, and if Netty’s IdleStateHandler scheme is a good one, we should take it over.
I think it all makes sense, but I don’t know who to vote for.
But what finally convinced me to vote for Lao Xu was this article he wrote: “One Heartbeat, Two Designs”.
In this article, he wrote the evolution process of Dubbo heartbeat in detail, which also involves part of the source code.
He ended up with a diagram like this, a comparison of heartbeat designs:
And then, this:
Lao Xu is engaged in middleware in Ali, the original engaged in middleware people think about these things every day.
That’s interesting.
Look at the code
Take a look at the code, but don’t do a detailed analysis, rather point the way, if you want to know more about it, go to the source code.
First up here:
org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient
You can see that the startHeartBeatTask method is called in the constructor of HeaderExchangeClient to start the heartbeat.
And there’s a HashedWheelTimer, which I’m familiar with, the time wheel, which I’ve analyzed before.
Then we look at this method startHeartBeatTask:
There is no complicated logic to building a heartbeat task and then throwing it into a time wheel.
This implementation is Dubbo’s default handling of heartbeats.
CanHandleIdle (canHandleIdle, canHandleIdle, canHandleIdle, canHandleIdle);
So, the result of the previous if test is true.
When is canHandleIdle true?
True when Netty4 is used.
That is, Netty4 does not walk the default set of heartbeat implementation.
So how does it work?
Since the thinking on the server and client is the same, let’s just look at the code on the client side.
Take a look at its doOpen method:
org.apache.dubbo.remoting.transport.netty4.NettyClient#doOpen
The IdleStateHandler event is added to the pipeline. This event will trigger a method if the heartbeatInterval has no read/write event within milliseconds, which is equivalent to a callback.
The default heartbeatInterval is 6000, which is 60s.
And then you add the nettyClientHandler, what does it do?
Take a look at this method:
org.apache.dubbo.remoting.transport.netty4.NettyClientHandler#userEventTriggered
This method is sending heartbeat events.
If the client has not been triggered for 60 seconds, Netty will trigger the userEventTriggered method. In this method, we can send a heartbeat to check whether the server is normal.
From the current code, Dubbo ultimately follows Xu’s recommendations, but the default implementation remains the same, with the IdleStateHandler mechanism in Netty4.
Which makes it even weirder to me, actually.
Also for Netty, one uses the time wheel and the other uses the IdleStateHandler.
At the same time, I also understand that the step should not be too big, easy to pull the egg.
However, in the process of looking through the source code, I found a small problem in the code.
org.apache.dubbo.remoting.exchange.codec.ExchangeCodec#decode(org.apache.dubbo.remoting.Channel, org.apache.dubbo.remoting.buffer.ChannelBuffer, int, byte[])
In the above method, there are two lines of code that look like this:
Forget what they do, and I’ll show you how they work:
You can see that both methods execute this logic:
int payload = getPayload(channel);
boolean overPayload = isOverPayload(payload, size);
Copy the codeIf finishRespWhenOverPayload returned is not null, nothing to say, return to return, do not perform checkPayload method.
If finishRespWhenOverPayload returns null, will perform checkPayload method.
In this case, the operation of checking the size of the packet is repeated.
Therefore, I think this line of code is redundant and can be deleted directly.
You know what I mean?
Another opportunity to contribute source code to Dubbo. Here you go.
Finally, I will send you some reference materials.
The first is to understand the heartbeat mechanism of SOFA-RPC. SOFA-PRC is also an open source framework for Ali.
The implementation of the heartbeat is completely based on the IdleStateHandler.
Check out these two official articles:
www.sofastack.tech/search/?pag…
The second one is geek time “Learning Microservices from Zero”. In lecture 17, the teacher mentioned a protection mechanism in a little sharing about heartbeat, which I had never thought of before:
Anyway, I think that some of the links mentioned in my article, you have to carefully read, so for the heartbeat of this piece of things, also grasp the seven, eight, eight, enough.
All right, that’s it.
This article has been included in personal blog, welcome to play.
www.whywhy.vip/