A detailed explanation of ant Financial distributed link component SOFATracer buried point mechanism
SOFATracer is a component used for distributed system call tracing. It uses unified TraceId to record various network calls in the invocation link in a log to achieve the purpose of perspective network calls. The link data can be used for fault discovery and service governance.
Making the address: https://github.com/sofastack/sofa-tracer/pulls (star) welcome to the official documents address: https://www.sofastack.tech/projects/sofa-tracer/overview/
From the end of 2018 to the beginning of 2019, the SOFA team launched a series of articles on parsing – Sofatracer – framework source code. In this series, the basic capabilities provided by SOFATracer and the implementation principle have done a more comprehensive analysis, interested students can take a look.
According to official documentation and PR, SOFATracer currently supports buried support for the following open source components:
- Spring MVC
- RestTemplate
- HttpClient
- OkHttp3
- JDBC
- Dubbo (2.6/2.7)
- SOFARPC
- Redis
- MongoDB
- Spring Message
- Spring Cloud Stream (Buried point based on Spring Message)
- RocketMQ
- Spring Cloud FeignClient
- Hystrix
Most of the capabilities are available in 3.x version. As you can see from the official issue of 2.x version, new features will not be provided in the future; This is also related to SpringBoot’s announcement that it will no longer maintain the 1.x version.
This article will analyze from the perspective of plug-ins, SOFATracer is how to achieve the above components buried; In this article, in addition to understanding SOFATracer’s buried point mechanism, you can also learn a little about the basic extension mechanism and basic principles of the above components.
The standard Servlet specification buried point principle
SOFATracer supports web MVC burying points for standard Servlet specifications, including regular servlets and Springmvc. The basic principle is based on the extended implementation of the Javax.servlet.filter Filter interface provided by the Servlet specification.
The filter sits between the client and the Web application and is used to examine and modify the request and response information that flows between the two. The filter intercepts the request before it reaches the Servlet. The filter intercepts the response before it is sent to the client. Multiple filters form a FilterChain. The sequence of different filters in the FilterChain is determined by the sequence of filter mappings in the deployment file web. The filter that picks up the client request first will pick up the Servlet response last.
The Web application is usually the receiver of the request. In Tracer, the application exists as a server. The event corresponding to the SpanContext resolution is server Receive (SR).
SOFATracer parses and generates spans in sofA-Tracer-SpringMVC-plugin as follows:
- The Servlet Filter intercepts the Request
- Resolve SpanContext from the request
- Build the span of the current MVC using the SpanContext
- Add a tag and log to the current span.
- At the end of the filter processing, end the span.
Of course, there are many other details to design, such as which tag attributes to set for the SPAN, how to handle asynchronous thread passthrough, and so on. This article does not go into details, interested students can read the code or communicate with me.
Dubbo buried point principle
Dubbo burying point actually provides two plug-ins in SOFATracer to support Dubbo 2.6.x and Dubbo 2.7.x. The Duddo buried point is also based on the Filter, which is an implementation of the SPI extension-call interception extension mechanism provided by Dubbo.
Buried points of RPC frameworks such as Dubbo or SOFARpc usually need to be considered. First, RPC frameworks are divided into client and server, so the client and server of RPC must be distinguished during buried points. In addition, there are many ways to call RPC, such as common synchronous call, asynchronous call, Oneway and so on. Different call ways will result in different end times of span. It is important that basically all RPC frameworks will use thread pools to initiate and process requests. It is also important to ensure that tracers do not string in a multithreaded environment.
In addition, Dubbo 2.6.x and Dubbo 2.7.x are quite different in asynchronous callback processing. Dubbo 2.7.x provides onResponse method (later upgraded to Listener, including onResponse and onError methods). Dubbo 2.6.x does not provide the corresponding mechanism, only through the future hard code processing to complete the burying and reporting.
Zipkin Brave also failed to consider this problem when it was embedded in Dubbo 2.6.x. It was discovered and fixed when SOFATracer supported Dubbo 2.6.x.
DubboSofaTracerFilter class provided with SOFATracer:
@Activate(group = { CommonConstants.PROVIDER, CommonConstants.CONSUMER }, value = "dubboSofaTracerFilter", order = 1)
public class DubboSofaTracerFilter implements Filter {
// todo trace
}
Copy the codeThe core code for SOFATracer to handle asynchronous callback handling in Dubbo 2.6.x:
Dubbo asynchronous processing depends on the ResponseFuture interface, but the ResponseFuture does not exist on the core link as data or a list, so only one ResponseFuture exists on the core link. So if I were to create a custom class that implements the ResponseFuture interface, I wouldn’t be able to do that because I would have a run-time problem overriding ResponseFuture. So by design, SOFATracer will build a new FutureAdapter out of ResponseFuture for delivery.
boolean ensureSpanFinishes(Future {
boolean deferFinish = false;
if (future instanceof FutureAdapter) {
deferFinish = true;
ResponseFuture original = ((FutureAdapter<Object>) future).getFuture();
ResponseFuture wrapped = new AsyncResponseFutureDelegate(invocation, invoker, original);
// Ensures even if no callback added later, for example when a consumer, we finish the span
wrapped.setCallback(null);
RpcContext.getContext().setFuture(new FutureAdapter<>(wrapped));
}
return deferFinish;
}
Copy the codePrinciple of BURYING HTTP clients
HTTP client buried points include HttpClient, OkHttp, RestTemplate, etc. Such buried points are generally implemented based on the interceptor mechanism. For example, HttpRequestInterceptor and HttpResponseInterceptor are available from HttpClient. OkHttp uses the okhttp3.Interceptor; RestTemplate using ClientHttpRequestInterceptor.
Taking OkHttp as an example, the implementation principle of HTTP client burying point is briefly analyzed:
@Override
public Response intercept(Chain chain) throws IOException {
// Get the request
Request request = chain.request();
// Parse out the SpanContext and build the Span
SofaTracerSpan sofaTracerSpan = okHttpTracer.clientSend(request.method());
// Initiate a specific call
Response response = chain.proceed(appendOkHttpRequestSpanTags(request, sofaTracerSpan));
/ / the end of the span
okHttpTracer.clientReceive(String.valueOf(response.code()));
return response;
}
Copy the codeDatasource Buried point principle
As with the standard servlet specification implementation, all DataSource implementations based on javax.sql.DataSource can be buried using SOFATracer. Because DataSource doesn’t provide filters or interceptors like servlets, SOFATracer can’t directly bury-point in the normal way (Filter/SPI extension interceptors/interceptors, etc.), but instead uses proxy mode.
The diagram above shows the class inheritance architecture of SOFATracer’S DataSource proxy class. Javax.sql.DataSource interface. SmartDataSource is the only subclass of BaseDataSource. This is the proxy class used in SOFATracer. So if you use the SOFA – Tracer-datasource -plugin, Can see the runtime type of the Datasource is com. Eventually alipay. Sofa. Tracer. Plugins. The Datasource. SmartDataSource.
public abstract class BaseDataSource implements DataSource {
// The actual datasource of the agent
protected DataSource delegate;
// Sofatracer custom interceptor, used to intercept connection operations, DB operations, etc
protected List<Interceptor> interceptors;
protected List<Interceptor> dataSourceInterceptors;
}
Copy the codeInterceptor comes in three main types:
Take StatementTracerInterceptor StatementTracerInterceptor will will be blocked to all PreparedStatement interface method, the code is as follows:
public class StatementTracerInterceptor implements Interceptor {
// The tracer type is client
private DataSourceClientTracer clientTracer;
public void setClientTracer(DataSourceClientTracer clientTracer) {
// Tracer object instance
this.clientTracer = clientTracer;
}
@Override
public Object intercept(Chain chain) throws Exception {
// Record the current system time
long start = System.currentTimeMillis();
String resultCode = SofaTracerConstant.RESULT_SUCCESS;
try {
// Start a span
clientTracer.startTrace(chain.getOriginalSql());
/ / execution
return chain.proceed();
} catch (Exception e) {
resultCode = SofaTracerConstant.RESULT_FAILED;
throw e;
} finally {
System.currenttimemillis () -start
// End a span
clientTracer.endTrace(System.currentTimeMillis() - start, resultCode);
}
}
}
Copy the codeThe general idea is that the Datasource uses a combination of custom proxy classes, intercepts all target object methods, and creates a span for the Datasource before performing specific SQL or join operations. End span after the operation and report.
News buried point
Messaging framework components include many, such as the familiar RocketMQ, Kafka, etc. In addition to dealing with the clients provided by each component, Spring provides many Message component packages, including Spring Cloud Stream, Spring Integration, Spring Message, and so on. SOFATracer provides buried support for common messaging components and Spring Cloud Stream based on the Spring Message standard, as well as a buried implementation based on the RocketMQ client pattern.
Spring Messaging buried point implementation principle
The Spring-Messaging module provides support for integrating messaging APIS and messaging protocols. Here we’ll start with a pipes-and-filters architecture model:
The Spring-Messaging support module provides various MessageChannel implementations and channel interceptor support. Therefore, it is natural for us to use channel Interceptor when embedding Spring-Messaging.
// SOFATracer implements spring-messaging message interceptor
public class SofaTracerChannelInterceptor implements ChannelInterceptor.ExecutorChannelInterceptor {
// todo trace
}
// THIS IS ChannelInterceptor
public interface ChannelInterceptor {
// Before sending
@Nullable
defaultMessage<? > preSend(Message<? > message, MessageChannel channel) {
return message;
}
/ / after sending
default void postSend(Message<? > message, MessageChannel channel,boolean sent) {
}
// After sending
default void afterSendCompletion(Message<? > message, MessageChannel channel,boolean sent, @Nullable Exception ex) {
}
// Before receiving the message
default boolean preReceive(MessageChannel channel) {
return true;
}
/ / after receiving
@Nullable
defaultMessage<? > postReceive(Message<? > message, MessageChannel channel) {
return message;
}
// After receiving the message
default void afterReceiveCompletion(@Nullable Message
message, MessageChannel channel, @Nullable Exception ex) {
}
}
Copy the codeIt can be seen that the ChannelInterceptor realizes the management and control of the whole life cycle of message passing. Through the exposed method, it can easily achieve the expansion of buried points in each stage.
RocketMQ buried point implementation principle
RocketMQ itself provides support for the Opentracing specification, which is not compatible to some extent due to the inconsistency between the supported version and the Opentracing version implemented by SOFATracer. So SOFATracer (OpenTracing 0.22.0) itself provides a plug-in for RocketMQ separately.
The RocketMQ burying point is actually done through two hook interfaces, which are actually not mentioned in the official RocketMQ documentation.
// RocketMQ message consumer hook interface buried implementation class
public class SofaTracerConsumeMessageHook implements ConsumeMessageHook {
}
// RocketMQ message sender hook interface buried point implementation class
public class SofaTracerSendMessageHook implements SendMessageHook {}
Copy the codeThe first is SendMessageHook interface, SendMessageHook interface provides two methods, sendMessageBefore and sendMessageAfter, SOFATracer in the implementation of buried point, SendMessageBefore is used to parse and build the span, and sendMessageAfter is used to get the result and end the span.
Similarly, ConsumeMessageHook also provides two methods (consumeMessageBefore and consumeMessageAfter), SOFATracer can be provided to parse transparent tracer information from messages and then pass the tracer information transparently to downstream links.
Redis buried point principle
Redis embedding in SOFATracer is implemented based on Spring Data Redis, without embedding for a specific Redis client. In addition, the redis buried point refers to the implementation logic in the open source community Opentracing – Spring-Cloud-Redis – Starter.
Redis’ buried point implementation and Datasource’s anchor point implementation are the same basic idea, through a layer of proxies to achieve interception. Sofa-tracer-redis-plugin packages all redIS operations with RedisActionWrapperHelper. SOFATracer provides apis for buried operations before and after executing specific commands. The code is as follows:
public <T> T doInScope(String command, Supplier<T> supplier) {
/ / build span
Span span = buildSpan(command);
return activateAndCloseSpan(span, supplier);
}
// Execute specific commands during the span lifecycle
private <T> T activateAndCloseSpan(Span span, Supplier<T> supplier) {
Throwable candidateThrowable = null;
try {
// Execute the command
return supplier.get();
} catch (Throwable t) {
candidateThrowable = t;
throw t;
} finally {
if(candidateThrowable ! =null) {
// ...
} else {
// ...
}
// End a span with the tracer API
redisSofaTracer.clientReceiveTagFinish((SofaTracerSpan) span, "00");
}
}
Copy the codeIn addition, the buried point of mongodb is also based on Spring Data implementation, and the implementation idea of buried point is basically the same as redis, so it will not be analyzed separately here.
conclusion
This paper briefly introduces the buried point mechanism of Ant Financial distributed link component SOFATracer. The whole idea in terms of buried mechanisms for individual components is to wrap component operations, span and report before and after a request or command is executed. At present, some mainstream link tracing components such as Brave are also based on this idea. The difference lies in that Brave does not code directly based on OpenTracing specification, but encapsulates a whole set of APIS by itself, and then ADAPTS through OpenTracing API. Another very popular Skywalking is based on a Java Agent implementation, which is mechanically different from SOFATracer and Brave.
reference
- SOFATracer
- Spring source analysis of spring- Messaging module details