0 x00 the

SOFA is a finance-level distributed middleware independently researched and developed by Ant Financial, which contains various components required for the construction of finance-level cloud native architecture. SOFATracer is a component used for distributed system call tracking.

I have experience with Zipkin before and want to extend it to Opentracing. Therefore, BASED on learning the source code of SOFATracer’s official blog, I summarize this article and share it with you.

See industry Solutions for Learning Distributed tracing with SOFATracer (1).

0x07 SOFATracer plug-in burying mechanism

Tracing an application is all about client -> Web layer -> RPC service -> DAO back-end storage, cache cache, message queue MQ and other basic components. The SOFATracer plug-in is actually used to bury different components so that application link data can be collected based on these components.

Different components have different application scenarios and extension points, so the implementation of plug-ins should also adapt to local conditions, SOFATracer buried point is generally through the Filter, Interceptor mechanism to achieve.

7.1 Filter or Interceptor of component extensions

SOFATracer has been implemented plug-ins, such as SpringMVC plug-ins are embedded based on Filter, HttpClient, restTemplate, etc., are embedded based on the Interceptor mechanism. When implementing plug-ins, specific embedding methods should be selected according to the characteristics and extension points of different plug-ins. Just as the so-called all roads lead to Rome, no matter how to achieve buried points, are dependent on SOFATracer’s own API extension mechanism to achieve.

All SOFATracer plug-ins need to implement their own Tracer instances, such as SpringMVC’s SpringMvcTracer and HttpClient’s HttpClientTracer.

AbstractTracer is an abstract class used by SOFATracer for plug-in extensions. AbstractTracer can be divided into clientTracer and serverTracer according to different plug-in types. Corresponding to AbstractClientTracer and AbstractServerTracer respectively; AbstractClientTracer and AbstractServerTracer are used to derive the specific component Tracer implementation. Such as the HttpClientTracer, RestTemplateTracer, SpringMvcTracer and other plug-in Tracer implementations mentioned in the figure above.

How do you determine whether a component is client-side or server-side? It depends on whether the current component is the initiator of the request or the receiver of the request. If it is the initiator of the request, it is usually the client side. If it is the receiver of the request, it is the server side. So for RPC, both the initiator and the receiver of the request, so AbstractTracer class is implemented here.

7.2 Summary of basic ideas of Plug-in Extension

For a component, a single processing process typically produces a Span; The life cycle of this Span is from receiving the request to returning the response.

But the question here is how does it relate to upstream and downstream links? In the Opentracing specification, it is possible to extract a SpanContext from Tracer that is passed across processes. Then, the current node is associated to the entire Tracer link through the information carried by the SpanContext. Of course, extract will have corresponding inject.

Link construction generally follows the mode of client——server——client——server. It is clear here that the link will inject capital at the client side and extract at the server side. Over and over and over and over again.

After getting the SpanContext, the current Span can now be associated with the link, and all that remains is to collect some data for the current component; The whole process can be roughly divided into the following stages:

  • Extract the spanContext from the request
  • Build a Span and store the current Span in the current tracer context (sofatracecontext.push (Span)).
  • Set some information to Span
  • Returns a response
  • Span End & report

7.3 Burying Principles of the Standard Servlet specification

SOFATracer supports Web MVC burying points to the standard Servlet specification, including regular Servlets and Spring MVC, The basic principle is based on the extended implementation of the Javax.servlet.filter Filter interface provided by the Servlet specification.

The filter sits between the Client and the Web application and is used to examine and modify the request and response information that flows between the two. The filter intercepts the request before it reaches the Servlet. The filter intercepts the response before it is sent to the client. Multiple filters form a FilterChain. The sequence of different filters in the FilterChain is determined by the sequence of filter mappings in the deployment file web. The filter that picks up the client request first will pick up the Servlet response last.

Web applications usually serve as the receiver of requests, and in SOFATracer applications exist as servers. The event corresponding to the SpanContext resolution is Server Receive (SR).

SOFATracer parses and generates spans in sofA-Tracer-SpringMVC-plugin as follows:

  • Servlet Filter intercepts request requests;
  • Resolve SpanContext from the request;
  • Build the Span of the current MVC with SpanContext;
  • Add a tag and log to the current Span.
  • At the end of the Filter processing, end Span;

7.4 Principles of BURYING HTTP Clients

HTTP client buried points include HttpClient, OkHttp, RestTemplate, etc. Such buried points are generally implemented based on the interceptor mechanism. For example, HttpRequestInterceptor and HttpResponseInterceptor are available from HttpClient. OkHttp uses the okhttp3.Interceptor; RestTemplate using ClientHttpRequestInterceptor.

Taking OkHttp as an example, the implementation principle of HTTP client burying point is briefly analyzed:

@Override
public Response intercept(Chain chain) throws IOException {
    // Get the request
    Request request = chain.request();
    // Parse out the SpanContext and build the Span
    SofaTracerSpan sofaTracerSpan = okHttpTracer.clientSend(request.method());
    // Initiate a specific call
    Response response = chain.proceed(appendOkHttpRequestSpanTags(request, sofaTracerSpan));
    / / the end of the span
    okHttpTracer.clientReceive(String.valueOf(response.code()));
    return response;
}
Copy the code

0x08 Overall Request Process

Requests in SOFATracer are roughly divided into the following processes:

  • The client sends a request clientSend CS
  • The server accepts the request serverReceive SR
  • Server returns result serverSend SS
  • Client receives result clientReceive CR

Regardless of the plug-in, you can find the corresponding processing method from the above stages in the request processing cycle. Therefore, SOFATracer encapsulates these stages of processing.

In SOFA, the four stages actually produce two spans. The first Span starts with CS and ends with CR. The second Span starts with SR and ends with SS.

clientSend // When the client sends the request, which is the CS phase, a Span is generated.
    serverReceive // The server receives the request sr phase, which generates a Span.. serverSend clientReceiveCopy the code

In terms of time series, it is shown in the figure below.

     Client                             Server

+--------------+     Request        +--------------+
| Client Send  | +----------------> |Server Receive|
+------+-------+                    +------+-------+
       |                                   |
       |                                   v
       |                            +------+--------+
       |                            |Server Business|
       |                            +------+--------+
       |                                   |
       |                                   |
       v                                   v
+------+--------+    Response       +------+-------+
|Client Receive | <---------------+ |Server Send   |
+------+--------+                   +------+-------+
       |                                   |
       |                                   |
       v                                   v
Copy the code

8.1 TraceID

The Trace ID is generated when the client sends the request clientSend CS, that is, the ID is typically generated by the first system in the cluster to process the request and passed over the network to the next requested system in a distributed call. It’s the AbstractTracer # clientSend function.

  • Call buildSpan to build a SofaTracerSpan clientSpan, then call the start function to build a Span.

    • If no Parent context exists, a call to createRootSpanContext creates a new root Span context.

      • sofaTracerSpanContext = this.createRootSpanContext();
Copy the code
        • Call the String traceId = TraceIdGenerator. The generate (); To build the trace ID.

    • If there is a Parent context, createChildContext is called to establish the SPAN context.

  • Set various tags to clientSpan.

    • clientSpan.setTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_CLIENT);
Copy the code

  • Set the log for clientSpan.

    • clientSpan.log(LogData.CLIENT_SEND_EVENT_VALUE);
Copy the code

  • Set clientSpan to SpanContext.

    • sofaTraceContext.push(clientSpan);
Copy the code

The code that generates a traceId is in the TraceIdGenerator class. TraceId consists of the IP address, timestamp, increment sequence, and process ID. That is, TraceId is the server IP address + the time when the ID is generated + the increment sequence + the current process ID to ensure global uniqueness. This answers our earlier question: How is traceId generated, and what are the rules?

public class TraceIdGenerator {
    private static String IP_16 = "ffffffff";
    private static AtomicInteger count = new AtomicInteger(1000);

    private static String getTraceId(String ip, long timestamp, int nextId) {
        StringBuilder appender = new StringBuilder(30);
        appender.append(ip).append(timestamp).append(nextId).append(TracerUtils.getPID());
        return appender.toString();
    }

    public static String generate(a) {
        return getTraceId(IP_16, System.currentTimeMillis(), getNextId());
    }

    private static String getIP_16(String ip) {
        String[] ips = ip.split("\ \.");
        StringBuilder sb = new StringBuilder();
        String[] var3 = ips;
        int var4 = ips.length;

        for(int var5 = 0; var5 < var4; ++var5) {
            String column = var3[var5];
            String hex = Integer.toHexString(Integer.parseInt(column));
            if (hex.length() == 1) {
                sb.append('0').append(hex);
            } else{ sb.append(hex); }}return sb.toString();
    }

    private static int getNextId(a) {
        int current;
        int next;
        do {
            current = count.get();
            next = current > 9000 ? 1000 : current + 1;
        } while(! count.compareAndSet(current, next));return next;
    }

    static {
        try {
            String ipAddress = TracerUtils.getInetAddress();
            if(ipAddress ! =null) { IP_16 = getIP_16(ipAddress); }}catch (Throwable var1) {
        }
    }
}
Copy the code

8.2 SpanID

There are two places where SpanId is generated: CS, SR. SOFARPC differs from Dapper in that spanId already contains the context of the call chain, which contains information about the parent spanId. For example, in the process of processing a request, the system successively calls three systems B, C and D, and the SPANIDS of these three calls are 0.1, 0.2 and 0.3 respectively. If system C continues to call two systems E and F, the SPANIDS of these two calls are 0.2.1 and 0.2.2 respectively.

8.2.1 Client Send

Following the previous section, the Span is built at the stage when the client sends the request clientSend CS, resulting in the SpanID.

  • Call buildSpan to build a SofaTracerSpan clientSpan, then call the start function to build a Span.

    • If no Parent context exists, a call to createRootSpanContext creates a new root Span context.

      • sofaTracerSpanContext = this.createRootSpanContext();
Copy the code
        • Call SofaTracerSpanContext to generate a new SpanContext, which generates a new Span ID.

    • If there is a Parent context, then call createChildContext establish span the context, the preferredReference here. GetSpanId () will generate the span ID. Since the Parent Context already exists, the new Span Id is built on top of the Parent Span Id.

      • SofaTracerSpanContext sofaTracerSpanContext = new SofaTracerSpanContext(
    preferredReference.getTraceId(), preferredReference.nextChildContextId(),
    preferredReference.getSpanId(), preferredReference.isSampled());
Copy the code

8.2.2 Server Receive

Using the Server Receive action as an example, you can see the Span build process on the Server side.

  • SpringMvcSofaTracerFilter # doFilter SofaTracerSpanContext extracted from the Header.

    • Use SofaTracer # extract to extract SofaTracerSpanContext, using Spring MVC Cheaders Carrier.
      • Using RegistryExtractorInjector # extract extracted from SpringMvcHeadersCarrier SpanContext.
        • AbstractTextB3Formatter # extract SpanContext from Spring Mvcheaders Driver

  • AbstractTracer # serverReceive follows with SofaTracerSpanContext, SofaTracerSpanContext as follows:

    • sofaTracerSpanContext = {SofaTracerSpanContext@6056} "SofaTracerSpanContext{traceId='c0a80103159927161709310013925', spanId='0', parentId='', isSampled=true, bizBaggage={}, sysBaggage={}, childContextIndex=0}"
 traceId = "c0a80103159927161709310013925"
 spanId = "0"
 parentId = ""
 isSampled = true
 sysBaggage = {ConcurrentHashMap@6060}  size = 0
 bizBaggage = {ConcurrentHashMap@6061}  size = 0
 childContextIndex = {AtomicInteger@6062} "0"
Copy the code

    • The current SpanContext is extracted from the current thread, and then the serverSpan is extracted, which may be null or may have a value.

      • SofaTraceContext sofaTraceContext = SofaTraceContextHolder.getSofaTraceContext();
SofaTracerSpan serverSpan = sofaTraceContext.pop();
Copy the code

      • If serverSpan is null, a new newSpan is generated, then setSpanId is called to set the new SpanId on the SofaTracerSpanContext parameter passed in

        • sofaTracerSpanContext.setSpanId(sofaTracerSpanContext.nextChildContextId()); Now the content of sofaTracerSpanContext has changed, spanId. sofaTracerSpanContext = {SofaTracerSpanContext@6056} 
 traceId = "c0a80103159927161709310013925"
 spanId = "0.1"
 parentId = "".Copy the code

      • If serverSpan is not null, newSpan = serverSpan

    • Set the log

    • Set the Tag

    • Set the newSpan into the local context. sofaTraceContext.push(newSpan);

Note that later in the link, traceId and spanId are stored in the sofaTracerSpanContext of the local thread, not in the Span.

The specific code is as follows:

First of all, SpringMvcSofaTracerFilter # doFilter SofaTracerSpanContext extracted from the Header

public class SpringMvcSofaTracerFilter implements Filter {
    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) {
            // Extract the Context from the header
            SofaTracerSpanContext spanContext = getSpanContextFromRequest(request);
            // srspringMvcSpan = springMvcTracer.serverReceive(spanContext); }}Copy the code

Second, AbstractTracer # serverReceive does the following based on SofaTracerSpanContext

public abstract class AbstractTracer {
    public SofaTracerSpan serverReceive(SofaTracerSpanContext sofaTracerSpanContext) {
        SofaTracerSpan newSpan = null;
        SofaTraceContext sofaTraceContext = SofaTraceContextHolder.getSofaTraceContext();
        SofaTracerSpan serverSpan = sofaTraceContext.pop();
        try {
            if (serverSpan == null) {
                if (sofaTracerSpanContext == null) {
                    sofaTracerSpanContext = SofaTracerSpanContext.rootStart();
                    isCalculateSampled = true;
                } else {                    			
                sofaTracerSpanContext.setSpanId(sofaTracerSpanContext.nextChildContextId());
                }
                newSpan = this.genSeverSpanInstance(System.currentTimeMillis(),
                    StringUtils.EMPTY_STRING, sofaTracerSpanContext, null);
            } else{ newSpan = serverSpan; }}}}Copy the code

As you can see, the build rules for SpanID are relatively simple, which answers the question we asked earlier: how is SpanID generated and what are the rules? And where does ParentSpan come from?

public class SofaTracerSpanContext implements SpanContext {
  private AtomicInteger childContextIndex = new AtomicInteger(0);

  public String nextChildContextId(a) {
    return this.spanId + RPC_ID_SEPARATOR + childContextIndex.incrementAndGet(); }}Copy the code

0 x09 Client sent

In this section, take a look at how the RestTemplate sends requests.

First, print out the Stack as follows, so you can get an idea:

intercept:56, RestTemplateInterceptor (com.sofa.alipay.tracer.plugins.rest.interceptor)
execute:92, InterceptingClientHttpRequest$InterceptingRequestExecution (org.springframework.http.client)
executeInternal:76, InterceptingClientHttpRequest (org.springframework.http.client)
executeInternal:48, AbstractBufferingClientHttpRequest (org.springframework.http.client)
execute:53, AbstractClientHttpRequest (org.springframework.http.client)
doExecute:734, RestTemplate (org.springframework.web.client)
execute:669, RestTemplate (org.springframework.web.client)
getForEntity:337, RestTemplate (org.springframework.web.client)
main:40, RestTemplateDemoApplication (com.alipay.sofa.tracer.examples.rest)
Copy the code

In InterceptingClientHttpRequest # execute code here

class InterceptingClientHttpRequest extends AbstractBufferingClientHttpRequest {
    @Override
		public ClientHttpResponse execute(HttpRequest request, byte[] body) throws IOException {
			if (this.iterator.hasNext()) {
				ClientHttpRequestInterceptor nextInterceptor = this.iterator.next();
				return nextInterceptor.intercept(request, body, this); // There is interception}}}Copy the code

Finally, it comes to the SOFA interceptor, where it does the processing.

9.1 generate Span

The implementation code is in the RestTemplateInterceptor # Intercept function.

As you can see, the RestTemplateInterceptor has a member variable, restTemplateTracer, which is where the processing is implemented. You can see that there are clientSend and clientReceive procedures.

  • Start by generating a Span. SofaTracerSpan sofaTracerSpan = restTemplateTracer.clientSend(request.getMethod().name());

    • Start with serverSpan from SofaTraceContext. If this client is a service midpoint (that is, serverSpan is not empty), then you need to set the parent span for the new span.

    • Call clientSpan = (SofaTracerSpan) this. SofaTracer. BuildSpan (operationName). AsChildOf (serverSpan). The start (); Get its own Client Span. If there is a Server Span, the Client Span is the child of the Sever Span.

      • public Tracer.SpanBuilder asChildOf(Span parentSpan) {
    if (parentSpan == null) {
        return this;
    }
    return addReference(References.CHILD_OF, parentSpan.context());
}
Copy the code

    • Set the father clientSpan. SetParentSofaTracerSpan (serverSpan);

  • Then call appendRestTemplateRequestSpanTags to put Span in the Header of the Request.

    • Add various tags to Span like app, URL, method…
    • Carrier processing, injectCarrier (request, sofaTracerSpan);
      • Call AbstractTextB3Formatter. Inject set traceId spanId, parentId…

  • Send the request.

  • Further processing upon receipt of server return.

    • Get sofaTraceContext from ThreadLocal

    • Get currentSpan from SofaTracerSpan

    • Call appendRestTemplateResponseSpanTags set all kinds of Tag

    • Call restTemplateTracer. ClientReceive (the resultCode); To deal with

      • clientSpan = sofaTraceContext.pop(); Remove the previous SpanCopy the code

      • Call clientReceiveTagFinish, which in turn calls clientspan.Finish ();

        • callSpanTracer.reportSpanRun the Reporter Data report reportSpan or link Span SofaTracerSpan command to call the sampler sample method to check whether the link needs sampling. SamplingStatus whether the sampling identifier isSampled.

      • If you have a parent Span, then you need to push the parent Span into the Context. sofaTraceContext.push(clientSpan.getParentSofaTracerSpan()); For subsequent processing.

The specific code is as follows:

public class RestTemplateInterceptor implements ClientHttpRequestInterceptor {

    protected AbstractTracer restTemplateTracer; // Sofa internal logic implementation

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
                                        ClientHttpRequestExecution execution) throws IOException {
        SofaTracerSpan sofaTracerSpan = restTemplateTracer.clientSend(request.getMethod().name()); / / Span
        appendRestTemplateRequestSpanTags(request, sofaTracerSpan); / / in the Header
        ClientHttpResponse response = null;
        Throwable t = null;
        try {
            return response = execution.execute(request, body); // Send the request
        } catch (IOException e) {
            t = e;
            throw e;
        } finally {
            SofaTraceContext sofaTraceContext = SofaTraceContextHolder.getSofaTraceContext();
            SofaTracerSpan currentSpan = sofaTraceContext.getCurrentSpan();
            String resultCode = SofaTracerConstant.RESULT_CODE_ERROR;
            // is get error
            if(t ! =null) {
                currentSpan.setTag(Tags.ERROR.getKey(), t.getMessage());
                // current thread name
                sofaTracerSpan.setTag(CommonSpanTags.CURRENT_THREAD_NAME, Thread.currentThread()
                    .getName());
            }
            if(response ! =null) {
                //tag append
                appendRestTemplateResponseSpanTags(response, currentSpan);
                //finishresultCode = String.valueOf(response.getStatusCode().value()); } restTemplateTracer.clientReceive(resultCode); }}}Copy the code

9.2 Fomatter

Above mentioned when sending calls AbstractTextB3Formatter. Inject set traceId, spanId, parentId.

Fomatter is the interface responsible for the specific logic of serialization/deserialization context in a specific scenario, such as HttpCarrier usage, which usually has a corresponding HttpFormatter. The injection and extraction of Tracer are delegated to Formatter.

The stack looks like this at execution time:

inject:141, AbstractTextB3Formatter (com.alipay.common.tracer.core.registry)
inject:26, AbstractTextB3Formatter (com.alipay.common.tracer.core.registry)
inject:115, SofaTracer (com.alipay.common.tracer.core)
injectCarrier:146, RestTemplateInterceptor (com.sofa.alipay.tracer.plugins.rest.interceptor)
appendRestTemplateRequestSpanTags:141, RestTemplateInterceptor (com.sofa.alipay.tracer.plugins.rest.interceptor)
intercept:57, RestTemplateInterceptor (com.sofa.alipay.tracer.plugins.rest.interceptor)
execute:92, InterceptingClientHttpRequest$InterceptingRequestExecution (org.springframework.http.client)
executeInternal:76, InterceptingClientHttpRequest (org.springframework.http.client)
executeInternal:48, AbstractBufferingClientHttpRequest (org.springframework.http.client)
execute:53, AbstractClientHttpRequest (org.springframework.http.client)
doExecute:734, RestTemplate (org.springframework.web.client)
execute:669, RestTemplate (org.springframework.web.client)
getForEntity:337, RestTemplate (org.springframework.web.client)
main:40, RestTemplateDemoApplication (com.alipay.sofa.tracer.examples.rest)
Copy the code

OpenTracing provides two functions for handling “trace Context” :

  • “Extract (Format, carrier)” retrieves the trace context from the medium (usually HTTP headers).
  • Inject (SpanContext, Format, Carrier) adds trace context to media to ensure the continuity of trace chain.

Inject and extract correspond to serialization and deserialization respectively.

public abstract class AbstractTextB3Formatter implements RegistryExtractorInjector<TextMap> {
    public static final String TRACE_ID_KEY_HEAD = "X-B3-TraceId";
    public static final String SPAN_ID_KEY_HEAD = "X-B3-SpanId";
    public static final String PARENT_SPAN_ID_KEY_HEAD = "X-B3-ParentSpanId";
    public static final String SAMPLED_KEY_HEAD = "X-B3-Sampled";
    static final String FLAGS_KEY_HEAD = "X-B3-Flags";
    static final String BAGGAGE_KEY_PREFIX = "baggage-";
    static final String BAGGAGE_SYS_KEY_PREFIX = "baggage-sys-";

    public SofaTracerSpanContext extract(TextMap carrier) {
        if (carrier == null) {
            return SofaTracerSpanContext.rootStart();
        } else {
            String traceId = null;
            String spanId = null;
            String parentId = null;
            boolean sampled = false;
            boolean isGetSampled = false;
            Map<String, String> sysBaggage = new ConcurrentHashMap();
            Map<String, String> bizBaggage = new ConcurrentHashMap();
            Iterator var9 = carrier.iterator();

            while(var9.hasNext()) {
                Entry<String, String> entry = (Entry)var9.next();
                String key = (String)entry.getKey();
                if(! StringUtils.isBlank(key)) {if (traceId == null && "X-B3-TraceId".equalsIgnoreCase(key)) {
                        traceId = this.decodedValue((String)entry.getValue());
                    }

                    if (spanId == null && "X-B3-SpanId".equalsIgnoreCase(key)) {
                        spanId = this.decodedValue((String)entry.getValue());
                    }

                    if (parentId == null && "X-B3-ParentSpanId".equalsIgnoreCase(key)) {
                        parentId = this.decodedValue((String)entry.getValue());
                    }

                    String keyTmp;
                    if(! isGetSampled &&"X-B3-Sampled".equalsIgnoreCase(key)) {
                        keyTmp = this.decodedValue((String)entry.getValue());
                        if ("1".equals(keyTmp)) {
                            sampled = true;
                        } else if ("0".equals(keyTmp)) {
                            sampled = false;
                        } else {
                            sampled = Boolean.parseBoolean(keyTmp);
                        }

                        isGetSampled = true;
                    }

                    String valueTmp;
                    if (key.indexOf("baggage-sys-") = =0) {
                        keyTmp = StringUtils.unescapeEqualAndPercent(key).substring("baggage-sys-".length());
                        valueTmp = StringUtils.unescapeEqualAndPercent(this.decodedValue((String)entry.getValue()));
                        sysBaggage.put(keyTmp, valueTmp);
                    }

                    if (key.indexOf("baggage-") = =0) {
                        keyTmp = StringUtils.unescapeEqualAndPercent(key).substring("baggage-".length());
                        valueTmp = StringUtils.unescapeEqualAndPercent(this.decodedValue((String)entry.getValue())); bizBaggage.put(keyTmp, valueTmp); }}}if (traceId == null) {
                return SofaTracerSpanContext.rootStart();
            } else {
                if (spanId == null) {
                    spanId = "0";
                }

                if (parentId == null) {
                    parentId = "";
                }

                SofaTracerSpanContext sofaTracerSpanContext = new SofaTracerSpanContext(traceId, spanId, parentId, sampled);
                if (sysBaggage.size() > 0) {
                    sofaTracerSpanContext.addSysBaggage(sysBaggage);
                }

                if (bizBaggage.size() > 0) {
                    sofaTracerSpanContext.addBizBaggage(bizBaggage);
                }

                returnsofaTracerSpanContext; }}}public void inject(SofaTracerSpanContext spanContext, TextMap carrier) {
        if(carrier ! =null&& spanContext ! =null) {
            carrier.put("X-B3-TraceId".this.encodedValue(spanContext.getTraceId()));
            carrier.put("X-B3-SpanId".this.encodedValue(spanContext.getSpanId()));
            carrier.put("X-B3-ParentSpanId".this.encodedValue(spanContext.getParentId()));
            carrier.put("X-B3-SpanId".this.encodedValue(spanContext.getSpanId()));
            carrier.put("X-B3-Sampled".this.encodedValue(String.valueOf(spanContext.isSampled())));
            Iterator var3 = spanContext.getSysBaggage().entrySet().iterator();

            Entry entry;
            String key;
            String value;
            while(var3.hasNext()) {
                entry = (Entry)var3.next();
                key = "baggage-sys-" + StringUtils.escapePercentEqualAnd((String)entry.getKey());
                value = this.encodedValue(StringUtils.escapePercentEqualAnd((String)entry.getValue()));
                carrier.put(key, value);
            }

            var3 = spanContext.getBizBaggage().entrySet().iterator();

            while(var3.hasNext()) {
                entry = (Entry)var3.next();
                key = "baggage-" + StringUtils.escapePercentEqualAnd((String)entry.getKey());
                value = this.encodedValue(StringUtils.escapePercentEqualAnd((String)entry.getValue())); carrier.put(key, value); }}}}Copy the code

After serialization, the last Header that’s sent is the following, so we need to remember the concept of spanContext.

Context stores information that needs to be crossed, for example:

  • SpanId: INDICATES the ID of the current span
  • TraceId: The traceId to which the span belongs (that is, the unique ID of the call chain).
    • trace_idandspan_idTo distinguish betweenTraceIn theSpan; Any state associated with the OpenTraceing implementation (such as Trace and SPAN ID) needs to be identified by a cross-process parameterSpanContact.
  • Baggage: Other information that spans multiple calling units, that is, key value pairs across processes.Baggage Items 和 Span TagThe structure is the same, the only difference being:Span TagOnly in the currentSpanExists in, not in the wholetraceIn the transmission, whileBaggage ItemsWill be called by the chain.

As you can see, the spanContext has been decomposed and serialized into the Header.

request = {InterceptingClientHttpRequest@5808} 
 requestFactory = {SimpleClientHttpRequestFactory@5922} 
 interceptors = {ArrayList@5923}  size = 1
 method = {HttpMethod@5924} "GET"
 uri = {URI@5925} "http://localhost:8801/rest"
 bufferedOutput = {ByteArrayOutputStream@5926} ""
 headers = {HttpHeaders@5918}  size = 6
  "Accept" -> {LinkedList@5938}  size = 1
  "Content-Length" -> {LinkedList@5940}  size = 1
  "X-B3-TraceId" -> {LinkedList@5942}  size = 1
   key = "X-B3-TraceId"
   value = {LinkedList@5942}  size = 1
    0 = "c0a800031598690915258100115720"
  "X-B3-SpanId" -> {LinkedList@5944}  size = 2
   key = "X-B3-SpanId"
   value = {LinkedList@5944}  size = 2
    0 = "0"
    1 = "0"
  "X-B3-ParentSpanId" -> {LinkedList@5946}  size = 1
  "X-B3-Sampled" -> {LinkedList@5948}  size = 1
 executed = false
body = {byte[0] @5810} 
Copy the code

9.3 the Report

The final step in sending is clientspan.finish ().

As mentioned in the Opentracing specification, the Span# Finish method is the last execution of the span lifecycle, meaning that a span is about to end. When a span is about to end, the current span is at its most complete. So in SOFATracer, the entry point for data reporting is the Span#finish method, which has the following call stack:

doReportStat:43, RestTemplateStatJsonReporter (com.sofa.alipay.tracer.plugins.rest)
reportStat:179, AbstractSofaTracerStatisticReporter (com.alipay.common.tracer.core.reporter.stat)
statisticReport:143, DiskReporterImpl (com.alipay.common.tracer.core.reporter.digest)
doReport:60, AbstractDiskReporter (com.alipay.common.tracer.core.reporter.digest)
report:51, AbstractReporter (com.alipay.common.tracer.core.reporter.facade)
reportSpan:141, SofaTracer (com.alipay.common.tracer.core)
finish:165, SofaTracerSpan (com.alipay.common.tracer.core.span)
finish:158, SofaTracerSpan (com.alipay.common.tracer.core.span)
clientReceiveTagFinish:176, AbstractTracer (com.alipay.common.tracer.core.tracer)
clientReceive:157, AbstractTracer (com.alipay.common.tracer.core.tracer)
intercept:82, RestTemplateInterceptor (com.sofa.alipay.tracer.plugins.rest.interceptor)
execute:92, InterceptingClientHttpRequest$InterceptingRequestExecution (org.springframework.http.client)
executeInternal:76, InterceptingClientHttpRequest (org.springframework.http.client)
executeInternal:48, AbstractBufferingClientHttpRequest (org.springframework.http.client)
execute:53, AbstractClientHttpRequest (org.springframework.http.client)
doExecute:734, RestTemplate (org.springframework.web.client)
execute:669, RestTemplate (org.springframework.web.client)
getForEntity:337, RestTemplate (org.springframework.web.client)
main:40, RestTemplateDemoApplication (com.alipay.sofa.tracer.examples.rest)
Copy the code

SOFATracer itself provides two reporting modes, one to disk and the other to Zipkin. In terms of implementation details, rather than separate the two strategies to provide independent functional support, SOFATracer combines the two reporting methods together and controls whether specific reporting is performed by parameters in the process of executing specific reporting.

There are three reporting points involved in this process, the first is reporting to Zipkin, followed by falling disk; In terms of logging, SOFATracer provides separate logging space for different components. In addition, SOFATracer provides two different logging modes for link data collection: Abstract: Log and statistical log, which provides a powerful data support for the subsequent construction of some control end such as fault fast discovery, service governance and so on.

For example, Zipkin reports:

public class ZipkinSofaTracerSpanRemoteReporter implements SpanReportListener.Flushable.Closeable {
    public void onSpanReport(SofaTracerSpan span) {
        //convert
        Span zipkinSpan = zipkinV2SpanAdapter.convertToZipkinSpan(span);
        this.delegate.report(zipkinSpan); }}Copy the code

It will be called to zipkin2. Reporter. AsyncReporter for detailed report.

9.4 Sampling calculation

Sampling is for the entire link, meaning that from the time RootSpan is created, it determines whether the current link data will be recorded. In the SofaTracer class, Sapmler instances exist as member variables and are set to final, meaning that once a SofaTracer instance is built, the sampling strategy is not changed. When the Sampler Sampler is bound to SofaTracer instances, SofaTracer will rely on the Sampler’s calculation of the drop behavior of the Span data generated (for a particular link).

0x10 Server Receives Packets

Class SpringMvcSofaTracerFilter completed the server receives the relevant work. Basically, you set SpanContext and Span.

public class SpringMvcSofaTracerFilter implements Filter {
    private SpringMvcTracer springMvcTracer;
   
    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) {... }}Copy the code

Recall: on the client side

  • Inject traceId information generated by the current request thread into SpanContext.
  • Then Fomatter serializes the SpanContext into the Header.

The server extracts the spanContext from the Header of the request to restore the context of the request thread. Because the context is specific to the thread being processed, it is placed in a ThreadLocal.

The general process can be roughly demonstrated as follows:

Client Span Server Span ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ │ │ │ │ TraceContext │ Http Request Headers │ TraceContext │ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ │ │ TraceId │ │ │ X - B3 - TraceId │ │ │ TraceId │ │ │ │ │ │ │ ParentSpanId │ Inject │ X-B3-PARENtSPANID │Extract │ ParentSpanId │ │ ├ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ > │ ├ ─ ─ ─ ─ ─ ─ ─ ─ ┼ > │ │ │ │ │ SpanId │ │ │ X - B3 - SpanId │ │ │ SpanId │ │ │ │ │ │ │ │ │ │ │ │ │ │ the javax.media.sound.sampled │ │ │ X - B3 - the javax.media.sound.sampled │ │ │ the javax.media.sound.sampled │ │ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ │ │ │ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘Copy the code

This answers the previous question: What does the server do once it receives the request? How is SpanContext handled on the server side?

SpringMvcSofaTracerFilter here has a member variable SpringMvcTracer, it is the Server Tracer, here is the logic.

public class SpringMvcTracer extends AbstractServerTracer {
    private static volatile SpringMvcTracer springMvcTracer = null;
}
Copy the code

Specific SpringMvcSofaTracerFilter doFilter roughly the logic is as follows:

  • Call getSpanContextFromRequest from acquiring SpanContext request, which USES the tracer. The extract function.

    • SofaTracerSpanContext spanContext = (SofaTracerSpanContext)tracer.extract(Builtin.B3_HTTP_HEADERS, new SpringMvcHeadersCarrier(headers));
Copy the code

  • Call serverReceive to get the Span

    • springMvcSpan = this.springMvcTracer.serverReceive(spanContext);
Copy the code
      • SofaTracerSpan serverSpan = sofaTraceContext.pop(); // Retrieve the father Span, if not present, then
sofaTracerSpanContext.setSpanId(sofaTracerSpanContext.nextChildContextId()); // Set it to the next child ID
Copy the code
      • sofaTraceContext.push(newSpan); // Put the Span into the SpanContext
Copy the code

  • Span Sets various settags

  • Call this. SpringMvcTracer. ServerSend (String. The valueOf (httpStatus)); To end Span.

    • End & report

      • this.clientReceiveTagFinish(clientSpan, resultCode);
Copy the code
        • Set log, resultCode, end Client Span: clientSpan. Finish ();
          • Call SofaTracer # reportSpan to report. This part is similar to the Client code function.

    • Restore restore restore parent SPAN

      • sofaTraceContext.push(clientSpan.getParentSofaTracerSpan());
Copy the code

The function code is as follows

public class SpringMvcSofaTracerFilter implements Filter {
    private SpringMvcTracer springMvcTracer;

    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) {
        if (this.springMvcTracer == null) {
            this.springMvcTracer = SpringMvcTracer.getSpringMvcTracerSingleton();
        }

        SofaTracerSpan springMvcSpan = null;
        long responseSize = -1L;
        int httpStatus = -1;

        try {
            HttpServletRequest request = (HttpServletRequest)servletRequest;
            HttpServletResponse response = (HttpServletResponse)servletResponse;
            SofaTracerSpanContext spanContext = this.getSpanContextFromRequest(request);
            springMvcSpan = this.springMvcTracer.serverReceive(spanContext);
            if (StringUtils.isBlank(this.appName)) {
                this.appName = SofaTracerConfiguration.getProperty("spring.application.name");
            }

            springMvcSpan.setOperationName(request.getRequestURL().toString());
            springMvcSpan.setTag("local.app".this.appName);
            springMvcSpan.setTag("request.url", request.getRequestURL().toString());
            springMvcSpan.setTag("method", request.getMethod());
            springMvcSpan.setTag("req.size.bytes", request.getContentLength());
            SpringMvcSofaTracerFilter.ResponseWrapper responseWrapper = new SpringMvcSofaTracerFilter.ResponseWrapper(response);
            filterChain.doFilter(servletRequest, responseWrapper);
            httpStatus = responseWrapper.getStatus();
            responseSize = (long)responseWrapper.getContentLength();
        } catch (Throwable var15) {
            httpStatus = 500;
            throw new RuntimeException(var15);
        } finally {
            if(springMvcSpan ! =null) {
                springMvcSpan.setTag("resp.size.bytes", responseSize);
                this.springMvcTracer.serverSend(String.valueOf(httpStatus)); }}}}Copy the code

0x11 Problem Solved

The questions we asked in the beginning are now answered.

  • How is traceId generated and what are the rules? Here are the answers:
    • In clientSend CS, when a Span is created, if no Parent context exists, createRootSpanContext is called to create a new root Span context. A traceId is generated
    • TraceId consists of IP addresses, time stamps, increasing sequences, and process ids. For details, see TraceIdGenerator.
  • How is spanId generated and what are the rules? Here are the answers:
    • During the Server Receive phase, if the current thread SpanContext does not have a Span, a new newSpan is generated, Then call setSpanId to set the new SpanId for the SofaTracerSpanContext parameter passed in.
    • The simple rule is to monotonically increase the Span ID from the previous one, see SofaTracerSpanContext #nextChildContextId.
  • Where does the client generate the Span? Here are the answers:
    • The AbstractTracer # clientSend function calls buildSpan to build a SofaTracerSpan clientSpan, The start function is then called to establish a Span.
  • Where does ParentSpan come from? Here are the answers:
    • In the clientSend phase, first fetch serverSpan from SofaTraceContext. If this client is a service midpoint (serverSpan is not empty) and serverSpan is parentSpan, set the parentSpan for the new span.
  • ChildSpan was created by ParentSpan, so when was it created? Here are the answers:
    • Call ParentSpan if ParentSpan existsclientSpan = (SofaTracerSpan)this.sofaTracer.buildSpan(operationName).asChildOf(serverSpan).start();Get its own Client Span.
    • That is, if active span exists, generate a context for the CHILD_OF relationship if it exists, and createNewContext if it does not;
  • How do Trace messages pass? Here are the answers:
    • In OpenTracing, Trace information is passed through the SpanContext.
    • SpanContext stores information that needs to cross boundaries, such as trace Id, SPAN Id, Baggage. This information is passed by different components according to their own serialization, such as the serialization into HTTP headers.
    • The current node is then associated to the entire Tracer link using the information carried by the SpanContext
  • What does the server do after receiving the request? Here are the answers:
    • The server extracts the spanContext from the Header of the request to restore the context of the request thread. Because the context is specific to the thread being processed, it is placed in a ThreadLocal.
  • How is SpanContext handled on the server side? See answers above.
  • How to collect link information? Here are the answers:
    • Sampling is for the entire link, meaning that from the time RootSpan is created, it determines whether the current link data will be recorded.
    • In the SofaTracer class, Sapmler instances exist as member variables and are set to final, meaning that once a SofaTracer instance is built, the sampling strategy is not changed. When the Sampler Sampler is bound to SofaTracer instances, SofaTracer will rely on the Sampler’s calculation of the drop behavior of the Span data generated (for a particular link).

0xEE Personal information

★★★★ Thoughts on life and technology ★★★★★

Wechat official account: Rosie’s Thoughts

If you want to get a timely news feed of personal articles, or want to see the technical information of personal recommendations, please pay attention.

0 XFF reference

Distributed tracing system — Opentracing

Introduction to Open Distributed Tracing with Jaeger implementation

Description of OpenTracing semantics

Overview of distributed tracking systems and comparison of mainstream open source systems

Skywalking Distributed Tracking and Monitoring: The beginning

Distributed Full link Monitoring – OpenTracing Demo

Opentracing of actual combat

Go microservices full link tracing details

OpenTracing Java Library Tutorial (3) — Passing SpanContext across services

OpenTracing Java Library Tutorial (1) introduction to Trace and Span

Ant gold uniform distributed link tracking component SOFATracer overview | anatomy

Analysis on the principle of transparent link transmission of Ant Financial open source distributed link tracking component SOFATracer and the extended capability of SLF4J MDC

Ant Financial open source distributed link tracking component SOFATracer sampling strategy and source code analysis

Github.com/sofastack-g…

The OpenTracing Semantic Specification

Ant Financial distributed link tracking component SOFATracer data reporting mechanism and source code analysis

Ant Financial open source distributed link tracking component SOFATracer buried point mechanism analysis

Analysis of distributed link component SOFATracer buried point mechanism

【 analyze | SOFARPC framework 】 SOFARPC link of tracking