Learn the distributed tracing system Opentracing with Jaeger

0 x00 the

I have experience with Zipkin before and want to extend it to Opentracing, so I summarize this article based on learning from Jaeger and share it with you.

0x01 Cause & Problem

1.1 choose Jaeger

Jaeger is a call chain server product developed by Uber. The development language is golang, which is compatible with receiving data in OpenTracing format. Based on its history, it’s an improved Version of Zipkin. In addition, its udp (or HTTP) based transport protocol, more positioning its efficient, fast characteristics.

In the previous article, We used SOFATracer to learn distributed tracing system Opentracing (1) and SOFATracer to learn distributed tracing system Opentracing (2). Why did we choose Jaeger? Details are as follows:

Jaeger is officially featured by Opentracing.
Jaeger supports higher versions of Opentracing.

And we can exactly compare it with SOFATracer.

1.2 the problem

Let’s use questions to guide our reading.

How does Jaeger compare with SOFATracer?
How is spanId generated and what are the rules?
How is traceId generated and what are the rules?
Where does the client generate the Span?
Where does ParentSpan come from?
ChildSpan was created by ParentSpan, so when was it created?
How do Trace messages pass?
What does the server do after receiving the request?
How is SpanContext handled on the server side?
How to collect link information?

1.3 Scope of this article

1.3.1 Jaeger constitute

Jaeger is mainly composed of the following parts:

Jaeger Client: implements the OpenTracing standard SDK for different languages. The application program writes data through the API, and the Client Library passes the trace information to the Jaeger-Agent according to the sampling strategy formulated by the application program.
Agent: This is a network daemon that listens for span data on UDP ports and sends it to the collector in batches. It is designed as a base component that is deployed to all hosts. The Agent decouples the Client library from the Collector, shielding the client Library from the details of routing and discovering the Collector.
Collector: Receives the data sent by the Jaeger-agent and writes the data to the back-end storage. The Collector is designed to be a stateless component, so users can run any number of collectors.
Data Store: Back-end storage is designed as a pluggable component that allows Data to be written to Cassandra, Elastic Search.
Query: Receives a Query request, retrieves the TARCE from the back-end storage system and presents it through the UI. Query is stateless and can start multiple instances. Deploy them behind a load balancer like Nginx.

This article only discusses the Jaeger Client functionality.

1.3.2 Full link Tracing

Full-link tracing is divided into three tracing levels:

Cross-process tracing (calling another microservice)
Database tracing
In-process tracing (tracing within a function)

This article only discusses cross-process, because cross-process tracing is the simplest. For cross-process tracing, you can write interceptors or filters to track each request, and it requires very little code.

0x02 Background

Since the background has been covered in some detail, this article will only mention a few essential concepts.

Distributed tracking system develops rapidly and has many kinds, but the core steps are generally three: code burial site, data storage, query and display

In the process of data acquisition, it is necessary to invade user code as a burying point, and the API incompatibility of different systems will lead to the change of switching tracking system. To address this issue, the OpenTracing specification was created.

   +-------------+  +---------+  +----------+  +------------+
   | Application |  | Library |  | OSS |  | RPC/IPC |
   | Code |  | Code |  | Services |  | Frameworks |
   +-------------+  +---------+  +----------+  +------------+
          | |             | |
          | |             | |
          v              v             v             v
     +-----------------------------------------------------+
     | · · · · · · · · · · OpenTracing · · · · · · · · · · |
     +-----------------------------------------------------+
       | |                | |
       | |                | |
       v               v                v               v
 +-----------+  +-------------+  +-------------+  +-----------+
 | Tracing |  | Logging |  | Metrics |  | Tracing |
 | System A |  | Framework B |  | Framework C |  | System D |
 +-----------+  +-------------+  +-------------+  +-----------+
Copy the code

Most of the thought models for distributed tracing systems come from Google’s Dapper paper, and OpenTracing uses similar terminology. There are a few basic concepts that need to be understood in advance:

Trace: Dapper builds a call process into a call tree (called Tracer), where each node represents a module or system in a link call. A request invocation chain is identified by a globally unique traceId. In a broad sense, a trace represents the execution of a transaction or process in a (distributed) system. In the OpenTracing standard, a trace is a directed acyclic graph (DAG) composed of multiple spans, each of which represents a continuous execution segment named and timed in the trace.
Span: A Span represents a logical operation unit in the system with a start time and execution time. It is a logical operation in the application. A logical causal relationship is established between spans by nesting or ordering them. A SPAN can be understood as a method call, a block call, or an RPC/ database access, as long as a program access has a full time cycle. In Dapper, a SPAN contains the following phases (different software may have different implementations, for example, some may be subdivided into Client span and Server span) :
- Start: initiates a call
- Cleint Send (CS): the client sends a request
- Server Recv(SR) : The Server receives a request
- Server Send(SS): The Server sends a response
- Client Recv(CR) : Indicates that the Client receives a response from the server
- End: The link is complete.

     Client                             Server

+--------------+     Request        +--------------+
| Client Send  | +----------------> |Server Receive|
+------+-------+                    +------+-------+
       |                                   |
       |                                   v
       |                            +------+--------+
       |                            |Server Business|
       |                            +------+--------+
       |                                   |
       |                                   |
       v                                   v
+------+--------+    Response       +------+-------+
|Client Receive | <---------------+ |Server Send   |
+------+--------+                   +------+-------+
       |                                   |
       |                                   |
       v                                   v
Copy the code

Logs: Each span can perform multiple Logs operations. For each Logs operation, a timestamp time name is required, and optionally a storage structure of any size. It is suitable for logging, exception stack and other time-related information.
Tags: Each span can have multiple Tags in the form of key: value pairs. Tags are time-stamped and can be easily annotated and supplemented by a span. The recorded information is applicable at any point in the span from creation to completion. Again straightforward is to record and time – independent information, this is mainly distinguished from the following Logs.
Baggage Items: This is primarily used to transfer data globally across processes
SpanContext:SpanContextMore of a “concept” than a useful feature of the general OpenTracing layer. When creating aSpan, to transfer protocolInject(injected) and from the transport protocolExtractCall chain information,SpanContextPlay an important role.

0x03 Sample code

3.1 code

All from github.com/yurishkuro/… , you can go to download.

The tracer here uses the JaegerTracer.

public class Hello {

    private final Tracer tracer;
    private final OkHttpClient client;

    private Hello(Tracer tracer) {
        this.tracer = tracer;
        this.client = new OkHttpClient();
    }

    private String getHttp(int port, String path, String param, String value) {
        try {
            HttpUrl url = new HttpUrl.Builder().scheme("http").host("localhost").port(port).addPathSegment(path)
                    .addQueryParameter(param, value).build();
            Request.Builder requestBuilder = new Request.Builder().url(url);
            
            Span activeSpan = tracer.activeSpan();
            Tags.SPAN_KIND.set(activeSpan, Tags.SPAN_KIND_CLIENT);
            Tags.HTTP_METHOD.set(activeSpan, "GET");
            Tags.HTTP_URL.set(activeSpan, url.toString());
            tracer.inject(activeSpan.context(), Format.Builtin.HTTP_HEADERS, Tracing.requestBuilderCarrier(requestBuilder));

            Request request = requestBuilder.build();
            Response response = client.newCall(request).execute();

            Tags.HTTP_STATUS.set(activeSpan, response.code());
            if(response.code() ! =200) {
                throw new RuntimeException("Bad HTTP result: " + response);
            }
            return response.body().string();
        } catch (Exception e) {
            Tags.ERROR.set(tracer.activeSpan(), true);
            tracer.activeSpan().log(ImmutableMap.of(Fields.EVENT, "error", Fields.ERROR_OBJECT, e));
            throw newRuntimeException(e); }}private void sayHello(String helloTo, String greeting) {
        Span span = tracer.buildSpan("say-hello").start();
        try (Scope scope = tracer.scopeManager().activate(span)) {
            span.setTag("hello-to", helloTo);
            span.setBaggageItem("greeting", greeting);

            String helloStr = formatString(helloTo);
            printHello(helloStr);
        } finally{ span.finish(); }}private String formatString(String helloTo) {
        Span span = tracer.buildSpan("formatString").start();
        try (Scope scope = tracer.scopeManager().activate(span)) {
            String helloStr = getHttp(8081."format"."helloTo", helloTo);
            span.log(ImmutableMap.of("event"."string-format"."value", helloStr));
            return helloStr;
        } finally{ span.finish(); }}private void printHello(String helloStr) {
        Span span = tracer.buildSpan("printHello").start();
        try (Scope scope = tracer.scopeManager().activate(span)) {
            getHttp(8082."publish"."helloStr", helloStr);
            span.log(ImmutableMap.of("event"."println"));
        } finally{ span.finish(); }}public static void main(String[] args) {
        try (JaegerTracer tracer = Tracing.init("hello-world")) {
            new Hello(tracer).sayHello("helloTo"."greeting"); }}}Copy the code

3.2 dropwizard

This is not an essential difference between SOFATracer and Jaeger, but it is interesting that SOFATracer uses SprintBoot for the example code, whereas DropWizard is used for the example.

For those of you who are not familiar with DropWizard, here’s what it looks like:

Dropwizard isCoda HaleinYammerThe company was founded to improve the company’s distributed system architecture (now called microservices). Although it was first used to build REST Web services and now has more and more capabilities, its goal is always to be a Web framework that is lightweight, ready for production, and easy to use.
Dropwizard is similar to Spring Boot in that it is an optional tool for building microservices, but it is a little more formal than Spring Boot. The components it uses are generally not optional, and the advantage is that you don’t need as many refinements, such as writing REST-based Web services.
Dropwizard also doesn’t have a dependency injection container (like Spring or CDI) by default, so you can add your own, but Dropwizard recommends that you keep microservices simple and don’t need these extra components.
Just like Spring Boot, Dropwizard recommends packaging the entire project into an executable JAR, so developers don’t have to worry about what application server the application is running on, what extra configuration is required, and applications don’t need to be built into war packages anymore. And there won’t be as many complex hierarchies of classloaders.

Dropwizard provides a nice abstraction layer, aided by an excellent three-way library, making it more efficient and easier to write microservices for production purposes.

Servlet container usageJetty
REST/JAX-RS implementation is usedJersey
JSON serialization usingJackson
integrationHibernate Validator
Guava
Metrics
SLF4J + Logback
Used at the data access layerJDBI

Dropwizard is paranoid that frameworks are just for writing code, so Dropwizard refuses to tweak the underlying technology stack of frameworks in principle. Because it does so, Dropwizard is faster to develop code and easier to configure.

For our sample code, use Dropwizard as an example to set up two services and a test client.

io.dropwizard.Application

public class Formatter extends Application<Configuration> {

    private final Tracer tracer;

    private Formatter(Tracer tracer) {
        this.tracer = tracer;
    }

    @Path("/format")
    @Produces(MediaType.TEXT_PLAIN)
    public class FormatterResource {

        @GET
        public String format(@QueryParam("helloTo") String helloTo, @Context HttpHeaders httpHeaders) {
            Span span = Tracing.startServerSpan(tracer, httpHeaders, "format");
            try (Scope scope = tracer.scopeManager().activate(span)) {
                String greeting = span.getBaggageItem("greeting");
                if (greeting == null) {
                    greeting = "Hello";
                }
                String helloStr = String.format("%s, %s!", greeting, helloTo);
                span.log(ImmutableMap.of("event"."string-format"."value", helloStr));
                return helloStr;
            } finally{ span.finish(); }}}@Override
    public void run(Configuration configuration, Environment environment) throws Exception {
        environment.jersey().register(new FormatterResource());
    }

    public static void main(String[] args) throws Exception {
        System.setProperty("dw.server.applicationConnectors[0].port"."8081");
        System.setProperty("dw.server.adminConnectors[0].port"."9081");
        try (JaegerTracer tracer = Tracing.init("formatter")) {
            new Formatter(tracer).run("server"); }}}Copy the code

0x04 Link Logic

For a component, a single processing process typically produces a Span; The life cycle of this Span is from receiving the request to returning the response.

So the question here is how does it relate to upstream and downstream links? In the Opentracing specification, it is possible to extract a SpanContext from Tracer that is passed across processes. Then, the current node is associated to the entire Tracer link through the information carried by the SpanContext. Of course, extract will have corresponding inject.

Link construction is generally in client-server-client-server mode. It is clear here that the client side will inject, and then extract on the server side, repeatedly, and then continuously pass the link.

After getting the SpanContext, the current Span can now be associated with the link, and all that remains is to collect some data for the current component; The whole process can be roughly divided into the following stages:

Extract the spanContext from the request
Build a Span and store the current Span in the current tracer context (sofatracecontext.push (Span)).
Set some information to Span
Returns a response
Span End & report

0x05 Data model

5.1 Tracer & JaegerTracer

Tracer in Jaeger controls the tracing of a complete service, including registration serviceName (serviceName), sending span (reporter), sampling (sampler), Serialization and deserialization of spans and transcribing (Registry injector, Extractor), statistics tracking system information (metrics, number of success sending spans, etc.).

Therefore, OpenTracing recommends that each service use a Tracer, which is also responsible for constructing a span, retrieving the current SPAN, and retrieving scopeManager.

As can be seen from the specification of OpenTracing, Functions of Tracer are described as follows: Tracer is a simple, thin interface for Span creation and propagation across arbitrary transports. Jaeger just adds other features to it.

Tracer is an interface presented by OpenTracing.

package io.opentracing;
public interface Tracer extends Closeable {
    ScopeManager scopeManager(a);
    Span activeSpan(a);
    Scope activateSpan(Span var1);
    Tracer.SpanBuilder buildSpan(String var1);
    <C> void inject(SpanContext var1, Format<C> var2, C var3);
    <C> SpanContext extract(Format<C> var1, C var2);
    void close(a);
}
Copy the code

JaegerTracer implements io.openTracing.Tracer.

public class JaegerTracer implements Tracer.Closeable {
    private final String version;
    private final String serviceName;
    private final Reporter reporter;
    private final Sampler sampler;
    private finalMap<String, ? > tags;private final boolean zipkinSharedRpcSpan;
    private final boolean expandExceptionLogs;
    private final boolean useTraceId128Bit;
    private final PropagationRegistry registry;
    private final Clock clock;
    private final Metrics metrics;
    private final ScopeManager scopeManager;
    private final BaggageSetter baggageSetter;
    private final JaegerObjectFactory objectFactory;
    private final int ipv4;
}
Copy the code

5.2 Span & JaegerSpan

IO. Opentracing.Span is a concept given by OpenTracing.

public interface Span {
    SpanContext context(a);
    Span setTag(String var1, String var2);
    Span setTag(String var1, boolean var2);
    Span setTag(String var1, Number var2);
    <T> Span setTag(Tag<T> var1, T var2);
    Span setBaggageItem(String var1, String var2);
    String getBaggageItem(String var1);
    Span setOperationName(String var1);
    void finish(a);
    void finish(long var1);
}
Copy the code

JaegerSpan implements io.openTracing.SPan.

public class JaegerSpan implements Span {
  private final JaegerTracer tracer;
  private final long startTimeMicroseconds;
  private final long startTimeNanoTicks;
  private final boolean computeDurationViaNanoTicks;
  private final Map<String, Object> tags;
  private long durationMicroseconds; // span durationMicroseconds
  private String operationName;
  private final List<Reference> references;
  private JaegerSpanContext context;
  private List<LogData> logs;
  private boolean finished = false; // to prevent the same span from getting reported multiple times
}
Copy the code

In Jaeger’s implementation, the Span information is divided into the following aspects:

Span Core information, such as traceId, spanId, parentId, and Baggage
Log information differs from a tag in that it is time-stamped
The tag information
Span other information, such as startTime,duration

The core information of the SPAN is stored in the SpanContext.

5.3 SpanContext & JaegerSpanContext

JaegerSpanContext implements the IO. Opentracing. SpanContext

public interface SpanContext {
    String toTraceId(a);
    String toSpanId(a);
    Iterable<Entry<String, String>> baggageItems();
}
Copy the code

public class JaegerSpanContext implements SpanContext {
  protected static final byte flagSampled = 1;
  protected static final byte flagDebug = 2;
  private final long traceIdLow;
  private final long traceIdHigh;
  private final long spanId;
  private final long parentId;
  private final byte flags;
  private final Map<String, String> baggage;
  private final String debugId;
  private final JaegerObjectFactory objectFactory;
  private final String traceIdAsString;
  private final String spanIdAsString;
}
Copy the code

The core information of the SPAN is stored in the SpanContext, which is created when the SPAN is built. To prevent users from tampering with the core information, all members of the SpanContext are final.

According to the specification of OpenTracing, SpanContext represents Span state that must propagate to descendant Spans and across process boundaries. SpanContext is logically divided into two pieces: (1) the user-level “Baggage” that propagates across Span boundaries and (2) any Tracer-implementation-specific fields that are needed to identify or otherwise contextualize the associated Span instance (e.g., a tuple).

SpanContext refers to the information that must be passed in a SPAN, which is logically divided into two parts. One part is common traceId, spanId and other information, and the other part is user-defined information that needs to be passed, such as Baggage.

JaegerSpanContext just holds the information that the context should have, unlike SofaTraceContext, which also holds the Span, but in Jaeger this is done in ScopeManager.

5.4 Reporter

The default RemoteReporter implements Reporter, which is what we said in the previous article about sending reports.

public class RemoteReporter implements Reporter {
  private static final int DEFAULT_CLOSE_ENQUEUE_TIMEOUT_MILLIS = 1000;
  public static final int DEFAULT_FLUSH_INTERVAL_MS = 1000;
  public static final int DEFAULT_MAX_QUEUE_SIZE = 100;

  private final Sender sender;
  private final int closeEnqueueTimeout;

  @ToString.Exclude private final BlockingQueue<Command> commandQueue;
  @ToString.Exclude private final Timer flushTimer;
  @ToString.Exclude private final Thread queueProcessorThread;
  @ToString.Exclude private final QueueProcessor queueProcessor;
  @ToString.Exclude private final Metrics metrics;
}
Copy the code

5.5 the Scope

OpenTracing abstracts the concepts of Scope(Active Span) and ScopeManager(Set Scope and get current Scope). In a nutshell, the OpenTracing implementation uses Scope and ScopeManager to handle the context in OpenTracing (that is, the get_current_SPAN procedure);

Why abstract the concept of Scope? How about using ThreadLocal to store spans?

A: First understand what Scope is? Scope is a container of Active spans, and Scope represents the currently Active Span; Is an abstraction of the currently active Span, representing a process in the current context;

In addition, ThreadLocalScope records the toRestore Span so that when it finishes, it can revert to the state of the previous Span;

I understand that if only the get_current_SPAN () logic is used, the span can be passed directly into a ThreadLocal; But ScopeManager looks at the code like this. ScopeManager contains a Scope, which contains the current Span, recover Scope; The benefits I understand are: This ensures that if you start a child Span, then when the child Span ends, You can also go back to the parent span (thus continuing to produce sibling spans based on the parent span), but simply stuffing a current span into ThreadLocal will not solve this situation.

Or,

In a multithreaded environment ScopeManager manages the Scope of each thread, and the Scope in each thread manages the Span in that thread. In this way, when a thread needs to retrieve the span currently active in its thread, the ScopeManager can find the Scope of the appropriate thread and retrieve the span of the thread’s activity from the Scope.

The Scope object is a container for Active Span; Use Scope to retrieve the Active Span in the current context.

IO. Opentracing. Util. ThreadLocalScope is an implementation of the Scope, through the ThreadLocal to store;

The constructor simply holds the current Span and sets the passed Span to the current Span. That is, the previously active scope is used as an attribute of the current scopetoRestoreTo store, and sets the current scope to the scopeManager as the current thread’s latest scope.
When the span operation completes (sp.finish), the scope.close method needs to be called to restore, triggering the association of a new activation span, otherwise the call chain will fail.

Specific definitions are as follows:

public class ThreadLocalScope implements Scope {
    private final ThreadLocalScopeManager scopeManager;
    private final Span wrapped; // Current Active Span
    private final ThreadLocalScope toRestore; // Wrap an Active Span, and when wrapped ends, the Span is restored

    ThreadLocalScope(ThreadLocalScopeManager scopeManager, Span wrapped) {
        this.scopeManager = scopeManager;
        this.wrapped = wrapped;
        // Set the current active Scope
        this.toRestore = scopeManager.tlsScope.get();
        scopeManager.tlsScope.set(this);
    }

    @Override
    public void close(a) {
        if(scopeManager.tlsScope.get() ! =this) {
            // This shouldn't happen if users call methods in the expected order. Bail out.
            return;
        }
        scopeManager.tlsScope.set(toRestore);
    }

    Span span(a) {
        returnwrapped; }}Copy the code

5.6 ScopeManager

Scope is enabled or disabled from the PERSPECTIVE of the CPU. ScopeManager manages scopes. A Scope can have multiple spans, but only one active span.

In a multithreaded environment ScopeManager manages the Scope of each thread, and the Scope in each thread manages the Span in that thread. In this way, when a thread needs to retrieve the span currently active in its thread, the ScopeManager can find the Scope of the appropriate thread and retrieve the span of the thread’s activity from the Scope.

With ScopeManager, we can get the current Span with the scopemanager.Activespan () method and set the current context activeSpan with the ScopeManager ().activate(Span) method.

IO. Opentracing. Util. ThreadLocalScopeManager is opentracing ScopeManager implementation, Jaeger and does not have to rewrite a new class, Instead, use threadLocalScope emanager directly.

The activate function activates the Span. Returns Scope (interpreted as representing a phase of the current active Span). That is, the constructor of ThreadLocalScope is called to activate the passed span as the currently active span. If we look at the ThreadLocalScope constructor, instead of activating the span passed in, we activate the wrapped span whose scope is the currently active scope.

When Span activity ends, you need to close the Scope. Try-with-resources is recommended.
The activeSpan function returns Span for the current active state, or null if not.

public class ThreadLocalScopeManager implements ScopeManager {
    // Use the original ThreadLocal to store Active spans; ScopeManager contains only one Scope(Active Span), which is the Active Span in the current context
    final ThreadLocal<ThreadLocalScope> tlsScope = new ThreadLocal<ThreadLocalScope>();

    // As you can see, the Activate function puts span into a newly generated ThreadLocalScope member variable.
    @Override
    public Scope activate(Span span) {
        return new ThreadLocalScope(this, span);
    }

    @Override
    public Span activeSpan(a) { 
        ThreadLocalScope scope = tlsScope.get();
        return scope == null ? null: scope.span(); }}Copy the code

Jaeger uses scopeManager to manage the context. You can get the current context Span from scopeManager. So where exactly is the father-son relationship set up?

In the OpenTracing-Java implementation, this is handled in the tracer.start() method; The start() method determines by scopeManager that an active span exists. If it exists, a CHILD_OF context is generated, and if not, a createNewContext is generated.

This is different from SOFATtacer, which puts this context management functionality in SofaTraceContext and does feel a bit confused when parsing code.

5.7 SpanID & TraceID

Both SpanId and TraceID are generated when the SpanContext is built.

private JaegerSpanContext createNewContext(a) {
  String debugId = getDebugId();
  long spanId = Utils.uniqueId();  // span
  long traceIdLow = spanId;  // trace
  long traceIdHigh = isUseTraceId128Bit() ? Utils.uniqueId() : 0; . }Copy the code

The specific rules are as follows:

public static long uniqueId(a) {
  long val = 0;
  while (val == 0) {
    val = Java6CompatibleThreadLocalRandom.current().nextLong();
  }
  return val;
}
Copy the code

ThreadLocalRandom # Current is then called.

public static Random current(a) {
  if (threadLocalRandomPresent) {
    return ThreadLocalRandomAccessor.getCurrentThreadLocalRandom();
  } else {
    returnthreadLocal.get(); }}static class ThreadLocalRandomAccessor {
    @IgnoreJRERequirement
    private static Random getCurrentThreadLocalRandom(a) {
      returnThreadLocalRandom.current(); }}Copy the code

The final format is as follows:

context = {JaegerSpanContext@1701} "c29c9e0f4a0a681c:36217443515fc248:c29c9e0f4a0a681c:1"
 traceIdLow = -4423486945480775652
 traceIdHigh = 0
 spanId = 3900526584756421192
 parentId = -4423486945480775652
 flags = 1
 baggage = {HashMap@1693}  size = 1
 debugId = null
 objectFactory = {JaegerObjectFactory@1673} 
 traceIdAsString = "c29c9e0f4a0a681c"
 spanIdAsString = "36217443515fc248"
Copy the code

0 x06 start

6.1 Manual Burying Point

To report Java application data to the link tracing console via Jaeger, you first need to do the burying work. This example is a manual burying point.

6.2 the pom configuration

A dependency on the Jaeger client has been added to pom.xml.

<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-client</artifactId>
    <version>${jaeger.version}</version>
</dependency>
Copy the code

6.3 start

Instead of using injected components, the sample code starts manually, with the following startup/initialization code:

public final class Tracing {
    private Tracing(a) {}public static JaegerTracer init(String service) {
        SamplerConfiguration samplerConfig = SamplerConfiguration.fromEnv()
                .withType(ConstSampler.TYPE)
                .withParam(1);

        ReporterConfiguration reporterConfig = ReporterConfiguration.fromEnv()
                .withLogSpans(true);

        // Start here
        Configuration config = new Configuration(service)
                .withSampler(samplerConfig)
                .withReporter(reporterConfig);

        returnconfig.getTracer(); }}Copy the code

Example start IO. Dropwizard. Application will be called init is initialized.

try (JaegerTracer tracer = Tracing.init("publisher")) {
    new Publisher(tracer).run("server");
}
Copy the code

Specific startup logic in IO. Jaegertracing. Complete the Configuration. We can see that many configurations and a Tracer are implemented.

6.4 build a Tracer

The previous code had config.gettracer (); That’s why Jaeger used the Builder model to build Tracer.

public class Configuration {
    private String serviceName;
    private Configuration.SamplerConfiguration samplerConfig;
    private Configuration.ReporterConfiguration reporterConfig;
    private Configuration.CodecConfiguration codecConfig;
    private MetricsFactory metricsFactory;
    private Map<String, String> tracerTags;
    private boolean useTraceId128Bit;
    private JaegerTracer tracer;
  
    public synchronized JaegerTracer getTracer(a) {
      if(tracer ! =null) {
        return tracer;
      }

      tracer = getTracerBuilder().build(); / / build
      returntracer; }... }Copy the code

The build() method finally completes the construction of the Tracer object.

Use the defaultRemoteReporterTo reportSpanTo the agent,
Sampling by defaultRemoteControlledSampler.
Common usemetricsIs a member variable that has a default value in the Builder inner classmetrics.

public JaegerTracer build(a) {
  if (reporter == null) {
    reporter = new RemoteReporter.Builder()
        .withMetrics(metrics)
        .build();
  }
  if (sampler == null) {
    sampler = new RemoteControlledSampler.Builder(serviceName)
        .withMetrics(metrics)
        .build();
  }
  return createTracer();
}

protected JaegerTracer createTracer(a) {
      return new JaegerTracer(this);
}
Copy the code

Tracer objects can be used to create Span objects to record distributed operation times, pass data transparently across machines through the Extract/Inject method, or set the current Span. The gateway address, local IP address, sampling rate, and service name of the data to be reported are configured for the Tracer object. Users can adjust the sampling rate to reduce the cost of reporting data.

After startup, the user gets Tracer for subsequent manual burials.

JaegerTracer tracer = Tracing.init("hello-world")
Copy the code

0x07 Sent by client

The following are all manual burial points.

7.1 build a Span

Constructing a Span object is a simple matter. The Tracer interface is described by OpenTracing, and we “started” a Span (actually just constructed the object) :

Span span = tracer.buildSpan("printHello").start();
Copy the code

The start method in Tracer (which opens a Span) uses scopeManager to get context to handle parent-child relationships;

public JaegerSpan start(a) {
      // Get the active Span from the ScopeManager context (thread) here, and then create the parent-child relationship
      if (this.references.isEmpty() && !this.ignoreActiveSpan && null! = JaegerTracer.this.scopeManager.activeSpan()) {
                this.asChildOf(JaegerTracer.this.scopeManager.activeSpan());
      }

      JaegerSpanContext context;
      if (!this.references.isEmpty() && ((Reference)this.references.get(0)).getSpanContext().hasTrace()) {
                context = this.createChildContext();
      } else {
                context = this.createNewContext(); }...return jaegerSpan;
}
Copy the code

7.2 the Parent Span

There are two spans involved in this example: Parent Span and Child Span. Let’s start with Parent Span.

Its general strategy is:

Call tracer.buildspan (“say-hello”).start() to generate a Span
- asChildOf(scopeManager.activeSpan()); This builds relationships between spans, that is, this Span first builds relationships with previous spans when initialized.
- CreateNewContext () or createChildContext(). Random generated ids are used as traceId and spanId if it is root span, or as spanId if it is not root spanreferenceParent span (child_of) gets its traceId as its own traceId and spanId as its parentId.
Calling tracer.scopemanager ().activate puts the span into a newly generated ThreadLocalScope member variable. The result is the follow-up by tracer. ScopeManager. ActiveSpan (); Get span information.
setTag
setBaggageItem
The final finish

The specific code is as follows:

private void sayHello(String helloTo, String greeting) {
        Span span = tracer.buildSpan("say-hello").start();
        try (Scope scope = tracer.scopeManager().activate(span)) {
            span.setTag("hello-to", helloTo);
            span.setBaggageItem("greeting", greeting);
            String helloStr = formatString(helloTo);
            printHello(helloStr);
        } finally{ span.finish(); }}Copy the code

The resulting runtime Span looks like this:

span = {JaegerSpan@1685} 
 startTimeMicroseconds = 1598707136698000
 startTimeNanoTicks = 1018098763618500
 computeDurationViaNanoTicks = true
 tags = {HashMap@1700}  size = 2
 durationMicroseconds = 0
 operationName = "say-hello"
 references = {ArrayList@1701}  size = 0
 context = {JaegerSpanContext@1666} "c8b87cc5fb01ef31:c8b87cc5fb01ef31:0:1"
  traceIdLow = -3983296680647594191
  traceIdHigh = 0
  spanId = -3983296680647594191
  parentId = 0
  flags = 1
  baggage = {Collections$EmptyMap@1704}  size = 0
  debugId = null
  objectFactory = {JaegerObjectFactory@994} 
  traceIdAsString = "c8b87cc5fb01ef31"
  spanIdAsString = "c8b87cc5fb01ef31"
 logs = null
 finished = false
Copy the code

7.3 the Child Span

The sample code then in the formatString will:

Generate a subspan
Joined the Tag
Call the Inject method to pass in the Context information.
And the HTTP request will be invoked.

The specific code is as follows:

private String getHttp(int port, String path, String param, String value) {
		HttpUrl url = new HttpUrl.Builder().scheme("http").host("localhost").port(port).addPathSegment(path)
                    .addQueryParameter(param, value).build();
		Request.Builder requestBuilder = new Request.Builder().url(url);
  
  	Span activeSpan = tracer.activeSpan();

    Tags.SPAN_KIND.set(activeSpan, Tags.SPAN_KIND_CLIENT);
    Tags.HTTP_METHOD.set(activeSpan, "GET");
    Tags.HTTP_URL.set(activeSpan, url.toString());

    tracer.inject(activeSpan.context(), Format.Builtin.HTTP_HEADERS, 
                  Tracing.requestBuilderCarrier(requestBuilder));

    Request request = requestBuilder.build();
    Response response = client.newCall(request).execute();
}  
Copy the code

7.4 the Inject

The tracer.inject function above is used to serialize SpanContext information into request. Builder. This allows subsequent operations to convert the serialized information into the Header.

tracer.inject(activeSpan.context(), Format.Builtin.HTTP_HEADERS, 
              Tracing.requestBuilderCarrier(requestBuilder));
Copy the code

The serialization code is as follows:

public void inject(JaegerSpanContext spanContext, TextMap carrier) {
  carrier.put(contextKey, encodedValue(contextAsString(spanContext)));
  for(Map.Entry<String, String> entry : spanContext.baggageItems()) { carrier.put(keys.prefixedKey(entry.getKey(), baggagePrefix), encodedValue(entry.getValue())); }}Copy the code

7.5 Finish

When the server returns, on the Client, Jaeger does the following: Finish, report.

Calling the sp.finish () method marks the end of the span. The Finish method should be the last method invoked for the SPAN instance. Finish is just a checksum record in a span. The actual span is sent by tracer. Tracer contains sampler, report, and other global functions, so tracer.report(SPAN) is called in Finish. The report method in Tracer is the report method that uses its member report. As mentioned above, the default implementation is RemoteReporter, which uses UdpSender by default.

Span. finish triggers a SPAN report. Call the JaegerSpan finishWithDuration. It determines whether the Trace is sampled. If it was sampled, it would be reported.

  @Override
  public void finish(long finishMicros) {
    finishWithDuration(finishMicros - startTimeMicroseconds);
  }

  private void finishWithDuration(long durationMicros) {
    synchronized (this) {
      if (finished) {
        log.warn("Span has already been finished; will not be reported again.");
        return;
      }
      finished = true;

      this.durationMicroseconds = durationMicros;
    }

    if (context.isSampled()) {
      tracer.reportSpan(this); }}Copy the code

7.6 Reporter

The report is in RemoteReporter.

There is a BlockingQueue queue in RemoteReporter that receives the implementation class of the Command interface, the length of which can be passed in the constructor. Two daemons are started in RemoteReporter’s constructor. One thread periodically adds flush to the BlockingQueue queue, and the other thread continually takes data from the BlockingQueue queue and then executes the command-.excute () method. The Report (SPAN) method adds the AppendCommand class to the BlockingQueue queue.

  @Override
  public void report(JaegerSpan span) {
    // Its better to drop spans, than to block here
    boolean added = commandQueue.offer(new AppendCommand(span));

    if(! added) { metrics.reporterDropped.inc(1); }}Copy the code

It can be seen that if the added variable returned is false, that is, the queue is full and no more data can be added, the SPAN will be discarded, and the information of the SPAN will not be sent to the agent at last. So the length of the queue also matters.

The AppendCommand excute() method is:

class AppendCommand implements Command {
    private final Span span;

    public AppendCommand(Span span) {
      this.span = span;
    }

    @Override
    public void execute(a) throws SenderException { sender.append(span); }}Copy the code

So, we see that the execute() method doesn’t really send a span anymore, it just adds a span to the sender, which implements the sending of a span, and the Reporter class only sends the refresh and send commands.

If we drill down, we can see that UdpSender is an implementation of the abstract class ThriftSender. The sender.appEnd (span) method calls the append(span) method of ThriftSender, This, in turn, calls flush() on ThriftSender, and finally, this flush() calls send(Process Process, List Spans) on this abstract class ThriftSender.

Other reporters in Jaeger are as follows:

CompositeReporterAs the name implies, it is a combination of each reporter, internal has a list, it implements the interfacereport(Span span)The reporter method simply calls all reporters in the list in turnreport(Span span)It’s just method.
InMemoryReporterClass is toSpanThis class contains a list to store the span. The report method in this class is to add the span to the list through the add methodgetSpans()The getList () method gets the listclear()Method to clear the list data.
LoggingReporterClass to print span as log content, its report method islog.info()Prints the contents of span.
NoopReporterIt’s an implementationReporterInterface but implements a class with an empty method, meaning that using this class report Span has no impact.

0x08 Server Accepts

8.1 Manual burying point

The server side is also a manual burial point.

public class FormatterResource {
    @GET
    public String format(@QueryParam("helloTo") String helloTo, @Context HttpHeaders httpHeaders) {
        Span span = Tracing.startServerSpan(tracer, httpHeaders, "format");
        try (Scope scope = tracer.scopeManager().activate(span)) {
            String greeting = span.getBaggageItem("greeting");
            if (greeting == null) {
                greeting = "Hello";
            }
            String helloStr = String.format("%s, %s!", greeting, helloTo);
            span.log(ImmutableMap.of("event"."string-format"."value", helloStr));
            return helloStr;
        } finally{ span.finish(); }}}Copy the code

8.2 Service Logic

The business logic is in startServerSpan:

Call the Extract method to parse the Context information.
Span is built based on whether there is a Parent Context, and SpanContext is used.

The specific code is as follows:

public static Span startServerSpan(Tracer tracer, javax.ws.rs.core.HttpHeaders httpHeaders, String operationName) {
        // format the headers for extraction
        MultivaluedMap<String, String> rawHeaders = httpHeaders.getRequestHeaders();
        final HashMap<String, String> headers = new HashMap<String, String>();
        for (String key : rawHeaders.keySet()) {
            headers.put(key, rawHeaders.get(key).get(0));
        }

        Tracer.SpanBuilder spanBuilder;
        try {
            SpanContext parentSpanCtx = tracer.extract(Format.Builtin.HTTP_HEADERS, new TextMapAdapter(headers));
            if (parentSpanCtx == null) {
                spanBuilder = tracer.buildSpan(operationName);
            } else{ spanBuilder = tracer.buildSpan(operationName).asChildOf(parentSpanCtx); }}catch (IllegalArgumentException e) {
            spanBuilder = tracer.buildSpan(operationName);
        }
        // TODO could add more tags like http.url
        return spanBuilder.withTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_SERVER).start();
}

Copy the code

8.3 analytic Context

The parse code is as follows:

public JaegerSpanContext extract(TextMap carrier) {
  JaegerSpanContext context = null;
  Map<String, String> baggage = null;
  String debugId = null;
  for (Map.Entry<String, String> entry : carrier) {
    // TODO there should be no lower-case here
    String key = entry.getKey().toLowerCase(Locale.ROOT);
    if (key.equals(contextKey)) {
      context = contextFromString(decodedValue(entry.getValue()));
    } else if (key.equals(Constants.DEBUG_ID_HEADER_KEY)) {
      debugId = decodedValue(entry.getValue());
    } else if (key.startsWith(baggagePrefix)) {
      if (baggage == null) {
        baggage = new HashMap<String, String>();
      }
      baggage.put(keys.unprefixedKey(key, baggagePrefix), decodedValue(entry.getValue()));
    } else if(key.equals(Constants.BAGGAGE_HEADER_KEY)) { baggage = parseBaggageHeader(decodedValue(entry.getValue()), baggage); }}if (debugId == null && baggage == null) {
    return context;
  }
  return objectFactory.createSpanContext(
    context == null ? 0L : context.getTraceIdHigh(),
    context == null ? 0L : context.getTraceIdLow(),
    context == null ? 0L : context.getSpanId(),
    context == null ? 0L : context.getParentId(),
    context == null ? (byte)0 : context.getFlags(),
    baggage,
    debugId);
}
Copy the code

0x09 Problem solved

How does Jaeger compare with SOFATracer?
- Jaeger has more complete support for OpenTracing, with higher versions.
How is spanId generated and what are the rules?

How is traceId generated and what are the rules?

ThreadLocalRandom # current # nextLong = ThreadLocalRandom # current # nextLong

traceIdLow = -4423486945480775652
traceIdHigh = 0
spanId = 3900526584756421192
parentId = -4423486945480775652
Copy the code

Where does the client generate the Span?
- This example code generates a Span manually by calling tracer.buildspan (“say-hello”).start().
Where does ParentSpan come from?
- In the client send phase, the current activity span is first retrieved from scopemanager.Activespan. If it is not empty, you need to set the parent span for the new span.
  - ```
  if(references.isEmpty() && ! ignoreActiveSpan &&null! = scopeManager.activeSpan()) { asChildOf(scopeManager.activeSpan()); }Copy the code
```
ChildSpan was created by ParentSpan, so when was it created?
- In the OpenTracing-Java implementation, is in thetracer.start()Of a method;start()ScopeManager determines if active span exists. If it exists, CHILD_OF generates the context; if not, createNewContext.
How do Trace messages pass?
- Serialize SpanContext information into request. Builder. Subsequent operations convert the serialized information into the Header, which can then be passed.
What does the server do after receiving the request?
- Call the Extract method to parse the Context information.
- Span is built based on whether there is a Parent Context, and SpanContext is used.
- Carry out specific other business.
How is SpanContext handled on the server side? See answers to the above questions.
How to collect link information?
- Sampling is for the entire link, meaning that from the time RootSpan is created, it determines whether the current link data will be recorded.
- If it is determined that this Trace has been sampled, a report will be sent.

0xEE Personal information

★★★★ Thoughts on life and technology ★★★★★

Wechat official account: Rosie’s Thoughts

If you want to get a timely news feed of personal articles, or want to see the technical information of personal recommendations, please pay attention.

0 XFF reference

Distributed tracing system — Opentracing

Introduction to Open Distributed Tracing with Jaeger implementation

Description of OpenTracing semantics

Overview of distributed tracking systems and comparison of mainstream open source systems

Skywalking Distributed Tracking and Monitoring: The beginning

Distributed Full link Monitoring – OpenTracing Demo

Opentracing of actual combat

Go microservices full link tracing details

OpenTracing Java Library Tutorial (3) — Passing SpanContext across services

OpenTracing Java Library Tutorial (1) introduction to Trace and Span

Ant gold uniform distributed link tracking component SOFATracer overview | anatomy

Analysis on the principle of transparent link transmission of Ant Financial open source distributed link tracking component SOFATracer and the extended capability of SLF4J MDC

Ant Financial open source distributed link tracking component SOFATracer sampling strategy and source code analysis

Github.com/sofastack-g…

The OpenTracing Semantic Specification

OpenTracing Java Library Tutorial (2) — Passing SpanContext between processes

OpenTracing Java Library Tutorial (4) — Baggage Introduction

Github.com/yurishkuro/…

A distributed traceId reference implementation for microservices architecture

Monitoring the traceid

Jaeger code reading ideas

How to elegantly trace traceid logs in distributed Systems

Description Sky-walking traceId is generated

Distributed link tracking series one (Jaeger asynchronous batch send SPAN)

Distributed Link Tracing (Spark Job)

Jaeger server buried point analysis

Report Java application data through Jaeger

OpenTracing(Jaeger) encounters multithreading

OpenTracing – Java Scope and ScopeManager

OpenTracing- Ten questions about the Soul of Java implementations

OpenTracing- Java

OpenTracing API automatic burial point investigation

Jaeger server buried point analysis

OpenTracing(Jaeger) encounters multithreading

Jaegeropentracing complete java-client distributed tracing chain

Full link tracing was implemented based on OpenTracing + Jaeger

Jaeger code reading ideas

Learn the distributed tracing system Opentracing with Jaeger

0 x00 the

0x01 Cause & Problem

1.1 choose Jaeger

1.2 the problem

1.3 Scope of this article

1.3.1 Jaeger constitute

1.3.2 Full link Tracing

0x02 Background

0x03 Sample code

3.1 code

3.2 dropwizard

0x04 Link Logic

0x05 Data model

5.1 Tracer & JaegerTracer

5.2 Span & JaegerSpan

5.3 SpanContext & JaegerSpanContext

5.4 Reporter

5.5 the Scope

5.6 ScopeManager

5.7 SpanID & TraceID

0 x06 start

6.1 Manual Burying Point

6.2 the pom configuration

6.3 start

6.4 build a Tracer

0x07 Sent by client

7.1 build a Span

7.2 the Parent Span

7.3 the Child Span

7.4 the Inject

7.5 Finish

7.6 Reporter

0x08 Server Accepts

8.1 Manual burying point

8.2 Service Logic

8.3 analytic Context

0x09 Problem solved

0xEE Personal information

0 XFF reference

Related Posts

Introduction to docker

Spring source code parsing | core functionality

Mobile Brain -SpringMVc to build RestFul background Service (I)