Under the existing microservice architecture, a large software system may cover dozens of service units, and the invocation of each service is complicated. It may be necessary to invoke N services for one customer request to form a business closed loop. When encountering a bug, developers have to check each service log one by one. The whole process is time-consuming and inefficient, and may even lead to the system being unavailable for a long time, directly causing a large amount of business loss.

In response to this, Google developed the open source Dapper link tracking component, In 2010, he published the paper “Dapper, A Large-scale Distributed Systems Tracing Infrastructure”.

Link: static.googleusercontent.com/media/resea…

Since its publication, this paper has been the benchmark and theoretical basis for the realization of link tracing in the industry and has great reference value. Well-known link tracking components such as Uber’s Jeager, Twitter’s Zipkin, Ali’s Eagleeye, Skywalking and DdTrace were all developed based on this paper.

To put it simply, link tracing is to track the whole process of a request from the release to the response, and learn the detailed process of each request, such as which services are involved, in what order, and how many times each service invoked the database. This way, when abnormal problems occur, development can quickly locate the root cause of the problem and resolve the problem quickly.

Link tracing benefits:

  • Service relationships: Dependencies or invocation relationships between services can be clearly displayed.

  • Fault location: can quickly and efficiently locate the problem when the system fails;

  • System monitoring: when the system service performance deteriorates, it can be found in time and preventive measures can be taken in advance.

  • System bottleneck analysis and optimization: Based on full-link analysis, it is easy to find the system performance bottleneck points and make optimization improvements to verify whether the optimization measures are effective.

Link tracking technology is warmly welcomed by the market, and related monitoring products emerge one after another. However, due to different technical details, such as different data encoding formats (JSON, Protobuf, thrift, etc.), different data transmission modes (HTTP, UDP, RPC, etc.), and different APIS of the same language SDK, the interoperability of different products and clients is very poor. To solve this problem, OpenTracing emerged. OpenTracing has developed a set of platform-independent and vendor-independent link-tracing API specifications. As long as each vendor implementing the link-tracing technology complies with the specifications, there is no extra work when switching from one technology implementation to another. Developers only need to make a few configuration changes in the initialization section.

OpenTracing GitHub: github.com/opentracing

The emergence of OpenTracing has reduced the workload of development coding. However, with the continuous development of cloud computing technology and the continuous adjustment of enterprise system and product structure, enterprises are still unable to get rid of the development dilemma of bug locating and data monitoring caused by different data format conversion, storage mode and front-end UI interface style.

Is there a product that is compatible with the mainstream link tracking technology on the market, but also can analyze and display data in multiple dimensions?

3 steps to play link tracking, easy to locate bugs!

Domestic DataFlux — one-stop data monitoring cloud platform, not only compatible with Jeager, Zipkin, Skywalking and DDTrace and other mainstream technologies, but also to help users focus on business development, more intuitive, more professional, more efficient display of data monitoring and analysis results.

On DataFlux we can follow the following three steps for distributed link tracing:

  • Enable link data collection at Datakit;

  • Enable applications that need to be monitored (some require code burying points depending on which link-tracing technology is used);

  • View link data on the DataFlux front-end page.

01 DataKit Enables link data collection

Within DataFlux there is a tool dedicated to all kinds of data acquisition – DataKit. For link data, it provides four types of collectors corresponding to different technologies: traceJaeger, traceZipkin, traceSkywalking, and DDTrace. Using DDTrace as an example, which does not require a code burial point, let’s take a look at its basic use on Linux.

Quick Installation of DataKit Collector in 3 Minutes

Help. Dataflux. Cn/doc / 0 c6ebce…

After installed DataKit, in/usr/local/cloudcare dataflux/DataKit/conf. D/ddtrace/directory, a copy of ddtrace link data collection configuration.

$ sudo cp ddtrace.conf.sample ddtrace.conf
Copy the code

Edit ddtrace. Conf:

[inputs. Ddtrace] # traces = "/v0.4/traces" The default path is the same as that officially defined by ddtrace. # [inputs. Ddtrace. Tags] # custom tag group # tag1 = "tag1" # custom tag1 # tag2 = "tag2" # custom tag2 # tag3 = "tag3" # # env = "your_env_name" # version = "your_version" # set version informationCopy the code

At this point, the link data acquisition is configured, restart the DataKit.

Help. Dataflux. Cn/doc / 0 c6ebce…

02 Enable the application to be monitored

Data collection through DDTARCE should refer to the Datadog Tracing help document based on the current project development language.

Here’s a Python application as a demonstration:

The first step is to install the dependency packages

pip install ddtrace
Copy the code

The second step is to set the reporting address during application initialization

Os. environ["DD_SERVICE"] = "your_service_name" Os. environ["DD_TAGS"] = "project:your_project_name,env=test,version=v1" Configure (# datakit hostname="127.0.0.1", # datakit port="9529",)Copy the code

Step 3: Start the application

ddtrace-run python your_app.py
Copy the code

If gunicorn is used, perform the following configuration during application initialization; otherwise, the same traceID will be generated

patch(gevent=True)
Copy the code

In other languages, you can view the link data in DataFlux Studio’s Link Trace in about 1-2 minutes after the configuration is successful.

In addition to setting the project name, environment name, and version number during application initialization, you can also set it in the following two ways:

1. Set environment variables

export DD_TAGS="project:your_project_name,env=test,version=v1"
Copy the code

2. Customize the label on the collector

[inputs. Ddtrace] path = "/v0.4/traces" The default path is the same as that officially defined by DDTrace [elsions.ddtrace. tags] # Custom tag group project = "your_project_name" # set the project name env = "your_env_name" # Set the environment name Version = "your_version" # set version informationCopy the code

03 Viewing link Data Collection

Next, we can see the corresponding link data in DataFlux platform:

Statistics related to each service:

Call details:

Invocation relationships between services:

With the popularization and development of science and technology, link tracing technology will directly have a significant impact on the solution of enterprise or individual system anomalies, business bugs and other problems, and will become the consensus of more and more enterprise or individual developers.