Sentry Monitor - Distributed Tracing

A series of

1 minute Quick use of The latest version of Docker Sentry-CLI – create version
Quick use of Docker start Sentry-CLI – 30 seconds start Source Maps
Sentry For React
Sentry For Vue
Sentry-CLI usage details
Sentry Web Performance Monitoring – Web Vitals
Sentry Web performance monitoring – Metrics
Sentry Web Performance Monitoring – Trends
Sentry Web Front-end Monitoring – Best Practices (Official Tutorial)
Sentry Backend Monitoring – Best Practices (Official Tutorial)
Sentry Monitor – Discover Big data query analysis engine
Sentry Monitoring – Dashboards Large screen for visualizing data
Sentry Monitor – Environments Distinguishes event data from different deployment Environments
Sentry monitoring – Security Policy Reports Security policies
Sentry monitoring – Search Indicates the actual Search
Sentry monitoring – Alerts Indicates an alarm

What is stalking?

First, note what tracing is not: Tracing is not analysis. Although there is considerable overlap between the targets for analysis and tracking, and although both can be used to diagnose problems in applications, they differ in what is measured and how data is recorded.

Profilers can measure many aspects of an application’s operations: the number of instructions executed, the amount of memory used by various processes, the amount of time taken by a given function call, and so on. The generated profile is a statistical summary of these measurements.

En.wikipedia.org/wiki/Profil…

On the other hand, the tracing tool focuses on what happened (and when), rather than how many times it happened or how long it took. Result traces are logs of events that occur during program execution, typically across multiple systems. Although traces are most common — or, in the case of Sentry traces, always — include timestamps that allow calculating durations, measuring performance is not their sole purpose. They can also show how interconnected systems interact and how problems in one system can cause problems in another.

En.wikipedia.org/wiki/Tracin…

Why are we tracking?

Applications typically consist of interconnected components, also known as services. As an example, let’s look at a modern Web application that consists of the following components, separated by network boundaries:

Frontend (Single-Page Application)The front end
Backend (REST API)The back-end
Task QueueTask queue
Database ServerDatabase server
Cron Job SchedulerScheduled task scheduler

Each of these components can be written in a different language on a different platform. Each can be individually detected using the Sentry SDK to catch error data or crash reports, but this detection does not provide a complete picture because each part is considered separately. Tracing allows you to tie all your data together.

In our example Web application, tracing means being able to track requests from the front end to the back end and extract data from any background tasks or notification jobs created by the request. Not only does this allow you to correlate Sentry error reports to see how errors in one service propagate to another, but it also gives you more insight into which services might have a negative impact on the overall performance of your application.

Before you learn how to enable tracing in your application, it helps to know some of the key terms and their relationships.

Traces, Transactions, and Spans

Trace represents a record of the entire operation that you want to measure or track – such as a page load, an instance of a user completing something in the application, or a cron job on the back end. When tracing includes work in multiple services, such as the services listed above, it is called distributed tracing because the tracing is distributed among these services.

Each trace consists of one or more tree structures called Transactions, whose nodes are called SPANS. In most cases, each transaction represents a single instance of the service being invoked, and each span in that transaction performs a single unit of work on behalf of that service, whether calling functions within that service or calling different services. Here’s a sample trace, broken down into Transactions and Spans:

Because transactions have a tree structure, top-level SPANS themselves can be broken into smaller spans, reflecting the ways a function can call many other, smaller spans. This is expressed using the parent-child metaphor, so each span may be the parent span of multiple other child spans. In addition, because all trees must have a root, one span in each transaction always represents the transaction itself, while all other spans in the transaction fall from that root span. Here’s a larger view of one of the transactions above:

To make all of this more concrete, let’s consider our sample Web application again.

Example: Investigate slow page loading

Suppose your Web application loads slowly and you want to know why. Many things must happen to make your application usable in the first place: multiple requests to the back end, possibly some work — including calls to a database or an external API — is done before the response is returned and handled by the browser to render all the returned data into something meaningful to the user. So what part of the process slows down?

Suppose that in this simplified example, the following happens in each service when the user loads the application in the browser:

Browser(Browser)
- HTML,CSS 和 JavaScript 各 1A request
- 1Second render task, triggered2 次 JSONData request ^
Backend(back end)
- 3Provides static files (HTML,CSS 和 JSThe request of the)
- 2 个 JSONData request –1A need to call the database –1You need to call externallyAPIAnd process the result ^ before returning it to the front end
Database Server(Database server)
- 1One request is required2A query
  - 1Query to check authentication
  - 1Query to obtain data

Note: The external API is not listed precisely because it is external, so you can’t see its internals.

In this example, the entire page loading process, including all of the above, is represented by a single trace. The trace will consist of the following transactions:

1Four browser transactions (for page loading)
5Four back-end transactions (one per request)
1Database server transactions (for a singleDBRequest)

Each transaction is decomposed into the spans:

Browser page load transaction:7 个 span
- 1A rootspanRepresents the entire page load
- HTML,CSS 和 JSRequest the1A (total3A)
- Render task1 个 span, which itself contains
  - 2childspan, eachJSONTo request a

Let’s pause here to make an important point: some (though not all) of the spans of the browser transactions listed here correspond directly to the back-end transactions listed earlier. Specifically, each request span in a browser transaction corresponds to a separate request transaction in the back end. In this case, when a span in a service causes a transaction in a subsequent service, we refer to the original span as the parent span of the transaction and its root span. In the figure below, wavy lines represent this parent-child relationship.

In our example, every transaction other than the initial browser page load transaction is a child of a span in another service, meaning that every root span other than the browser transaction root has a parent span (albeit in a different service).

In fully instrumented systems (where every service has tracing enabled), this pattern will always apply. The only fatherless span will be the root of the initial transaction; Every other span has a parent. In addition, parents and children will always live in the same service, except in cases where the child span is the root of the child transaction, in which case the parent span will call the service, The sub-Transaction/Child root span will be in the invoked service.

In other words, a fully-instrumented system creates a trace that is itself a tree of connections — each transaction is a subtree — where the boundaries between subtrees/transactions are the boundaries between services. The figure above shows a branch of the complete trace tree for our example.

Now, just to be complete, go back to our SPANS:

Back-end HTML/CSS/JS request transactions: 1 eachspan
- One root span (a child of the browser span) ^ representing the entire request
Back-end requests with database invocation transactions: 2span
- 1 represents the root span of the entire request (a child of the browser span)
- One span is used to query the database (the parent of the database server transaction) ^
Back-end requests with API call transactions: 3span
- 1 represents the root span of the entire request (a child of the browser span)
- 1 span of THE API request (unlike the database call, not the parent span because the API is external)
- One span is used for processingAPIData ^
Database server requests transactions: threespan
- 1 represents the root span of the entire request (subterm of the upper back-end span)
- 1 span is used for authentication queries
- One span is used to query the retrieved data

To summarize the example: After checking all the services, you might find that, for some reason, the auth queries in the database server are causing the slowdowns, accounting for more than half of the time it takes to complete the entire page load process. Tracking won’t tell you why this is happening, but at least now you know where to look!

More examples

This section contains more trace examples, divided into transactions and spans.

Measure specific user actions

If your application involves e-commerce, you might want to measure the time between when the user clicks Submit Order and when the Order confirmation appears, including tracking the submission of fees to the payment processor and sending the Order confirmation email. The whole process is a trace, and normally you would have transactions (T) and spans (S) for:

Browser process (TAnd root spanS)
- On the back endXHRRequest * (S)
- Apply colours to a drawing to confirmscreen(S) ^
Your back-end processing of this request (TAnd root spanS)
- A function to calculate the total (Function) call(S)
- Store the order database (DB) call *(S)
- For the payment processorAPIcall(S)
- Email confirmation queue *(S) ^
Your database updates the customer order history (TAnd root spanS)
- A singleSQLThe query(S) ^
Queuing tasks for sending E-mail messages (TAnd root spanS)
- A function call to populate the E-mail template(S)
- To the email sending serviceAPIcall(S)

Note: The span marked with an asterisk represents the parent span of the subsequent transaction (and its root span).

Monitoring Background Processes

If your back end periodically polls data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you typically have the following transactions (T) and spans (S) :

To complete the processcron job(TAnd root spanS)
- APIInvoking external services(S)
- Processingfunction(S)
- Invoke the cache service *(S)
- API calls internal service *(S) ^
Work done in your caching service (TAnd root spanS)
- Check the cache of existing data(S)
- Store new data in the cache(S) ^
The processing of requests by your internal services (TAnd root spanS)
- Anything a service might do to process a request(S)

Note: The span marked with an asterisk represents the parent span of the subsequent transaction (and its root span).

Tracking data model

“Show me your flow chart and hide your chart, I’m still confused. If you show me your chart, I won’t need your flow charts anymore because they’re too obvious.”

6. Fred Brooks, “The Mythical Man-Month”

While this theory is interesting, ultimately any data structure is defined by the data types it contains, and the relationships between data structures are defined by how the links between them are recorded. Tracing, transactions, and spans are no exception.

Traces (track)

Traces itself is not an entity. Instead, a trace is defined as a collection of all transactions that share a trace_ID value.

The Transactions (Transactions)

Transactions shares most of its properties (start and end times, labels, and so on) with its root span, so the same options for the spans described below are available in Transactions, and they are equivalent when set in either place.

Transactions has an additional property that is not included in the span, called transaction_NAME, which is used in the UI to identify Transactions. Common examples of transaction_name values include the endpoint path of a back-end request transaction (e.g. /store/checkout/ or API /v2/users/

/), the task name of a Cron job transaction (e.g Data. The cleanup. Delete_inactive_users) and URL (like https://docs.sentry.io/performance-monitoring/distributed-tracing/) Used for page-load transactions.

Spans (span)

Most of the data in a transaction resides in a single span contained in the transaction. Span data includes:

parent_span_idWill:spanWith his fatherspanconnect
op: a short string that identifies the type or category of operation the span is measuring
start_timestamp: spanWhen they open
end_timestamp: spanclosed
description: spanA long description of an operation, uniquely identifiedspanBut acrossspanInstance consistency (optional)
status: Indicates the short of operation statuscode(optional)
tags: key-valueSave additional data about the span (optional)
data: aboutspanAdditional data for any structure of (optional)

Examples of op and description attributes used together are op: sql.query and description: SELECT * FROM users WHERE last_active < %s. The status attribute is typically used to indicate the success or failure of a SPAN operation or, in the case of AN HTTP request, response code. Finally, tags and data allow you to attach more context information to a span, such as function: middleware.auth.is_authenticated for function calls or request: {URL:… , headers: … , body: … } for HTTP requests.

For more information

Some more important points about tracing, transactions, and spans and the way they relate to each other:

Trace Duration

Because a trace is just a collection of transactions, it does not have its own start and end times. Instead, trace starts when its earliest transaction begins and ends when its latest transaction ends. Therefore, you cannot explicitly start or end trace directly. Instead, you create a trace by creating the first transaction in the trace and complete the trace by completing all transactions it contains.

Async Transactions

Because of the possibility of asynchronous processes, child transactions can be orders of magnitude longer than transactions with parent spans. For example, if a back-end API call starts a long-running processing task and then immediately returns a response, the back-end transaction will complete (and its data will be sent to Sentry) long before the asynchronous task transaction completes. Asynchrony also means that the order in which transactions are sent to (and received from) Sentries has nothing to do with the order in which they were created. (By contrast, the received order of transactions in the same trace is correlated with the order of completion, but the correlation is far from perfect due to factors such as variability in transfer times.)

Orphan Transactions

Theoretically, in a fully instrumented system, each trace should contain only one transaction and one span (the root of the transaction), with no parent, i.e., the transaction in the original service. In practice, however, you may not enable Trace in every service, or the detected service may not be able to report a transaction due to a network outage or other unforeseen circumstances. When this happens, you might see gaps in the trace hierarchy. Specifically, you might see a transaction halfway through a SPAN that its parent SPAN has not yet recorded as part of any known transaction. Such an uninitiated, fatherless transaction is called an orphan transaction.

A: Spans a Nested span

Although our example above has four levels in its hierarchy (Trace, transaction, span span, child span), there is no limit to how deep the span can be nested. However, there are practical limitations: transaction payloads sent to Sentry have a maximum allowable size and, as with any type of logging, need to be balanced between the granularity of the data and its availability.

Zero-duration Spans(Zero duration span)

The span may have the same start time and end time and thus be recorded as a non-occupancy time. This may be because span is used as a tag (as is done in the browser’s Performance API), or because the operation takes less time than the measured resolution (which will vary by service).

Developer.mozilla.org/en-US/docs/…

Clock Skew

If you collect transactions from multiple machines, you may encounter clock skew, where the timestamp in one transaction is inconsistent with the timestamp in another transaction. For example, if your back end makes database calls, the back-end transaction logically should start before the database transaction. However, this may not be the case if the system time on each machine (the one hosting the backend and the database, respectively) is not synchronized to a common standard. The ordering may also be correct, but the time frames of the two records are not arranged in a way that accurately reflects what actually happened. To reduce this possibility, we recommend using network Time Protocol (NTP) or your cloud provider’s clock synchronization service.

How to send data

Individual spans are not sent to Sentry; Instead, the entire transaction is sent as a unit. This means that Sentry’s servers do not record any SPAN data until the transaction to which they belong is closed and dispatched. However, the reverse is not true — although a span cannot be sent without a transaction, transactions are still valid and will be sent even if the only span they contain is their root span.

Data sampling

When you enable sampling in trace Settings, you can select the percentage of collected transactions to be sent to Sentry. For example, if you have an endpoint that receives 1000 requests per minute, a sampling rate of 0.25 would result in approximately 250 transactions per minute (25%) being sent to Sentry. (This number is approximate because each request is either tracked or independently and pseudo-randomly, with a probability of 25%. So, in the same way that 100 fair coins, when flipped, result in about 50 heads, the SDK will “decide” to collect traces in about 250 cases.) Because you know the sampling percentage, you can infer your total traffic.

When collecting traces, we recommend sampling your data for two reasons. First, while capturing individual traces is the least expensive, capturing traces per page load or per API request can add an unwanted load to your system. Second, enabling sampling allows you to better manage the number of events sent to Sentry so that you can customize them to your organization’s needs.

When selecting a sampling rate, the goal is not to collect too much data (for the reasons mentioned above), but to collect enough data to draw meaningful conclusions. If you are unsure what rate to choose, we recommend starting with a low value and gradually increasing as you learn about traffic patterns and traffic until you find a rate that allows you to balance performance and traffic with data accuracy.

Consistency in tracing

For tracing involving multiple transactions, Sentry uses a “head-based” approach: a sampling decision is made in the original service and then passed on to all subsequent services. To see how this works, let’s go back to the WebApp example above. Consider two users, A and B, each loading the application in their respective browsers. When A loads the application, the SDK pseudo-randomly “decides” to collect traces, while when B loads the application, the SDK “decides” not to collect traces. As each browser makes requests to your back end, it includes a decision of “Yes, please Collect Transactions” or “No, don’t Collect Transactions this time” in the title of those requests.

When your back end processes the request from browser A, it sees the “yes” decision, collects transaction and span data, and sends it to Sentry. In addition, it includes a “yes” decision in any request it makes to subsequent services, such as your database server, which also collect data, send it to Sentry, and pass the decision to any service they invoke. Through this process, all relevant transactions in A’s trace are collected and sent to Sentry.

On the other hand, when your back end processes a request from the B browser, it sees a “no” decision, so it does not collect and send transaction and span data to Sentry. However, it does the same thing in terms of propagating decisions to subsequent services as it did in the case of A, telling them not to collect or send data either. They then tell any services they invoke not to send data so that transactions from the B trace are not collected.

In short: The result of this head-based approach is that the decision is made once in the original service and passed on to all subsequent services, either collecting all or none of the transactions for a given trace, so there should be no incomplete trace.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Sentry Monitor – Distributed Tracing

A series of

directory