getsentry-snuba

ClickHouse is simply a real-time big data analysis engine, and it’s awesome 😂.

ClickHouse Chinese Manual

Why Snuba?

Sentry already runs on abstract service interfaces called Search, Tagstore (for event tags), and TSDB (a time series database that powers most graphs). Each of these services has its own production implementation, supported by standard relational SQL (for Search and Tagstore) and Redis (for TSDB), which has been used in Sentry for many years.

Our problems began when Sentry expanded its customer base and engineering team. On the one hand, we receive more events every second of every day. On the other hand, we have more engineers trying to develop more features for Sentry.

The increased volume of events means that we have to de-normalize a lot of data so that we can execute known queries very quickly. For example, the Tagstore consists of five different tables that record values, such as the times_seen count for each tag value of each issue on Sentry. (You might have a browser.name tag in one of your issues, Times_seen for Chrome is 10, and times_SEEN for Safari is 7). The increments of these de-normalized counters are buffered so we can merge them and ultimately reduce the write pressure.

Reduce write pressure by buffering increments to a nonnormalized counter

This is useful until we want to add a new dimension to the query, such as environment. Refactoring the existing data layout to de-normalize it on a whole new dimension took us months and required a full backfill of all the event data.

Adding the environment dimension means refactoring the existing data layout, which can cause problems.

It was clear that we needed a flat event model provided by Online Analytical Processing (OLAP) that could perform AD hoc queries without any de-normalization. It needs to be fast enough to satisfy user requests, and there’s no need to overhaul the back end when we want to add another way for users to see their data.

At the time, we thought of Facebook’s op-ed store, Scuba, because it solves a similar problem, but it’s closed source. Our team and project needed a name, and since we weren’t as mature as Scuba, Snuba (a combination of Snorkel and Scuba) was born.

Scuba: Diving into Data at Facebook

Why not just divide (Shard) Postgres?

The main data set responsible for aggregating and providing tag counts (called the “Tagstore”) reached a point where the number of mutations performed exceeded our ability to replicate them on a single Postgres machine. We extended it to a set of machines, but were hobbled by a set of problems that couldn’t be solved with hardware. We need an approach that reduces infrastructure work each time a new dimension of data is discovered, not an approach that extends the current data set. Despite our Postgres expertise, we decided it was time to expand to OLAP systems.

Out of a long list of reasons to switch to OLAP, here are some of our favorites:

  1. For the most part, our data is immutable. Multiversion concurrency control uses a security mechanism that didn’t work for us and ultimately slowed down our performance.

  2. Calculating another dimension of data or introducing another form of Query from the product meant writing new indices and new prayers to Postgres Query Planner to take advantage of them.

  3. Deleting data that has expired beyond the retention window means issuing expensive queries for batch deleted rows.

  4. The proliferation of incoming and outgoing rows affects the Postgres main heap. IO is wasted combing through dead rows to find live ones, and the disks that host these databases grow slowly but steadily.

Why ClickHouse?

We looked at a number of databases in the OLAP scenario, including Impala, Druid, Pinot, Presto, Drill, BigQuery, Cloud Spanner, and Spark Streaming. These are all powerful systems under active development, and the specific strengths and weaknesses of each may have changed since the beginning of 2018. We chose ClickHouse because we had engineers from our newly formed search and storage team each prototype snuna on a different system.

This is where ClickHouse stands out:

  1. It’s open source. We’re open source. Choosing a proprietary solution would chill everyone running Sentry outside of our domain.
  2. Whether it isscale-uporscale-downThe operation is very simple. It doesn’t require any additional services of its own, just the introductionZooKeeperAs a means of copy control. Once we understood its deployment, we spent a day starting to put the whole Sentry event togethervolumeWrite to a single cluster.
  3. Rows are sorted based on primary keys, and columns are stored separately and compressed in physical files. This makes the data behind the Tagstore on disk fromtbBytes intogbBytes.
  4. Data can be queried after real-time writing. Consistent readability allows us to turn everything intoAlert Rules(alert rule) provides supported queries to move toSnuba, which is approximately the number of queries issued per second40%.
  5. query plannerThere’s no magic. If we want to optimize our query pattern, ClickHouse offers a few but effective solutions. Most importantly, they offer thanks to the powerful filtering conditionsPREWHEREThe power of clauses allows us to skip large amounts of data.

Snuba internal

Snuba is a two-part service designed to separate ClickHouse from Sentry. In addition to the application code and ClickHouse, we utilized several other help services to complete Sentry’s event data flow.

Sentry data flow

To read (Reading)

Snuba’s query server is powered by the Flask Web Service, which provides Sentry developers with a rich query interface using JSON Schema. By providing a Snuba Client instead of using ClickHouse SQL directly, we can hide a lot of the potential complexity from application developers. For example, this Snuba Query gets the most popular tags sent to the project in the last 24 hours:

{
  "project": [1]."aggregations": [["count()".""."count"]],"conditions": [["project_id"."IN"[1]]],"groupby": ["tags_key"]."granularity": 3600."from_date": "The 2019-02-14 T20:10:02. 059803"."to_date": "The 2019-05-15 T20:10:02. 033713"."orderby": "-count"."limit": 1000
}
Copy the code

Convert to the corresponding ClickHouse-style SQL query:

SELECT
    arrayJoin(tags.key) AS tags_key,
    count(a)AS count
FROM sentry_dist
PREWHERE project_id IN 1
WHERE (project_id IN 1)
    AND (timestamp > = toDateTime('2019-02-14T20:10:02'))
    AND (timestamp < toDateTime('2019-05-15T20:10:02'))
    AND (deleted = 0)
GROUP BY tags_key
ORDER BY count DESC
LIMIT 0.1000
Copy the code

Presenting this higher-level Snuba query interface – rather than encouraging application developers to interact directly with ClickHouse – enabled our team to keep changes to the underlying data model inside Snuba, rather than requiring developers to constantly change queries as they iterated.

In addition, we are now making centralized changes that affect a wide variety of different query modes. For example, we used Redis to cache individual query results, which consolidated some of our more burst and frequently repeated queries into a single ClickHouse query and removed unnecessary load from the ClickHouse cluster.

Writing (Writing)

Writing data to Snuba begins by reading JSON events from the Kafka topic that have been normalized and processed by Sentry. It processes events in a batch manner, converting each event into a tuple mapped to a single ClickHouse row. Batch ClickHouse inserts are critical because each insert creates a new physical directory with a file for each column and a corresponding record in ZooKeeper. These directories are merged by ClickHouse’s background thread and it is recommended that you write once per second so that there are not too many ZooKeeper or disk file writes to deal with. Data is segmented by time and retention window, which makes it easy to delete data beyond the original retention window.

I am for less.

Wechat: uuhells123.

Public account: Hacker afternoon tea.

Thank you for your support 👍👍👍!