Sentry Monitoring – Snuba Data Mid – platform local development environment configuration combat

A series of

• 1 minute Quick use of The latest version of Docker Sentry-CLI – create version
• Quick use of Docker start Sentry-CLI – 30 seconds start Source Maps
• Sentry For React
• Sentry For Vue
• Sentry-CLI usage details
• Sentry Web Performance Monitoring – Web Vitals
• Sentry Web performance monitoring – Metrics
• Sentry Web Performance Monitoring – Trends
• Sentry Web Front-end Monitoring – Best Practices (Official Tutorial)
• Sentry Backend Monitoring – Best Practices (Official Tutorial)
• Sentry Monitor – Discover Big data query analysis engine
• Sentry Monitoring – Dashboards Large screen for visualizing data
• Sentry Monitor – Environments Distinguishes event data from different deployment Environments
• Sentry monitoring – Security Policy Reports Security policies
• Sentry monitoring – Search Indicates the actual Search
• Sentry monitoring – Alerts Indicates an alarm
• Sentry Monitor – Distributed Tracing
• Sentry Monitoring – Distributed Tracing 101 for Full-stack Developers
• Sentry Monitoring – Snuba Data Mid platform Architecture introduction (Kafka+Clickhouse)
• Sentry – Snuba Data Model
• Sentry Monitoring – Snuba Data Mid-Platform Architecture (Introduction to Query Processing)
• Sentry official JavaScript SDK introduction and debugging guide
• Sentry Monitoring – Snuba Data Mid-platform architecture (writing and testing Snuba queries)
• Sentry Monitoring – Snuba Data Medium Architecture (SnQL Query Language introduction)

Cloning of warehouse

Clone getSentry /sentry and getSentry /snuba respectively:

git clone https://github.com/getsentry/sentry.git
git clone https://github.com/getsentry/snuba.git
Copy the code

Installing system dependencies (Mac as an example)

Xcode CLI tools

xcode-select --install
Copy the code

Brewfile

Enter the Sentry folder and you will see a Brewfile:

cd sentry
Copy the code

Brewfile

# required to run devservices
cask 'docker'

brew 'pyenv'

# required for pyenv's python-build
brew 'openssl'
brew 'readline'

# required for yarn test -u
brew 'watchman'

# required to build some of sentry's dependencies
brew 'pkgconfig'
brew 'libxslt'
brew 'libxmlsec1'
brew 'geoip'

# Currently needed because on Big Sur there's no wheel for it
brew 'librdkafka'

# direnv isn't defined here, because we have it configured to check for a bootstrapped environment.
# If it's installed in the early steps of the setup process, it just leads to confusion.
# brew 'direnv'

tap 'homebrew/cask'

# required for acceptance testing
cask 'chromedriver'
Copy the code

If you have Docker Desktop installed locally and have it started, you can comment out cask ‘Docker’.

Next, run:

brew bundle --verbose
Copy the code

If you don’t have Docker Desktop locally, you’ll need to start it manually:

open -g -a Docker.app
Copy the code

Building a tool chain

Sentry relies on Python Wheels (a package containing binary extension modules) and is officially distributed for the following platforms:

• LinuxCompatible withPEP-513 (manylinux1)
• MacOS 10.15Or higher

If your development machine is not running one of these systems, you will need to install the Rust tool chain. Follow the instructions at https://www.rust-lang.org/tools/install to install the compiler and related tools. After installation, the Sentry installer will automatically build all binary modules using Rust with no additional configuration.

Officials usually keep track of the latest stable Rust release, which is updated every six weeks. So make sure to keep your Rust tool chain up to date by running it occasionally:

rustup update stable
Copy the code

Python

Sentry uses PyEnv to install and manage versions of Python. It is installed when you run the Brew bundle.

To install the required version of Python, you need to run the following command. This will take a while, because your computer is actually compiling Python!

make setup-pyenv
Copy the code

This assumes that you are a Zsh user.

If you type which python, you should see something like $HOME/.pyenv/shims/python instead of /usr/bin/python. This is because the following has been added to your startup script:

Cat ~/.zprofile, you should see the following:

# MacPorts Installer addition on 2021-10-20_at_11:48:22: adding an appropriate PATH variable for use with MacPorts.
export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
# Finished adapting your PATH environment variable for use with MacPorts.

# It is assumed that pyenv is installed via Brew, so this is all we need to do.
eval "$(pyenv init --path)"
Copy the code

A virtual environment

You are now ready to create the Python virtual environment. Run:

python -m venv .venv
Copy the code

And activate the virtual environment:

source .venv/bin/activate
Copy the code

If everything is normal, operation which python should now will lead to similar/Users/you/sentry /. Venv/bin/python.

Snuba Indicates the actual configuration

Start the Snuba dependency container

cd ../sentry
git checkout master
git pull

source .venv/bin/activate
sentry devservices up --exclude=snuba

# 11:17:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
# 11:18:01 [INFO] sentry.plugins.github: apps-not-configured
# > Pulling image 'Postgres :9.6-alpine'
# > Pulling Image 'Yandex/Clickhouse-server :20.3.9.70
# > Not starting container 'sentry_relay' because it should be started on-demand with devserver.
# > Creating 'sentry_redis' volume
# > Creating 'sentry_zookeeper_6' volume
# > Creating 'sentry_kafka_6' volume
# > Creating container 'sentry_redis'
# > Creating container 'sentry_zookeeper'
# > Creating container 'sentry_kafka'
# > Starting container 'sentry_redis' (listening: ('127.0.0.1', 6379))
# > Starting container 'sentry_kafka' (listening: ('127.0.0.1', 9092))
# > Starting container 'sentry_zookeeper'
# > Creating 'sentry_clickhouse' volume
# > Creating container 'sentry_clickhouse'
# > Creating 'sentry_postgres' volume
# > Creating 'sentry_wal2json' volume
# > Starting container 'sentry_clickhouse' (listening: ('127.0.0.1', 9000), ('127.0.0.1', 9009), ('127.0.0.1', 8123))
# > Creating container 'sentry_postgres'
# > Starting container 'sentry_postgres' (listening: ('127.0.0.1', 5432))
Copy the code

This will get the latest version of Sentry on the master and call up all snuba dependencies.

Snuba relies on clickHouse, ZooKeeper, Kafka, and Redis containers.

Docker PS check:

1149 a6f6ff23 postgres: 9.6 - alpine"Docker - entrypoint. S..."3 minutes ago Up 3 minutes 127.0.0.1:5432->5432/ TCP sentry_postgres a7f3af7d52bb Yandex/Clickhouse-server :20.3.9.70"/entrypoint.sh"3 minutes ago Up 3 minutes 127.0.0.1:8123->8123/ TCP, 127.0.0.1:9000->9000/ TCP, 127.0.0.1:9009-68913 > 9009 / TCP sentry_clickhouse ee15c43 confluentinc/cp - zookeeper: 6.2.0"/ etc/confluent/dock..."3 minutes ago Up 3 minutes 2181/ TCP, 2888/ TCP, 3888/ TCP sentry_zookeeper 5a248eb26ed3 confluentInc /cp-kafka:6.2.0"/ etc/confluent/dock..."3 minutes ago Up 3 minutes 127.0.0.1:9092->9092/ TCP sentry_kafka 0573aff7b5af Redis :5.0-alpine"Docker - entrypoint. S..."3 minutes ago Up 3 minutes 127.0.0.1:6379->6379/ TCP sentry_redisCopy the code

Set the Snuba virtual environment

cd snuba
make pyenv-setup
python -m venv .venv
sourceVenv /bin/activate PIP install --upgrade PIP ==21.1.3 make developCopy the code

Viewing the Migration List

snuba migrations list
# system
# [ ] 0001_migrations
#
# events
# [ ] 0001_events_initial
# [ ] 0002_events_onpremise_compatibility
# [ ] 0003_errors
# [ ] 0004_errors_onpremise_compatibility
# [ ] 0005_events_tags_hash_map (blocking)
# [ ] 0006_errors_tags_hash_map (blocking)
# [ ] 0007_groupedmessages
# [ ] 0008_groupassignees
# [ ] 0009_errors_add_http_fields
# [ ] 0010_groupedmessages_onpremise_compatibility (blocking)
# [ ] 0011_rebuild_errors
# [ ] 0012_errors_make_level_nullable
# [ ] 0013_errors_add_hierarchical_hashes
# [ ] 0014_backfill_errors (blocking)
# [ ] 0015_truncate_events
#
# transactions
# [ ] 0001_transactions
# [ ] 0002_transactions_onpremise_fix_orderby_and_partitionby (blocking)
# [ ] 0003_transactions_onpremise_fix_columns (blocking)
# [ ] 0004_transactions_add_tags_hash_map (blocking)
# [ ] 0005_transactions_add_measurements
# [ ] 0006_transactions_add_http_fields
# [ ] 0007_transactions_add_discover_cols
# [ ] 0008_transactions_add_timestamp_index
# [ ] 0009_transactions_fix_title_and_message
# [ ] 0010_transactions_nullable_trace_id
# [ ] 0011_transactions_add_span_op_breakdowns
# [ ] 0012_transactions_add_spans
#
# discover
# [ ] 0001_discover_merge_table
# [ ] 0002_discover_add_deleted_tags_hash_map
# [ ] 0003_discover_fix_user_column
# [ ] 0004_discover_fix_title_and_message
# [ ] 0005_discover_fix_transaction_name
# [ ] 0006_discover_add_trace_id
# [ ] 0007_discover_add_span_id
#
# outcomes
# [ ] 0001_outcomes
# [ ] 0002_outcomes_remove_size_and_bytes
# [ ] 0003_outcomes_add_category_and_quantity
# [ ] 0004_outcomes_matview_additions (blocking)
#
# metrics
# [ ] 0001_metrics_buckets
# [ ] 0002_metrics_sets
# [ ] 0003_counters_to_buckets
# [ ] 0004_metrics_counters
# [ ] 0005_metrics_distributions_buckets
# [ ] 0006_metrics_distributions
# [ ] 0007_metrics_sets_granularity_10
# [ ] 0008_metrics_counters_granularity_10
# [ ] 0009_metrics_distributions_granularity_10
# [ ] 0010_metrics_sets_granularity_1h
# [ ] 0011_metrics_counters_granularity_1h
# [ ] 0012_metrics_distributions_granularity_1h
# [ ] 0013_metrics_sets_granularity_1d
# [ ] 0014_metrics_counters_granularity_1d
# [ ] 0015_metrics_distributions_granularity_1d
#
# sessions
# [ ] 0001_sessions
# [ ] 0002_sessions_aggregates
# [ ] 0003_sessions_matview
Copy the code

Run the migration

snuba migrations migrate --force
#...
# 2021-12-01 19:45:57,557 Running Migration: 0014_metricS_Counters_granularity_1D
# 2021-12-01 19:45:57,575 Finished: 0014_metricS_CounterS_granularity_1D
# 2021-12-01 19:45:57,589 Running Migration: 0015_metricS_distributionS_granularity_1D
# 2021-12-01 19:45:57,610 Finished: 0015_metricS_distributionS_granularity_1D
# 2021-12-01 19:45:57,623 Running Migration: 0001_sessions
# 2021-12-01 19:45:57,656 Finished: 0001_Sessions
Running migration: 0002_sessionS_aggregates
# 2021-12-01 19:45:57,770 Finished: 0002_sessionS_aggregates
# 2021-12-01 19:45:57,792 Running migration: 0003_sessionS_matview
# 2021-12-01 19:45:57,849 Finished: 0003_sessionS_matView
# Finished running migrations
Copy the code

Check the migration

Enter the Clickhouse container:

docker exec -it sentry_clickhouse clickhouse-client
Run the following 'SQL' statement:
select count() from sentry_local

# ClickHouse Client Version 20.3.9.70 (Official build)
# Connecting to localhost:9000 as user default.
# Connected to ClickHouse server version 20.3.9 revision 54433.

# a7f3af7d52bb :) select count() from sentry_local

# SELECT count()
# FROM sentry_local

# ┌ ─ the count () ─ ┐
# │ │ 0
# └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

# 1 Elapsed: 0 SEC

# a7f3af7d52bb :) 
Copy the code

View the entity data set

snuba entities list

# Declared Entities:
# discover
# events
# groups
# groupassignee
# groupedmessage
# metrics_sets
# metrics_counters
# metrics_distributions
# outcomes
# outcomes_raw
# sessions
# org_sessions
# spans
# transactions
# discover_transactions
# discover_events
Copy the code

Start the development server

This command will start the API and all Snuba consumers to ingest data from Kafka:

snuba devserver
Copy the code

Go to http://localhost:1218/events/snql, you will see a simple query UI.