As part of the CI process, we ran a variety of tests in Sentry. This section aims to document some Sentry-specific helpers and provide guidance on what types of tests to consider when building new functionality.

Get set up

Acceptance and python testing require a valid set of devservices. It is recommended to use devservices to ensure that the required services are running. If you are also using the local environment for local tests, you will need to use the –project flag to separate the local test volume from the test suite volume:

# Close the local test service.
sentry devservices down

Open the service with the test prefix to use separate containers and volumes
sentry devservices up --project test

Verify that the test container appears correctly
docker ps --format '{{.Names}}'

# Later when you finish running tests and want to run the local server again
sentry devservices down --project test && sentry devservices up
Copy the code

With the –project option, you can confirm which containers are running Docker PS. Each running container should be prefixed with test_. For more information about managing services, see the devservices docs section.

  • Develop. Sentry. Dev/services/DE…

Python testing

For Python tests, we use pyTest and Django’s test tools. On top of this, we add some basic test cases (in sentry.testutils.cases).

  • docs.pytest.org/en/latest/

Endpoint integration testing is the focus of most of our test suites. These tests helped us ensure that our Customers, Integrations, and front-end application apis continue to work as expected. You should strive to include tests that cover various user roles, cross-organization/team access scenarios, and invalid data scenarios, as these are often overlooked in manual testing.

Run pytest

You can use PyTest to run a single directory, a single file, or a single test, depending on the scope of the change:

Run tests on the entire directory
pytest tests/sentry/api/endpoints/

Run tests on all files in the directory that match the pattern
pytest tests/sentry/api/endpoints/test_organization_*.py

Run tests from a single file
pytest tests/sentry/api/endpoints/test_organization_group_index.py

Run a single testpytest tests/snuba/api/endpoints/test_organization_events_distribution.py::OrganizationEventsDistributionEndpointTest::test_thi s_thingRun all tests in files that match substrings
pytest tests/snuba/api/endpoints/test_organization_events_distribution.py -k method_name
Copy the code

Some common options for PyTest are:

  • -kPass the substring filtering testMethod/class.
  • -sDo not capture standard output when running tests.

See the PyTest documentation for more usage options.

  • Doc.pytest.org/en/latest/u…

Create data in tests

Sentry also adds a set of Factory helper methods that help you build data to write tests against. Sentry. Testutils. Factories in the factory method can be used for all of our test suite. Use these methods to establish the required organization, project, and other Postgres-based states.

You should also use store_event() to store events in a manner similar to what your application does in production. Storing events requires that your tests inherit from SnubaTestCase. When using store_event(), be careful to set the past timestamp on the event. When omitted, timestamp uses ‘now’, which may result in events not being selected due to timestamp boundaries.

from sentry.testutils.helpers.datetime import before_now
from sentry.utils.samples import load_data

def test_query(self) :
    data = load_data("python", timestamp=before_now(minutes=1))
    event = self.store_event(data, project_id=self.project.id)
Copy the code

Set options and feature flags

If your tests are for endpoints with functional markers, or you need to set specific options. You can use helper methods to change the configuration data to the correct state:

def test_success(self) :
    with self.feature('organization:new-thing') :with self.options({'option': 'value'}) :# Run test logic with features and options set.

    # Disable the new-thing feature.
    with self.feature({'organization:new-thing': False}) :# Run you logic with a feature off.
Copy the code

External services

Use the Responses library to add stub responses to outbound API requests made by your code. This will help you simulate success and failure scenarios with relative ease.

Use your time reliably

When writing tests related to ingestion events, we must operate within the constraints of events for no more than 30 days. Because all events had to be recent, we couldn’t use the traditional time-freeze strategy to get consistent data in our tests. Instead of choosing an arbitrary point in time, we work backwards from now on, and there are some helper functions that can do this:

from sentry.testutils.helpers.datetime import before_now, iso_format

five_min_ago = before_now(minutes=5)
iso_timestamp = iso_format(five_min_ago)
Copy the code

These functions generate Datetime objects and datetime strings relative to the current ISO 8601 format, enabling you to have events at known time offsets without violating the 30-day limit imposed by Relay.

Check the SQL query in the test

Add the following to conftest.py in the project root directory:

import itertools
from django.conf import settings
from django.db import connection, connections, reset_queries
from django.template import Template, Context

@pytest.fixture(scope="function", autouse=True)
def log_sql() :
    reset_queries()
    settings.DEBUG = True

    yield

    time = sum([float(q["time"]) for q in connection.queries])
    t = Template(
        "{% for sql in sqllog %}{{sql.sql|safe}}{% if not forloop.last %}\n\n{% endif %}{% endfor %}"
    )
    queries = list(itertools.chain.from_iterable([conn.queries for conn in connections.all()]))
    log = t.render(Context({"sqllog": queries, "count": len(queries), "time": time}))
    print(log)
Copy the code

All SQL executed during the test will now be printed to standard output. It is recommended to use PyTest’s -k selector to narrow the output range. Also note that you need -s to view standard output.

The acceptance test

Our acceptance tests utilized selenium and chromedriver to simulate user use of the front-end application and the entire back-end stack. We use acceptance testing in Sentry for two purposes:

  1. Coverage passes endpoint tests only or is used onlyJestWorkflows that cannot be covered.
  2. Through our visual regressionGitHub ActionsPrepare snapshots for visual regression tests.

Acceptance tests can be found in Tests/Acceptance and run locally using PYTest.

Run acceptance test

When you run acceptance tests, WebPack will run automatically to build static assets. If you change a Javascript file while creating or modifying an acceptance test, rm.webpack.meta is required after each change to trigger a rebuild of the static resource.

Run a single acceptance test.
pytest tests/acceptance/test_organization_group_index.py \
    -k test_with_onboarding

Run a browser with a header so that you can view it.
pytest tests/acceptance/test_organization_group_index.py \
    --no-headless=true \
    -k test_with_onboarding

Open each snapshot image
SENTRY_SCREENSHOT=1 VISUAL_SNAPSHOT_ENABLE=1 \
    pytest tests/acceptance/test_organization_group_index.py \
    -k test_with_onboarding
Copy the code

If you see:

WARNING: Failed to gather log types: Message: unknown > command: Cannot call non W3C standard command while in W3C mode

saidWebpackResources are not compiled correctly.

Positioning elements

Because we are using emotion, our class name is usually not useful for browser automation. Instead, we freely use the data-test-ID attribute to define hook points for browser automation and Jest testing.

Handling asynchronous actions

All of our data is loaded asynchronously to the front end, and acceptance testing needs to account for various delays and response times. We prefer to use Selenium’s wait_until* feature to poll the DOM until the element is present or visible. We don’t use sleep().

Visual regression

Pixels are important, so we use visual regression to help catch unexpected changes in the way Sentry renders. During acceptance testing, we capture screenshots and compare the screenshots in your pull request to the approved baseline.

Although we have fairly extensive coverage of visual regression, there are still some important blind spots:

  • Hover (Hover) card and hover state
  • The modal window
  • Chart and data visualization

All of these components and interactions are typically not included in visual snapshots, and you should be careful when handling any of them.

Dealing with changing data

Because visual regression compares image snapshots, and because a large portion of our data deals with time series data, we often need to replace time-based content with ‘fixed’ data. You can use the getDynamicText helper to provide fixed content for components/data that depend on the current time or change too frequently to be included in a visual snapshot.

Jest test

Our Jest suite covers functional and unit testing for front-end components. We prefer to write functional tests that interact with components and observe the results (navigation, API calls) rather than check prop passes and state mutations. See the Frontend Handbook for more Jest testing tips.

  • Develop. Sentry. Dev/frontend / # t…
# Run jest in interactive mode
yarn test

# Run a single test
yarn test tests/js/spec/views/issueList/overview.spec.js
Copy the code

API Fixtures

Because our Jest tests are running without apis, we have various fixture builders available to help generate API response payloads. TestStubs globally includes all fixture functions in tests/js/sentry-test/fixtures/.

You should also use MockApiClient. AddMockResponse () to set up your component will be the response of the API calls. Failure to emulate the endpoint results in a test failure.

Kafka tests in CI

The Snuba test suite (.github/workflows/snuba-integration-test.yml) is the only test suite that actually lets Kafka run in CI. If you have a test that Kafka needs to run, these tests need to be nested under the Snuba Tests folder (Tests/Snuba /). If you do not, your test will time out and be cancelled in GH Actions.

More and more

  • Sentry official JavaScript SDK introduction and debugging guide
  • Sentry Monitor – Private Docker Compose deployment and troubleshooting