Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 25 additions & 23 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Costs are close to additive (0.1 + 17.5 + 58.5 + 120.1 = 196 vs 217 measured). *
twice Sentry**, which was not the expected ordering, and structlog's instrument costs nothing
until you actually log.

### 4a. OpenTelemetry: two knobs lite-bootstrap does not expose
### 4a. OpenTelemetry: the two knobs that pay

| scenario | RPS | µs/req | gain |
|---|---:|---:|---|
Expand All @@ -94,16 +94,15 @@ until you actually log.
| `+ ParentBased(TraceIdRatioBased(0.01))` sampler | 12484 | 80.1 | −55.5 µs |
| both | 15922 | 62.8 | **2.16x** |

1. `FastAPIInstrumentor.instrument_app` accepts `exclude_spans: list[Literal["receive","send"]]`.
lite-bootstrap passes only `app`, `tracer_provider` and `excluded_urls`, so **every request
produces three spans** - the server span plus one each for the ASGI `receive` and `send`
events. Two thirds of the spans, one quarter of the cost, and almost nobody looks at them.
2. `OpenTelemetryInstrument.bootstrap()` constructs `TracerProvider(resource=resource)` with no
sampler, which means the SDK default `parentbased_always_on`. **There is no configuration
surface for a sampler anywhere in lite-bootstrap**, so a service cannot head-sample its own
traces at all; every request is recorded, serialized and shipped. A 1% ratio sampler is worth
55 µs/req here. (Sampling rate is a user decision, not a default to change - the gap is that
it cannot be expressed.)
1. `FastAPIInstrumentor.instrument_app` accepts `exclude_spans: list[Literal["receive","send"]]`,
which `FastAPIConfig.opentelemetry_exclude_spans` passes through. It is empty by default, so
**every request still produces three spans** - the server span plus one each for the ASGI
`receive` and `send` events. Two thirds of the spans, one quarter of the cost, and almost
nobody looks at them. Set it to `["receive", "send"]` to drop the two event spans.
2. `OpenTelemetryConfig.opentelemetry_sampler` is passed to the `TracerProvider`. Left unset, the
SDK default `parentbased_always_on` applies and every request is recorded, serialized and
shipped. A 1% ratio sampler is worth 55 µs/req here. The default stays always-on deliberately:
sampling rate is a user decision, not something to pick on a service's behalf.

### 4b. Sentry: the cost is one thing, and it is not the one people tune

Expand Down Expand Up @@ -188,25 +187,26 @@ The last row is the interesting one: replacing `Scope.continue_trace` with a ver
`generate_propagation_context(headers)` and returns no Transaction loses **nothing** on the error
event and still saves ~30 µs/req. That is a pure upstream bug, not a trade-off.

Similarly, `exclude_spans=["receive","send"]` costs you the ASGI event spans and nothing else, and
`sentry_logs_level=None` costs nothing at all while Sentry Logs is disabled.
Similarly, `opentelemetry_exclude_spans=["receive","send"]` costs you the ASGI event spans and
nothing else, and `sentry_logs_level=None` costs nothing at all while Sentry Logs is disabled -
which is why lite-bootstrap now applies it by default.

## 6. The tuned configuration

What "tuned" means in §3, all reachable through today's public API except the two OTel knobs:
What "tuned" means in §3, all reachable through today's public API:

```python
FastAPIConfig(
# Sentry: OTel owns distributed tracing, Sentry is an error sink
sentry_integrations=[
StarletteIntegration(http_methods_to_capture=()),
FastApiIntegration(http_methods_to_capture=()),
LoggingIntegration(level=None, sentry_logs_level=None),
],
sentry_additional_params={"auto_session_tracking": False},
# OpenTelemetry: not expressible today, see issues
# exclude_spans=["receive", "send"] on FastAPIInstrumentor.instrument_app
# sampler=ParentBased(TraceIdRatioBased(0.01)) on TracerProvider
sentry_logging_breadcrumb_level=None,
sentry_auto_session_tracking=False,
# OpenTelemetry: sampling is the single biggest saving here
opentelemetry_exclude_spans=["receive", "send"],
opentelemetry_sampler=ParentBased(TraceIdRatioBased(0.01)),
)
```

Expand All @@ -215,14 +215,16 @@ Sentry-side trace correlation, 99% of OTel traces, ASGI event spans.

## 7. Filed issues

lite-bootstrap (all "possible improvement", nothing implemented):
lite-bootstrap (all "possible improvement"):

- [#184](https://github.com/modern-python/lite-bootstrap/issues/184) OpenTelemetry sampler is not
configurable (55 µs/req)
configurable (55 µs/req) - **implemented** as `opentelemetry_sampler`
- [#185](https://github.com/modern-python/lite-bootstrap/issues/185) `exclude_spans` is never passed
to `FastAPIInstrumentor` (33 µs/req)
to `FastAPIInstrumentor` (33 µs/req) - **implemented** as `opentelemetry_exclude_spans`
- [#186](https://github.com/modern-python/lite-bootstrap/issues/186) Sentry `sentry_logs_level`,
breadcrumb level and `auto_session_tracking` are not exposed (~9 µs/req plus ~2 µs/log record)
breadcrumb level and `auto_session_tracking` are not exposed (~9 µs/req plus ~2 µs/log record) -
**implemented**: `sentry_logs_level=None` is now the default, and the other two are
`sentry_logging_breadcrumb_level` and `sentry_auto_session_tracking`
- [#187](https://github.com/modern-python/lite-bootstrap/issues/187) Document what the stack costs

sentry-python:
Expand Down
24 changes: 23 additions & 1 deletion docs/introduction/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,36 @@ Additional parameters can also be supplied through the settings object:
- `sentry_max_breadcrumbs` - the total amount of breadcrumbs
- `sentry_max_value_length` - the max event payload length
- `sentry_attach_stacktrace` - if True, stack traces are automatically attached to all messages logged
- `sentry_auto_session_tracking` - whether every request opens and closes a Sentry release-health `Session` (default: `True`), measured at ~7 µs per request. Set it to `False` if you do not use Sentry release health.
- `sentry_integrations` - list of integrations to enable
- `sentry_logging_breadcrumb_level` - the minimum standard-library log level recorded as a breadcrumb (default: `logging.INFO`). Passed as `LoggingIntegration(level=...)`; see below.
- `sentry_tags` - key/value string pairs that are both indexed and searchable
- `sentry_additional_params` - additional params, which will be passed to `sentry_sdk.init`
- `sentry_additional_params` - additional params, which will be passed to `sentry_sdk.init`, overriding any of the above that map to the same keyword
- `sentry_default_integrations` - whether to use sentry's default integrations (default: `True`)
- `sentry_before_send` - optional callback chained after the built-in structlog enricher, passed to `sentry_sdk.init(before_send=...)`

Read more about sentry_sdk params [here](https://docs.sentry.io/platforms/python/configuration/options/).

### Sentry logging integration

Unless `sentry_integrations` already contains a `LoggingIntegration`, lite-bootstrap appends one built
as `LoggingIntegration(level=sentry_logging_breadcrumb_level, sentry_logs_level=None)`. Its event
handler keeps the sentry-sdk default (`ERROR`), so the only departure from sentry-sdk's own default
integration is `sentry_logs_level`.

Turning `sentry_logs_level` off is free. lite-bootstrap never sets `enable_logs`, so Sentry Logs is
off, but `SentryLogsHandler.emit` formats the record *before* it checks whether logs are enabled
([getsentry/sentry-python#7402](https://github.com/getsentry/sentry-python/issues/7402)) - it formats
every `INFO`+ record and discards the result. Dropping breadcrumbs as well, with
`sentry_logging_breadcrumb_level=None`, saves more but costs you log breadcrumbs on error events, so
it stays on by default. Both are measured in
[the benchmarks](https://github.com/modern-python/lite-bootstrap/blob/main/benchmarks/README.md#4c-logging-cost-per-record-not-per-request).

Two ways to opt out of the appended integration: supply your own `LoggingIntegration` in
`sentry_integrations`, which lite-bootstrap leaves untouched, or set
`sentry_default_integrations=False`, which suppresses it along with every other default integration.
Under either, `sentry_logging_breadcrumb_level` is ignored and lite-bootstrap warns.


## Prometheus

Expand Down
52 changes: 39 additions & 13 deletions lite_bootstrap/instruments/sentry_instrument.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
import dataclasses
import logging
import typing

from lite_bootstrap import import_checker
from lite_bootstrap.helpers.warn import warn_at_caller
from lite_bootstrap.instruments.base import BaseConfig, BaseInstrument
from lite_bootstrap.instruments.logging_factory import STRUCTLOG_META_KEYS, StructuredLogPayload

Expand All @@ -13,6 +15,7 @@

if import_checker.is_sentry_installed:
import sentry_sdk
from sentry_sdk.integrations.logging import LoggingIntegration


# Back-compat alias: this vocabulary moved to logging_factory and was renamed
Expand All @@ -28,7 +31,9 @@ class SentryConfig(BaseConfig):
sentry_max_breadcrumbs: int = 15
sentry_max_value_length: int = 16384
sentry_attach_stacktrace: bool = True
sentry_auto_session_tracking: bool = True
sentry_integrations: list["Integration"] = dataclasses.field(default_factory=list)
sentry_logging_breadcrumb_level: int | None = logging.INFO
sentry_additional_params: dict[str, typing.Any] = dataclasses.field(default_factory=dict)
sentry_tags: dict[str, str] | None = None
sentry_default_integrations: bool = True
Expand Down Expand Up @@ -93,21 +98,42 @@ def is_configured(cls, bootstrap_config: "SentryConfig") -> bool:
def dependencies_installed() -> bool:
return import_checker.is_sentry_installed

def _warn_breadcrumb_level_ignored(self, reason: str) -> None:
if self.bootstrap_config.sentry_logging_breadcrumb_level != logging.INFO:
warn_at_caller(f"sentry_logging_breadcrumb_level is ignored, {reason}")

def _build_integrations(self) -> list["Integration"]:
config = self.bootstrap_config
if any(integration.identifier == LoggingIntegration.identifier for integration in config.sentry_integrations):
self._warn_breadcrumb_level_ignored("sentry_integrations already supplies a LoggingIntegration")
return config.sentry_integrations
if not config.sentry_default_integrations:
self._warn_breadcrumb_level_ignored("sentry_default_integrations is False")
return config.sentry_integrations
return [
*config.sentry_integrations,
LoggingIntegration(level=config.sentry_logging_breadcrumb_level, sentry_logs_level=None),
]

def bootstrap(self) -> None:
config = self.bootstrap_config
sentry_sdk.init(
dsn=config.sentry_dsn,
sample_rate=config.sentry_sample_rate,
traces_sample_rate=config.sentry_traces_sample_rate,
environment=config.service_environment,
max_breadcrumbs=config.sentry_max_breadcrumbs,
max_value_length=config.sentry_max_value_length,
attach_stacktrace=config.sentry_attach_stacktrace,
integrations=config.sentry_integrations,
default_integrations=config.sentry_default_integrations,
before_send=wrap_before_send_callbacks(enrich_sentry_event_from_structlog_log, config.sentry_before_send),
**config.sentry_additional_params,
)
init_params: dict[str, typing.Any] = {
"dsn": config.sentry_dsn,
"sample_rate": config.sentry_sample_rate,
"traces_sample_rate": config.sentry_traces_sample_rate,
"environment": config.service_environment,
"max_breadcrumbs": config.sentry_max_breadcrumbs,
"max_value_length": config.sentry_max_value_length,
"attach_stacktrace": config.sentry_attach_stacktrace,
"auto_session_tracking": config.sentry_auto_session_tracking,
"integrations": self._build_integrations(),
"default_integrations": config.sentry_default_integrations,
"before_send": wrap_before_send_callbacks(
enrich_sentry_event_from_structlog_log, config.sentry_before_send
),
}
init_params.update(config.sentry_additional_params)
sentry_sdk.init(**init_params)
tags: dict[str, str] = config.sentry_tags or {}
sentry_sdk.set_tags(tags)

Expand Down
126 changes: 126 additions & 0 deletions tests/instruments/test_sentry_instrument.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
import copy
import dataclasses
import logging
import typing
from unittest.mock import patch

import pytest
import sentry_sdk
import structlog
from sentry_sdk.integrations.logging import LoggingIntegration

from lite_bootstrap.instruments.logging_instrument import LoggingConfig, LoggingInstrument
from tests.conftest import LoggingMock, SentryTestTransport
Expand Down Expand Up @@ -147,3 +149,127 @@ def test_sentry_teardown_runs_init_when_flush_raises(minimal_sentry_config: Sent

# init() still ran (in the finally), so SDK is now disabled.
assert sentry_sdk.get_client().dsn is None


def installed_logging_integration() -> LoggingIntegration:
integration = sentry_sdk.get_client().integrations[LoggingIntegration.identifier]
assert isinstance(integration, LoggingIntegration)
return integration


def test_sentry_bootstrap_disables_the_sentry_logs_handler(minimal_sentry_config: SentryConfig) -> None:
instrument = SentryInstrument(bootstrap_config=minimal_sentry_config)
instrument.bootstrap()

try:
integration = installed_logging_integration()
assert integration._sentry_logs_handler is None # noqa: SLF001
assert integration._breadcrumb_handler is not None # noqa: SLF001
assert integration._handler is not None # noqa: SLF001
assert minimal_sentry_config.sentry_integrations == []
finally:
instrument.teardown()


def test_sentry_bootstrap_keeps_a_user_supplied_logging_integration(minimal_sentry_config: SentryConfig) -> None:
supplied = LoggingIntegration(sentry_logs_level=logging.INFO)
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_integrations=[supplied])
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
assert sentry_sdk.get_client().integrations[LoggingIntegration.identifier] is supplied
finally:
instrument.teardown()


def test_sentry_bootstrap_adds_no_logging_integration_without_default_integrations(
minimal_sentry_config: SentryConfig,
) -> None:
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_default_integrations=False)
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
assert LoggingIntegration.identifier not in sentry_sdk.get_client().integrations
finally:
instrument.teardown()


@pytest.mark.parametrize("breadcrumb_level", [logging.INFO, None], ids=["info", "disabled"])
def test_sentry_logging_breadcrumb_level_controls_the_breadcrumb_handler(
minimal_sentry_config: SentryConfig, breadcrumb_level: int | None
) -> None:
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_logging_breadcrumb_level=breadcrumb_level)
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
integration = installed_logging_integration()
assert (integration._breadcrumb_handler is None) is (breadcrumb_level is None) # noqa: SLF001
finally:
instrument.teardown()


@pytest.mark.parametrize("auto_session_tracking", [True, False], ids=["on", "off"])
def test_sentry_auto_session_tracking_reaches_the_client(
minimal_sentry_config: SentryConfig, auto_session_tracking: bool
) -> None:
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_auto_session_tracking=auto_session_tracking)
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
assert sentry_sdk.get_client().options["auto_session_tracking"] is auto_session_tracking
finally:
instrument.teardown()


def test_sentry_additional_params_override_the_explicit_init_params(minimal_sentry_config: SentryConfig) -> None:
bootstrap_config = dataclasses.replace(
minimal_sentry_config,
sentry_auto_session_tracking=True,
sentry_additional_params={
**minimal_sentry_config.sentry_additional_params,
"auto_session_tracking": False,
},
)
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
assert sentry_sdk.get_client().options["auto_session_tracking"] is False
finally:
instrument.teardown()


@pytest.mark.parametrize(
("overrides", "expected_reason"),
[
({"sentry_integrations": [LoggingIntegration()]}, "already supplies a LoggingIntegration"),
({"sentry_default_integrations": False}, "sentry_default_integrations is False"),
],
ids=["user_integration", "no_default_integrations"],
)
def test_sentry_warns_when_the_breadcrumb_level_is_ignored(
minimal_sentry_config: SentryConfig, overrides: dict[str, typing.Any], expected_reason: str
) -> None:
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_logging_breadcrumb_level=None, **overrides)
instrument = SentryInstrument(bootstrap_config=bootstrap_config)

with pytest.warns(UserWarning, match=expected_reason):
instrument.bootstrap()
instrument.teardown()


def test_sentry_does_not_warn_when_the_breadcrumb_level_is_left_at_its_default(
minimal_sentry_config: SentryConfig, recwarn: pytest.WarningsRecorder
) -> None:
bootstrap_config = dataclasses.replace(minimal_sentry_config, sentry_integrations=[LoggingIntegration()])
instrument = SentryInstrument(bootstrap_config=bootstrap_config)
instrument.bootstrap()

try:
assert [one for one in recwarn if "sentry_logging_breadcrumb_level" in str(one.message)] == []
finally:
instrument.teardown()
Loading