Thanks to visit codestin.com
Credit goes to github.com

Skip to content

How to make Sentry with OTel integration work in gunicorn + gevent ? #3781

Closed
@awoimbee

Description

@awoimbee

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

I want to make Sentry with gunicorn work (see benoitc/gunicorn#1855), with the additional complexity that I want the Sentry <> OTel integration.

How are you getting stuck?

Following the python OTel docs, I setup OTel in gunicorn.conf.py, in post_fork().

But I can't import sentry_sdk and sentry_sdk.init() in post_fork() because sentry uses sockets and gunicorn patches sockets with patch_all() after post_fork(), right before post_worker_init(). So I need to put sentry_sdk.init() in post_worker_init().

But I can't move my OTLPSpanExporter() to post_worker_init() because then, when the OTel endpoint is down, my API stops responding (because of Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in <many>s.).

All in all, my current gunicorn.conf.py is:

import os
from importlib.metadata import version

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

bind = "0.0.0.0:8080"
wsgi_app = "myapp.entrypoint"

worker_class = "gevent"
workers = int(os.getenv("GUNICORN_WORKERS", "2"))
worker_connections = int(os.getenv("GUNICORN_WORKER_CONNECTIONS", "1000"))
max_requests = 5000
max_requests_jitter = 1000
preload_app = False
timeout = 120
keepalive = 60
graceful_timeout = 75

errorlog = "-"


# Setup tracing
# https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html?highlight=gunicorn#working-with-fork-process-models
# https://docs.sentry.io/platforms/python/tracing/instrumentation/opentelemetry/
def post_fork(server, worker):
    resource = Resource.create({SERVICE_NAME: "myapp", "worker": worker.pid})
    tracer_provider = TracerProvider(resource=resource)

    if otlp_endpoint := os.getenv("TRACING_OTLP_ENDPOINT"):
        otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
        span_processor = BatchSpanProcessor(otlp_exporter)
        tracer_provider.add_span_processor(span_processor)

    print(f"Sentry enabled: {os.getenv('SENTRY_DSN') is not None}")  # noqa: T201
    print(f"OTel export enabled: {otlp_endpoint is not None}")  # noqa: T201
    trace.set_tracer_provider(tracer_provider)


def post_worker_init(worker):
    import sentry_sdk
    from sentry_sdk.integrations.opentelemetry import (
        SentryPropagator,
        SentrySpanProcessor,
    )

    if sentry_dsn := os.getenv("SENTRY_DSN"):
        sentry_env = os.getenv("SENTRY_ENVIRONMENT")
        release = version("myapp")
        sentry_sdk.init(
            dsn=sentry_dsn,
            environment=sentry_env,
            traces_sample_rate=1.0,
            instrumenter="otel",
            release=f"{release}@{os.environ.get('GIT_REF', '(none)')}",
        )
        tracer_provider = trace.get_tracer_provider()
        tracer_provider.add_span_processor(SentrySpanProcessor())
        set_global_textmap(SentryPropagator())

The final issue here is that I keep adding span processors without removing them...

Where in the product are you?

Issues

Link

No response

DSN

https://[email protected]/5465401

Version

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Waiting for: Product Owner

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions