Description
Environment
SaaS (https://sentry.io/)
What are you trying to accomplish?
I want to make Sentry with gunicorn work (see benoitc/gunicorn#1855), with the additional complexity that I want the Sentry <> OTel integration.
How are you getting stuck?
Following the python OTel docs, I setup OTel in gunicorn.conf.py
, in post_fork()
.
But I can't import sentry_sdk
and sentry_sdk.init()
in post_fork()
because sentry uses sockets and gunicorn patches sockets with patch_all()
after post_fork()
, right before post_worker_init()
. So I need to put sentry_sdk.init()
in post_worker_init()
.
But I can't move my OTLPSpanExporter()
to post_worker_init()
because then, when the OTel endpoint is down, my API stops responding (because of Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in <many>s.
).
All in all, my current gunicorn.conf.py
is:
import os
from importlib.metadata import version
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
bind = "0.0.0.0:8080"
wsgi_app = "myapp.entrypoint"
worker_class = "gevent"
workers = int(os.getenv("GUNICORN_WORKERS", "2"))
worker_connections = int(os.getenv("GUNICORN_WORKER_CONNECTIONS", "1000"))
max_requests = 5000
max_requests_jitter = 1000
preload_app = False
timeout = 120
keepalive = 60
graceful_timeout = 75
errorlog = "-"
# Setup tracing
# https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html?highlight=gunicorn#working-with-fork-process-models
# https://docs.sentry.io/platforms/python/tracing/instrumentation/opentelemetry/
def post_fork(server, worker):
resource = Resource.create({SERVICE_NAME: "myapp", "worker": worker.pid})
tracer_provider = TracerProvider(resource=resource)
if otlp_endpoint := os.getenv("TRACING_OTLP_ENDPOINT"):
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
span_processor = BatchSpanProcessor(otlp_exporter)
tracer_provider.add_span_processor(span_processor)
print(f"Sentry enabled: {os.getenv('SENTRY_DSN') is not None}") # noqa: T201
print(f"OTel export enabled: {otlp_endpoint is not None}") # noqa: T201
trace.set_tracer_provider(tracer_provider)
def post_worker_init(worker):
import sentry_sdk
from sentry_sdk.integrations.opentelemetry import (
SentryPropagator,
SentrySpanProcessor,
)
if sentry_dsn := os.getenv("SENTRY_DSN"):
sentry_env = os.getenv("SENTRY_ENVIRONMENT")
release = version("myapp")
sentry_sdk.init(
dsn=sentry_dsn,
environment=sentry_env,
traces_sample_rate=1.0,
instrumenter="otel",
release=f"{release}@{os.environ.get('GIT_REF', '(none)')}",
)
tracer_provider = trace.get_tracer_provider()
tracer_provider.add_span_processor(SentrySpanProcessor())
set_global_textmap(SentryPropagator())
The final issue here is that I keep adding span processors without removing them...
Where in the product are you?
Issues
Link
No response
DSN
https://[email protected]/5465401
Version
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status