Thanks to visit codestin.com
Credit goes to github.com

Skip to content

HAProxy OpenTelemetry does not update traceparent with current spanId, breaking downstream trace hierarchy #3353

@rinaldou

Description

@rinaldou

Detailed Description of the Problem

I am testing the OpenTelemetry feature using the provided fe-be example for analysis.

The spans are being successfully exported to Grafana Tempo, and the trace structure between frontend (FE) and backend (BE) appears to be correct at first glance.

However, when analyzing the Service & Operation (Span Metrics) view, I noticed that the spans are not aligned as expected. After investigating, it seems this is related to how the trace context is being propagated.

Below are the relevant spans generated by HAProxy:

HAProxy (FE) – root span (no parentSpanId, as expected):

{
"endTimeUnixNano": "1777463127268600775",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"spanId": "431fabb527b08de1",
"startTimeUnixNano": "1777463127266280454",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}

HAProxy (BE) – child span (correctly referencing FE spanId as parent):

{
"endTimeUnixNano": "1777463127268398253",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"parentSpanId": "431fabb527b08de1",
"spanId": "dcaa9092ca4438f2",
"startTimeUnixNano": "1777463127266521210",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}

From the tracing data perspective, this relationship appears correct: FE is the root span, and BE is its child.

🔎 Observed issue

The problem seems to be in trace context propagation.

For correct distributed tracing behavior (as expected by OpenTelemetry), each hop should:

Create its own span
Propagate its own span context downstream (i.e., update the traceparent header)

However, what I observe is:

HAProxy (FE) sends:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
HAProxy (BE), instead of propagating its own spanId, forwards the same traceparent:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
⚠️ Expected behavior

HAProxy (BE) should propagate its own spanId as the parent for the next hop:

traceparent: 00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01
🧠 Impact

Because of this, downstream services receive the FE span as their parent instead of the BE span.

This leads to:

Incorrect trace hierarchy (flattened structure)
Misleading service dependency graphs
Misalignment in the Service & Operation (Span Metrics) view

Expected Behavior

Actual sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01)

Desired sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01)

Steps to Reproduce the Behavior

Run the fe-be example on otel addon. Check the traceparent generated by fe and be servers.

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

My configuration is exactly the fe-be servers in the example.

Output of haproxy -vv

[root@spcdmvm9511 ~]# haproxy -vv
HAProxy version 3.4-dev9-1cdb6bf3f8859 2026/04/15 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.14.0-611.47.1.el9_7.s390x #1 SMP Tue Mar 31 06:02:39 EDT 2026 s390x
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_OTEL=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   =

Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +OTEL -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=4).
Built with SSL library version : OpenSSL 3.5.1 1 Jul 2025
Running on SSL library version : OpenSSL 3.5.1 1 Jul 2025
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /etc/pki/tls/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 1.0.2-844).
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.5.0 20240719 (Red Hat 11.5.0-11)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       qmux : mode=HTTP  side=FE|BE  mux=QMUX  flags=HTX|NO_UPG
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] comp-req
        [COMP] comp-res
        [COMP] compression
        [FCGI] fcgi-app
        [OTEL] opentelemetry
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces


Additional Information

running the 3.4-dev9 and applied patch related to the 'inject' and 'extract' bug. now it understands when '-otx-ctx' it sends the http header as traceparent instead of otx-ctx-traceparent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: feedback requiredThe developers are waiting for a reply from the reporter.type: bugThis issue describes a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions