Detailed Description of the Problem
I am testing the OpenTelemetry feature using the provided fe-be example for analysis.
The spans are being successfully exported to Grafana Tempo, and the trace structure between frontend (FE) and backend (BE) appears to be correct at first glance.
However, when analyzing the Service & Operation (Span Metrics) view, I noticed that the spans are not aligned as expected. After investigating, it seems this is related to how the trace context is being propagated.
Below are the relevant spans generated by HAProxy:
HAProxy (FE) – root span (no parentSpanId, as expected):
{
"endTimeUnixNano": "1777463127268600775",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"spanId": "431fabb527b08de1",
"startTimeUnixNano": "1777463127266280454",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}
HAProxy (BE) – child span (correctly referencing FE spanId as parent):
{
"endTimeUnixNano": "1777463127268398253",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"parentSpanId": "431fabb527b08de1",
"spanId": "dcaa9092ca4438f2",
"startTimeUnixNano": "1777463127266521210",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}
From the tracing data perspective, this relationship appears correct: FE is the root span, and BE is its child.
🔎 Observed issue
The problem seems to be in trace context propagation.
For correct distributed tracing behavior (as expected by OpenTelemetry), each hop should:
Create its own span
Propagate its own span context downstream (i.e., update the traceparent header)
However, what I observe is:
HAProxy (FE) sends:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
HAProxy (BE), instead of propagating its own spanId, forwards the same traceparent:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
⚠️ Expected behavior
HAProxy (BE) should propagate its own spanId as the parent for the next hop:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01
🧠 Impact
Because of this, downstream services receive the FE span as their parent instead of the BE span.
This leads to:
Incorrect trace hierarchy (flattened structure)
Misleading service dependency graphs
Misalignment in the Service & Operation (Span Metrics) view
Expected Behavior
Actual sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01)
Desired sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01)
Steps to Reproduce the Behavior
Run the fe-be example on otel addon. Check the traceparent generated by fe and be servers.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
My configuration is exactly the fe-be servers in the example.
Output of haproxy -vv
[root@spcdmvm9511 ~]# haproxy -vv
HAProxy version 3.4-dev9-1cdb6bf3f8859 2026/04/15 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.14.0-611.47.1.el9_7.s390x #1 SMP Tue Mar 31 06:02:39 EDT 2026 s390x
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -fwrapv -fvect-cost-model=very-cheap
OPTIONS = USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_OTEL=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +OTEL -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=4).
Built with SSL library version : OpenSSL 3.5.1 1 Jul 2025
Running on SSL library version : OpenSSL 3.5.1 1 Jul 2025
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /etc/pki/tls/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 1.0.2-844).
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.5.0 20240719 (Red Hat 11.5.0-11)
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
qmux : mode=HTTP side=FE|BE mux=QMUX flags=HTX|NO_UPG
quic : mode=HTTP side=FE|BE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] comp-req
[COMP] comp-res
[COMP] compression
[FCGI] fcgi-app
[OTEL] opentelemetry
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
Additional Information
running the 3.4-dev9 and applied patch related to the 'inject' and 'extract' bug. now it understands when '-otx-ctx' it sends the http header as traceparent instead of otx-ctx-traceparent.
Detailed Description of the Problem
I am testing the OpenTelemetry feature using the provided fe-be example for analysis.
The spans are being successfully exported to Grafana Tempo, and the trace structure between frontend (FE) and backend (BE) appears to be correct at first glance.
However, when analyzing the Service & Operation (Span Metrics) view, I noticed that the spans are not aligned as expected. After investigating, it seems this is related to how the trace context is being propagated.
Below are the relevant spans generated by HAProxy:
HAProxy (FE) – root span (no parentSpanId, as expected):
{
"endTimeUnixNano": "1777463127268600775",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"spanId": "431fabb527b08de1",
"startTimeUnixNano": "1777463127266280454",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}
HAProxy (BE) – child span (correctly referencing FE spanId as parent):
{
"endTimeUnixNano": "1777463127268398253",
"flags": 1,
"kind": 2,
"name": "HAProxy session",
"parentSpanId": "431fabb527b08de1",
"spanId": "dcaa9092ca4438f2",
"startTimeUnixNano": "1777463127266521210",
"traceId": "08e099d11e4339feed6cf4843fa66c51"
}
From the tracing data perspective, this relationship appears correct: FE is the root span, and BE is its child.
🔎 Observed issue
The problem seems to be in trace context propagation.
For correct distributed tracing behavior (as expected by OpenTelemetry), each hop should:
Create its own span
Propagate its own span context downstream (i.e., update the traceparent header)
However, what I observe is:
HAProxy (FE) sends:
⚠️ Expected behavior
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
HAProxy (BE), instead of propagating its own spanId, forwards the same traceparent:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01
HAProxy (BE) should propagate its own spanId as the parent for the next hop:
traceparent: 00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01
🧠 Impact
Because of this, downstream services receive the FE span as their parent instead of the BE span.
This leads to:
Incorrect trace hierarchy (flattened structure)
Misleading service dependency graphs
Misalignment in the Service & Operation (Span Metrics) view
Expected Behavior
Actual sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01)
Desired sequence:
haproxy(fe) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-431fabb527b08de1-01) -> haproxy(be) -> traceparent(00-08e099d11e4339feed6cf4843fa66c51-dcaa9092ca4438f2-01)
Steps to Reproduce the Behavior
Run the fe-be example on otel addon. Check the traceparent generated by fe and be servers.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
Output of
haproxy -vvLast Outputs and Backtraces
Additional Information
running the 3.4-dev9 and applied patch related to the 'inject' and 'extract' bug. now it understands when '-otx-ctx' it sends the http header as traceparent instead of otx-ctx-traceparent.