feat: migrate kncloudevents, broker filter, broker ingress, inmemorychannel dispatcher to otel #8635

Cali0707 · 2025-07-15T02:30:28Z

Proposed Changes

Migrate kncloudevents to OTel
Move the broker filter to OTel
Move the broker ingress to OTel
Move the InMemoryChannel Dispatcher to OTel

Release Note

The broker filter, ingress, and InMemoryChannel deployments now expose metrics and traces with OpenTelemetry instead of Zipkin/OpenCensus

Signed-off-by: Calum Murray <[email protected]>

knative-prow · 2025-07-15T02:30:30Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Cali0707 · 2025-07-15T02:31:08Z

FYI @creydr after a few tries I'm feeling good about the approach I've taken here so far.

Lmk what you think and I'll finish this up quickly tomorrow!

creydr

Thanks @Cali0707 for checking on this. For me the approach looks good so far. As I haven't done much with otel so far, maybe @evankanderson can check on the non-WIP PR later too (@evankanderson I only saw you reviewed some of the otel PRs in serving :D )

creydr · 2025-07-15T06:18:13Z

pkg/broker/filter/filter_handler.go

+	triggerInformer v1.TriggerInformer,
+	brokerInformer v1.BrokerInformer,
+	subscriptionInformer messaginginformers.SubscriptionInformer,
+	reporter StatsReporter,


do we still need the statsreporter?

I need to refactor the metrics bits, but when I do we can drop it

pkg/broker/filter/filter_handler.go

pkg/kncloudevents/event_dispatcher.go

Signed-off-by: Calum Murray <[email protected]>

Cali0707 · 2025-07-15T15:50:40Z

/cc @creydr @evankanderson

This should be ready now

Signed-off-by: Calum Murray <[email protected]>

Cali0707 · 2025-07-15T15:58:17Z

/hold

Looks like some other builds are breaking...

Signed-off-by: Calum Murray <[email protected]>

Cali0707 · 2025-07-16T03:40:29Z

/cc @creydr

I have all the unit tests passing locally now 😄

Unfortunately, this ended up making the PR quite large, as I needed to instrument the broker ingress and the IMC dispatcher as well...

Lmk what you think!

codecov · 2025-07-16T03:46:27Z

Codecov Report

Attention: Patch coverage is 60.84559% with 213 lines in your changes missing coverage. Please review.

Project coverage is 52.17%. Comparing base (6e5cb3a) to head (3da86dc).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/broker/filter/main.go	0.00%	39 Missing ⚠️
cmd/broker/ingress/main.go	0.00%	37 Missing ⚠️
pkg/broker/filter/filter_handler.go	61.05%	34 Missing and 3 partials ⚠️
pkg/observability/newcontext.go	55.95%	37 Missing ⚠️
pkg/observability/otel/handler.go	0.00%	20 Missing ⚠️
pkg/channel/fanout/fanout_event_handler.go	76.92%	6 Missing and 3 partials ⚠️
pkg/channel/event_receiver.go	72.41%	8 Missing ⚠️
...econciler/inmemorychannel/dispatcher/controller.go	76.92%	4 Missing and 2 partials ⚠️
pkg/broker/ingress/ingress_handler.go	89.36%	4 Missing and 1 partial ⚠️
pkg/broker/filter/server_manager.go	0.00%	4 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8635      +/-   ##
==========================================
- Coverage   52.22%   52.17%   -0.05%     
==========================================
  Files         402      401       -1     
  Lines       25174    25199      +25     
==========================================
+ Hits        13147    13148       +1     
- Misses      11224    11261      +37     
+ Partials      803      790      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

evankanderson

A bunch of smaller comments, but nothing to block doing the cleanup later (and I realize we're up against time boundaries on the release process).

/lgtm

evankanderson · 2025-07-16T06:49:19Z

cmd/broker/filter/main.go


+	pprof := k8sruntime.NewProfilingServer(sl.Named("pprof"))
+
+	mp, tp := otel.SetupObservabilityOrDie(ctx, "broker.filter", sl, pprof)


Out of curiousity, why don't we set the global metrics / trace provider here, rather than needing to pass them into filter.NewHandler and filter.NewServerManager?

(This doesn't need to change for this review, just wondering.)

evankanderson · 2025-07-16T06:54:25Z

pkg/broker/filter/filter_handler.go

+	ctx = observability.WithMinimalEventLabels(ctx, event)
+	ctx = observability.WithBrokerLabels(ctx, broker)
+
+	ctx, span := h.tracer.Start(ctx, tracing.TriggerMessagingDestination(triggerRef.NamespacedName))


Why both this and line 280?

evankanderson · 2025-07-16T07:24:35Z

pkg/broker/filter/filter_handler.go

 	if subscriberURI == nil {
 		// Record the event count.
 		writer.WriteHeader(http.StatusNotFound)
-		_ = h.reporter.ReportEventCount(reportArgs, http.StatusNotFound)


Why stop recording this metric?

This was discussed on the google doc, the idea is that it can be derived from the dispatch duration histogram (which implicitly has a total count)

Since we are instrumenting the handler with otel as well, there is the http.server.request.duration histogram as well, so rejected/invalid events can be found from the difference between the two

evankanderson · 2025-07-16T07:32:51Z

pkg/broker/ingress/server_manager.go


-func NewServerManager(ctx context.Context, logger *zap.Logger, cmw configmap.Watcher, httpPort, httpsPort int, handler *Handler) (*eventingtls.ServerManager, error) {
+func NewServerManager(
+	ctx context.Context, 


You have trailing whitespace here:

Suggested change

ctx context.Context,

ctx context.Context,

evankanderson · 2025-07-16T07:34:08Z

pkg/broker/ingress/server_manager.go

+	otelHandler := otelhttp.NewHandler(handler, "broker.ingress",
+		otelhttp.WithMeterProvider(meterProvider),
+		otelhttp.WithTracerProvider(traceProvider),
+		otelhttp.WithFilter(func(r *http.Request) bool {
+			return !network.IsKubeletProbe(r)
+		}),
+		otelhttp.WithPropagators(tracing.DefaultTextMapPropagator()),
+		otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
+			if r.URL.Path == "" {
+				return r.Method + " /"
+			}
+			return fmt.Sprintf("%s %s", r.Method, r.URL.Path)
+		}),
+
+	)


This seems very similar to the code in filter. It would be great to have a common library function for setting this up, unless there are differences I'm missing.

pkg/channel/event_receiver.go

pkg/inmemorychannel/event_dispatcher_test.go

Signed-off-by: Calum Murray <[email protected]>

test/conformance/broker_tracing_test.go

test/conformance/channel_tracing_test.go

Co-authored-by: Christoph Stäbler <[email protected]>

creydr

Thanks a lot @Cali0707 for working on this 💪

/lgtm

knative-prow · 2025-07-16T14:51:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Cali0707, creydr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Cali0707,creydr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Cali0707 · 2025-07-16T16:52:14Z

/test reconciler-tests

Cali0707 · 2025-07-16T18:10:00Z

/unhold

feat: migrate tracing in observability to otel

5bca04d

Signed-off-by: Calum Murray <[email protected]>

knative-prow bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 15, 2025

knative-prow bot requested review from aslom and lionelvillard July 15, 2025 02:30

knative-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2025

creydr reviewed Jul 15, 2025

View reviewed changes

Cali0707 added 4 commits July 15, 2025 11:02

feat: instrument broker filter with otel

eaf8130

Signed-off-by: Calum Murray <[email protected]>

feat: handlers are correctly wrapped with otelhttp

8ca3ae1

Signed-off-by: Calum Murray <[email protected]>

feat: broker filter main sets up observability properly

911936d

Signed-off-by: Calum Murray <[email protected]>

chore: update deps

f1de068

Signed-off-by: Calum Murray <[email protected]>

knative-prow bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 15, 2025

Cali0707 changed the title ~~[WIP] feat: migrate kncloudevents and broker filter to otel~~ feat: migrate kncloudevents and broker filter to otel Jul 15, 2025

Cali0707 requested a review from creydr July 15, 2025 15:50

Cali0707 marked this pull request as ready for review July 15, 2025 15:50

knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2025

knative-prow bot requested a review from Leo6Leo July 15, 2025 15:50

knative-prow bot requested a review from evankanderson July 15, 2025 15:50

cleanup: fix formatting

dded87b

Signed-off-by: Calum Murray <[email protected]>

knative-prow bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 15, 2025

Cali0707 added 4 commits July 15, 2025 22:39

feat: instrument broker ingress with otel

96733cd

Signed-off-by: Calum Murray <[email protected]>

fix(build): test image builds again

53fe514

Signed-off-by: Calum Murray <[email protected]>

fix: kncloudevents properly instruments metrics and traces

9cacf44

Signed-off-by: Calum Murray <[email protected]>

feat: inmemory channel dispatcher is instrumented with otel

f00a291

Signed-off-by: Calum Murray <[email protected]>

knative-prow bot added the area/test-and-release Test infrastructure, tests or release label Jul 16, 2025

evankanderson reviewed Jul 16, 2025

View reviewed changes

knative-prow bot assigned evankanderson Jul 16, 2025

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2025

Cali0707 added 2 commits July 16, 2025 09:57

fix(test): skip conformance tracing tests for now

38638d0

Signed-off-by: Calum Murray <[email protected]>

fix: set url.scheme metric attribute

351ae2c

Signed-off-by: Calum Murray <[email protected]>

knative-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2025

fix: address review comments

496dc16

Signed-off-by: Calum Murray <[email protected]>

Cali0707 changed the title ~~feat: migrate kncloudevents and broker filter to otel~~ feat: migrate kncloudevents, broker filter, broker ingress, inmemorychannel dispatcher to otel Jul 16, 2025

creydr reviewed Jul 16, 2025

View reviewed changes

test/conformance/broker_tracing_test.go Outdated Show resolved Hide resolved

creydr reviewed Jul 16, 2025

View reviewed changes

test/conformance/channel_tracing_test.go Outdated Show resolved Hide resolved

Apply suggestions from code review

3da86dc

Co-authored-by: Christoph Stäbler <[email protected]>

creydr approved these changes Jul 16, 2025

View reviewed changes

knative-prow bot assigned creydr Jul 16, 2025

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2025

knative-prow bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2025

knative-prow bot merged commit bedd48f into knative:main Jul 16, 2025
40 of 42 checks passed

This was referenced Jul 16, 2025

[OTel]: Update Broker filter to use OTel for metrics/tracing #8631

Closed

[OTel]: Update broker ingress to use OTel for metrics/tracing #8630

Closed

[OTel]: Update channel to use OTel for metrics/tracing #8629

Closed


		pprof := k8sruntime.NewProfilingServer(sl.Named("pprof"))

		mp, tp := otel.SetupObservabilityOrDie(ctx, "broker.filter", sl, pprof)

Uh oh!

feat: migrate kncloudevents, broker filter, broker ingress, inmemorychannel dispatcher to otel #8635

feat: migrate kncloudevents, broker filter, broker ingress, inmemorychannel dispatcher to otel #8635

Uh oh!

Conversation

Cali0707 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Uh oh!

knative-prow bot commented Jul 15, 2025

Uh oh!

Cali0707 commented Jul 15, 2025

Uh oh!

creydr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Cali0707 commented Jul 15, 2025

Uh oh!

Cali0707 commented Jul 15, 2025

Uh oh!

Cali0707 commented Jul 16, 2025

Uh oh!

codecov bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

evankanderson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

creydr left a comment

Choose a reason for hiding this comment

Uh oh!

knative-prow bot commented Jul 16, 2025

Uh oh!

Cali0707 commented Jul 16, 2025

Uh oh!

Cali0707 commented Jul 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Cali0707 commented Jul 15, 2025 •

edited

Loading

codecov bot commented Jul 16, 2025 •

edited

Loading