-
Notifications
You must be signed in to change notification settings - Fork 145
Description
What are you really trying to do?
We expect consistent worker performance, as we have a roughly constant load of workflows being processed.
Describe the bug
We've been running our worker with this patch applied #1834 using the 1.13.1 SDK.
OTEL interceptors are in use, as well as some custom interceptors. All AsyncLocalStorage instances created during workflow execution are being disposed by an Internals interceptor.
When we use the patched 1.13.1 SDK we see steady metrics. Updating to 1.13.2 shows a significant degradation in performance.
On the screenshot you can see on the left the workers CPU usage when running the patched 1.13.2 SDK, on the right reverting to the patched 1.13.1 SDK.
It's worth noting that the projects dependency tree has no duplicated temporal packages (eg: only a single version of each is in use), and there were no changes to other dependencies (eg: OTEL core packages) between the two builds compared.
I've scanned through the PR's linked in the release notes for 1.13.2, and whilst there are notable changes around the new plugin functionality, nothing screamed out as obviously responsible for this regression. It's quite possible I've missed something though.
Minimal Reproduction
TBD
Environment/Versions
- OS and processor: Linux
- Temporal Version: 1.25.2
- Self hosted in kubernetes
Additional context
Semi-related: #1859