Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Bug] Performance regression from 1.13.1 -> 1.13.2 #1860

@mnahkies

Description

@mnahkies

What are you really trying to do?

We expect consistent worker performance, as we have a roughly constant load of workflows being processed.

Describe the bug

We've been running our worker with this patch applied #1834 using the 1.13.1 SDK.

OTEL interceptors are in use, as well as some custom interceptors. All AsyncLocalStorage instances created during workflow execution are being disposed by an Internals interceptor.

When we use the patched 1.13.1 SDK we see steady metrics. Updating to 1.13.2 shows a significant degradation in performance.

On the screenshot you can see on the left the workers CPU usage when running the patched 1.13.2 SDK, on the right reverting to the patched 1.13.1 SDK.

Image

It's worth noting that the projects dependency tree has no duplicated temporal packages (eg: only a single version of each is in use), and there were no changes to other dependencies (eg: OTEL core packages) between the two builds compared.

I've scanned through the PR's linked in the release notes for 1.13.2, and whilst there are notable changes around the new plugin functionality, nothing screamed out as obviously responsible for this regression. It's quite possible I've missed something though.

Minimal Reproduction

TBD

Environment/Versions

  • OS and processor: Linux
  • Temporal Version: 1.25.2
  • Self hosted in kubernetes

Additional context

Semi-related: #1859

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions