Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support for configuring group.instance.id in Kafka consumers used by Storm #8046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mwkang opened this issue Apr 23, 2025 · 2 comments
Open

Comments

@mwkang
Copy link

mwkang commented Apr 23, 2025

Hi Storm maintainers,

I’m currently working on integrating Kafka static membership with Storm, specifically by setting the group.instance.id property in Kafka consumers.

In Kubernetes environments, it's relatively straightforward to assign a stable value (e.g., using the Pod UID) to group.instance.id. However, in Apache Storm, I haven’t found an obvious way to assign a unique and stable identifier to each consumer instance.

It seems that Storm workers or tasks don't have any persistent or externally visible ID that can be used for this purpose, which makes it challenging to use Kafka static membership effectively.

I have a few questions:

  • Is there any existing identifier within Storm (e.g., worker ID, task ID) that could be used as a stable group.instance.id?
  • If not, are there any recommended workarounds to achieve this?
  • Would it be feasible to consider adding a feature that allows users to inject a stable instance ID (perhaps via configuration or environment variable) for use with Kafka consumers?

Any suggestions or guidance would be appreciated. Thank you for your time and for maintaining this great project!

@rzo1
Copy link
Contributor

rzo1 commented Apr 25, 2025

Hi,

afaik task IDs are assigned when the topology is created and remain static. Even if a task fails, restarts, or is reassigned to a different location, it will retain the same ID.

The bolt will consistently receive the same task ID as long as the overall configuration—specifically, the number of executors across the topology—doesn't change.

@mwkang
Copy link
Author

mwkang commented Apr 26, 2025

Thank you so much for the clarification!
Knowing that task IDs remain stable is exactly the information I needed.
Really appreciate your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants