You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I’m currently working on integrating Kafka static membership with Storm, specifically by setting the group.instance.id property in Kafka consumers.
In Kubernetes environments, it's relatively straightforward to assign a stable value (e.g., using the Pod UID) to group.instance.id. However, in Apache Storm, I haven’t found an obvious way to assign a unique and stable identifier to each consumer instance.
It seems that Storm workers or tasks don't have any persistent or externally visible ID that can be used for this purpose, which makes it challenging to use Kafka static membership effectively.
I have a few questions:
Is there any existing identifier within Storm (e.g., worker ID, task ID) that could be used as a stable group.instance.id?
If not, are there any recommended workarounds to achieve this?
Would it be feasible to consider adding a feature that allows users to inject a stable instance ID (perhaps via configuration or environment variable) for use with Kafka consumers?
Any suggestions or guidance would be appreciated. Thank you for your time and for maintaining this great project!
The text was updated successfully, but these errors were encountered:
afaik task IDs are assigned when the topology is created and remain static. Even if a task fails, restarts, or is reassigned to a different location, it will retain the same ID.
The bolt will consistently receive the same task ID as long as the overall configuration—specifically, the number of executors across the topology—doesn't change.
Hi Storm maintainers,
I’m currently working on integrating Kafka static membership with Storm, specifically by setting the
group.instance.id
property in Kafka consumers.In Kubernetes environments, it's relatively straightforward to assign a stable value (e.g., using the Pod UID) to
group.instance.id
. However, in Apache Storm, I haven’t found an obvious way to assign a unique and stable identifier to each consumer instance.It seems that Storm workers or tasks don't have any persistent or externally visible ID that can be used for this purpose, which makes it challenging to use Kafka static membership effectively.
I have a few questions:
group.instance.id
?Any suggestions or guidance would be appreciated. Thank you for your time and for maintaining this great project!
The text was updated successfully, but these errors were encountered: