Fix waitFor and messagesLimit behaviour in Kafka consumer#584
Merged
yesnault merged 2 commits intoovh:masterfrom Sep 2, 2022
Merged
Fix waitFor and messagesLimit behaviour in Kafka consumer#584yesnault merged 2 commits intoovh:masterfrom
yesnault merged 2 commits intoovh:masterfrom
Conversation
Before this, it may collide with timeout set and it didn't wait for it. Signed-off-by: Enrique J. Hernández <[email protected]>
When there are no more messages to consume it may happen the Kafka consumer times out failing but it actually consume all messages Signed-off-by: Enrique J. Hernández <[email protected]>
6d6bd7f to
6244da7
Compare
philippgille
approved these changes
Sep 2, 2022
Contributor
philippgille
left a comment
There was a problem hiding this comment.
I forgot the second msg limit check in the other PR. Thanks for fixing it! LGTM
Contributor
|
I am using: @sixstone-qq @philippgille I am experiencing that my full consumer looks as follow: - type: kafka
clientType: consumer
withTLS: false
withSASL: false
withAVRO: true
kafkaVersion: "2.6.0"
markOffset: true
initialOffset: oldest
waitFor: 1
groupID: venom
addrs:
- "{{.kafkaHost}}:{{.kafkaPort}}"
schemaRegistryAddr: "{{.kafkaSchemaRegistryHost}}"
topics:
- "{{.ticketUpsertCommand}}"
assertions:
- result.err ShouldBeEmpty
- result.messages.__len__ ShouldEqual 1
- result.messagesjson.messagesjson0.value ShouldNotBeEmpty
- result.messagesjson.messagesjson0.value.description ShouldContainSubstring "Strong Winds in 2 hours" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR includes two fixes over Kafka consumer executor:
waitFor didn't work in some cases mixed with timeout property
as it is reused by test executor. Using a goroutine over
consume claim we can work on timeout and wait for without issues.
messageLimit after Improve Kafka executor #448 may only work if there are more messages to
consume in the topic, if the message limit is hit after consuming the
last message from the topic, the condition is never met and it waits
until timeout is reached leading to longer Kafka consumer step
executions.