Use synchronous queue when sink does not want to drop message to deal with back pressure #245
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… with back pressure.
|
NetflixOSS » suro » suro-pull-requests #61 FAILURE |
|
suro-pull-requests #229 FAILURE |
|
suro-pull-requests #230 SUCCESS |
|
NetflixOSS » suro » suro-pull-requests #62 SUCCESS |
There was a problem hiding this comment.
I thought you were just changing this offer to put. but that will be a big change.
you current change doesn't break any existing behavior. looks good/safe to me.
nitpicking: a if-else might be more straightforward.
|
NetflixOSS » suro » suro-pull-requests #63 SUCCESS |
|
suro-pull-requests #231 SUCCESS |
allenxwang
pushed a commit
that referenced
this pull request
May 19, 2015
Use synchronous queue when sink does not want to drop message to deal with back pressure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ElasticSearchSinkusesMessageQueue4Sinkfor message handoff between the input and the sink. ItswriteTo()method is simply implemented as enqueuing the message toMessageQueue4Sink, which will drop the message if queue is full. Therefore, the queue forElasticSearchSinkis typically configured as a large file based queue to minimize the possibility of dropping messages. This complicates some Suro operations and monitoring as it will have local storage and hence becomes stateful.In case that the input to
ElasticSearchSinkisKafkaConsumer, where there is already a file queue offered by Kafka, having another queue in the system makes things complicated and unnecessary. In this case, if sink is too busy, it should just naturally slow down the input (KafkaConsumer) which will cause lag in the consumer but without having to deal with back pressure.Therefore, a synchronous queue seems to be a good fit in this case where it can block the input if sink busy, and shift the back pressure to its upstream.
With this update, user can configure the queue to be synchronous queue by specifying the capacity of the queue to be 0.