Add blocking flow controls to the PubSub Sink connector#282
Conversation
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
|
@googlebot I signed it! |
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
|
@googlebot I signed it! |
dc4bf28 to
4525038
Compare
| .setDelayThreshold(Duration.ofMillis(maxDelayThresholdMs)) | ||
| .setElementCountThreshold(maxBufferSize) | ||
| .setRequestByteThreshold(maxBufferBytes) | ||
| .setFlowControlSettings(FlowControlSettings.newBuilder() |
There was a problem hiding this comment.
Can we only populate these if the maxOutstandingRequestBytes and maxOutstandingMessages is less than the default otherwise we'll unnecessarily slow down publishing by acquiring/releasing flow controller locks https://github.com/googleapis/java-pubsub/blob/0bba01661c01ca540625f63a0d7e89d43e2dab34/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L247-L257.
samarthsingal
left a comment
There was a problem hiding this comment.
Thanks for contributing this change! We were in the process of doing this ourselves.
…tBytes with blocking flow controls
ca0a90b to
390e961
Compare
|
Made the requested changes and did a squash rebase |
samarthsingal
left a comment
There was a problem hiding this comment.
Looks great, thanks!
Allowing the flow controls to limit the max outstanding bytes and elements. During large traffic spikes, we have observed Connect playing messages into the PubSub producer client as fast as possible resulting in several GB of off heap usage causing OOM errors. Configuring the connector to block on maxOutstandingRequestBytes should prevent this and stop polling until space is available.
We've tested this both with the default and with a specified low maxOutstandingRequestBytes in our loadtest environment and observed it working as expected.
I've set the default values to Long. MAX_VALUE to effectively preserve the existing behavior for anyone that may not want this change, but I'm open to setting these to a reasonable value if we have some consensus on what that is.