-
Notifications
You must be signed in to change notification settings - Fork 1.9k
fix(instrument): Buffer counter underflowed (#23872) #23973
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
fix(instrument): Buffer counter underflowed (#23872) #23973
Conversation
|
Hi @sialais, thank you for the PR! Please add a changelog explaining the change. |
fcc55dd to
b222f23
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the contribution, @sialais. It would be great to nail down this buffer counter underflow issue, as I have seen it to. That said, I am a little uncertain about some of the details in this change.
| } | ||
| } | ||
| WhenFull::Overflow => { | ||
| if let Some(item) = self.base.try_send(item).await? { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not totally familiar with the implementation here, so pardon if this is a naive question. The other two sends call increment_received_event_count_and_byte_size before sending, this one never calls it. Is that an omission?
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for that, My previous commit did indeed contain an error. I've never used Overflow before, and after my build is tested, I've forgot the last case of WhenFull::Overflow, apologize.
Because resolving this issue requires updating the counter before sending the event to the buffer, when handling the Overflow branch—if the overflow logic is triggered and the message is sent to overflow buffer—the counter needs to be decremented. However, neither using send nor drop to perform this countdown is quite appropriate. Therefore, to address the problem I was facing, I chose to ignore this particular scenario for temporarily.
Now that I think about it again, maybe using drop to update the counter is actually reasonable—since the event is effectively "dropped" into the overflow buffer?
I've updated my commit, but not tested.
b222f23 to
34da1d1
Compare
34da1d1 to
dedcf6a
Compare
|
Hi @sialais, is this ready for another review? Thanks! |
Yes, I've put my build into production, and it works fine until now. |
Summary
Fix issue #23872
How did you test this PR?
Because the issue is hard to reproduce, (I can only reproduce it in my production environment, with huge data), I put my build on my production environment, and the issue fixed.
This bug will make the self monitoring metrics incorrect: when buffer counter undeflowed, metrics will be reset.

Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References