Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Prevent loss of events under stress-load or sync-blockers #732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nolar merged 1 commit into from
Jul 4, 2021
Merged

Prevent loss of events under stress-load or sync-blockers #732

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions kopf/_core/reactor/queueing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,16 +282,27 @@ async def worker(
try:
while not shouldstop:

# Try ASAP, but give it few seconds for the new events to arrive, maybe.
# If the queue is empty for some time, then indeed finish the object's worker.
# If the queue is filled, use the latest event only (within the short timeframe).
# Try ASAP, but give it a few seconds for the new events to arrive.
# If the queue is empty for some time, then finish the object's worker.
# If the queue is filled, use the latest event only (within a short time window).
# If an EOS marker is received, handle the last real event, then finish the worker ASAP.
try:
raw_event = await asyncio.wait_for(
backlog.get(),
timeout=settings.batching.idle_timeout)
except asyncio.TimeoutError:
break
# A tricky part! Under high-load or with synchronous blocks of asyncio event-loop,
# it is possible that the timeout happens while the queue is filled: depending on
# the order in which the coros/waiters are checked once control returns to asyncio.
# As a work-around, we double-check the queue and exit only if it is truly empty;
# if not, run as normally. IMPORTANT: There MUST be NO async/await-code between
# "break" and "finally", so that the queue is not populated again.
# TODO: LATER: Test the described scenario. I have found no ways to simulate
# a timeout while the queue is filled -- neither with pure Python nor with mocks.
if backlog.empty():
break
else:
continue
else:
try:
while True:
Expand Down