Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

mkuratczyk
Copy link
Contributor

#13959 on top of v4.1.x branch for testing purposes

@mkuratczyk
Copy link
Contributor Author

Here are the results (after pushing the Prioritise delete calls commit):
https://grafana.lionhead.rabbitmq.com/goto/rkYSf0BHg?orgId=1

Observations:

  • disk spaced is freed much faster
  • much lower memory usage, which was unexpected
  • in most cases, similar throughput

Slower compactions/scanning remains a problem, although with prioritised deletion requests, less so. Without this change, sometimes the node would be busy compacting files, minutes after queues were deleted and therefore, the files could have been deleted altogether. Now, such files will be deleted and never compacted.

Some highlights:

  1. Disk space is reclaimed much faster (v4.1.x env, green, required 12 additional minutes to reclaim the space):
Screenshot 2025-05-29 at 13 18 01
  1. Much lower memory usage throughout the tests
Screenshot 2025-05-29 at 13 19 27

@gomoripeti
Copy link
Contributor

thanks a lot @mkuratczyk, for the detailed tests and results, they are really reassuring to read

@mkuratczyk mkuratczyk force-pushed the v41x-msg-store-index-delete-on-remove branch from 85d07a7 to a40eeeb Compare May 30, 2025 08:23
It was expensive to delete files because we had clean up
the index and to get the messages in the file we have to
scan it.

Instead of cleaning up the index on file delete this
commit deletes from the index as soon as possible.
There are two scenarios: messages that are removed
from the current write file, and messages that are
removed from other files. In the latter case, we
can just delete the index entry on remove. For messages
in the current write file, we want to keep the entry
in case fanout is used, because we don't want to write
the fanout message multiple times if we can avoid it.
So we keep track of removes in the current write file
and do a cleanup of these entries on file roll over.

Compared to the previous implementation we will no
longer increase the ref_count of messages that are
not in the current write file, meaning we may do more
writes in fanout scenarios. But at the same time the
file delete operation is much cheaper.

Additionally, we prioritise delete calls in rabbit_msg_store_gc.
Without that change, if the compaction was lagging behind,
we could have file deletion requests queued behind many compaction
requests, leading to many unnecessary compactions of files
that could already be deleted.
@mkuratczyk mkuratczyk force-pushed the v41x-msg-store-index-delete-on-remove branch from a40eeeb to ee0c6a6 Compare May 30, 2025 11:14
@mkuratczyk
Copy link
Contributor Author

Closing as this was just to test 4.1 with the fix. The change is in #13959

@mkuratczyk mkuratczyk closed this May 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants