Add parallelized restore capability to ghe-restore-storage #635
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
omgitsads
approved these changes
Sep 17, 2020
There was a problem hiding this comment.
Sorry for the delay on this. I think it looks good to me 👍 .
Have you tried this out with a large amount of data, to get an idea of the improvement? Given this is just rsync'ing data from /data/user/storage, you could generate a decent chunk of random data with dd and confirm it's doing what you expect.
|
Was originally bottlenecked by my home connection speed for effectively testing this, but wound up spinning up an EC2 instance with a 5Gbit NIC to remove that problem :) With that said, disk performance on my backup host will still likely be a bottleneck as I'm seeing transfer and disk speeds plummet the more I use my test instance. Added 6GB worth of 1GB files to random locations for the storage restore into the Time without parallelism (~9 min): Time with parallelism (3 min 40 sec): I'll note that I burned a fair bit of time on this getting I've also verified that restores to an HA environment still function (with parallelism enabled or disabled) and are otherwise unaffected by these updates 🎉 |
|
@maclarel I think you can just install |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a parallelized restore of
storagedata, with a number of rsync threads equal to the number of storage nodes. This is the same logic used forghe-restore-repositories, simply ported over toghe-restore-storage.For customers that are heavy users of LFS in a clustered environment, this can have significant performance improvements, with a reduction in run time equivalent to the number of nodes. Specifically,
rsynconly utilizes a single thread ofsshd, so when high transfer speeds are possible it is likely thatsshdwill become CPU bound resulting in limited transfer speed.For example, restoring ~5TB of data across 5 storage nodes would complete in approximately 16 hours assuming a transfer speed of 100MB/s (roughly where we see
sshdbecome CPU bound) as the restores would be run sequentially. Assuming sufficient bandwidth for transfers at 500MB/s (achievable on a 10Gbit connection) this could reduce the overall time to approximately 3 hours as all 5rsyncinvocations would be run simultaneously and would utilize 1 thread per server effectively quintupling performance.Verbose log confirms that all 3 are being kicked off at the same time, which aligns what what is seen for
ghe-restore-repositoriesbehaviour: