Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Beta: Execute ghe-backup tasks in parallel #597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Apr 7, 2020
Merged

Beta: Execute ghe-backup tasks in parallel #597

merged 15 commits into from
Apr 7, 2020

Conversation

ryansimmen
Copy link
Member

@ryansimmen ryansimmen commented Mar 31, 2020 •
edited
Loading

Execute long running backup tasks in parallel by leveraging moreutils parallel.

This resolves https://github.com/github/ghes-infrastructure/issues/386

snh
snh suggested changes Mar 31, 2020
View reviewed changes
@ryansimmen ryansimmen requested a review from dbussink April 1, 2020 00:23
jianghao0718
jianghao0718 reviewed Apr 1, 2020
View reviewed changes
jianghao0718
jianghao0718 reviewed Apr 1, 2020
View reviewed changes
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch 6 times, most recently from 1aa484c to 80f7862 Compare April 1, 2020 02:30
@lildude
Copy link
Member

lildude commented Apr 1, 2020

Two issues come to mind with this, and only because I've thought about this before when considering parallelisation in the past:

  1. This makes the assumption that parallel in the path is GNU parallel.
    This is a problem because GNU parallel is not the same as, and not compatible with, parallel as shipped with moreutils. These changes don't distinguish between the two and thus we could have some weird and unexpected failures.
  2. This is a breaking change with the only indication of it being a change hidden away in the requirements.md.
    This new requirement is going to need to be made explicitly clear in the release notes, and should probably be documented in the README.md too, especially as it might not be easy to install both on the same system on some operating systems. I know Debian/Ubuntu plays nice, but I don't know about other OSes.

With both in mind, I think we need a guard really early in the backup and restore steps (can bung it inshare/github-backup-utils/ghe-backup-config along with the other early checks) that checks for the GNU parallel and fails early if it can't be found.

@lildude
Copy link
Member

lildude commented Apr 1, 2020

As an aside, as this is a breaking change, it should not ship until Backup Utils 2.21.0 at the earliest.

@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch 2 times, most recently from 868a1dc to 7ee1099 Compare April 1, 2020 15:58
jianghao0718
jianghao0718 reviewed Apr 1, 2020
View reviewed changes
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch 8 times, most recently from fa53e9d to 6c3e4ad Compare April 2, 2020 02:27
snh
snh reviewed Apr 2, 2020
View reviewed changes
Copy link
Member

@snh snh left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making these changes @ryansimmen ❤️

It looks like there are some test failures that need to be resolved here.

As @lildude mentioned, we should also hold off merging this for now until closer to 2.21. This will also give us some time to exert this under more varied testing scenarios.

lildude
lildude suggested changes Apr 2, 2020
View reviewed changes
Copy link
Member

@lildude lildude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few recommendations. The test failures are legit too.

@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch from 6c3e4ad to 27511e0 Compare April 2, 2020 15:00
@ryansimmen
Copy link
Member Author

  1. This makes the assumption that parallel in the path is GNU parallel.
    This is a problem because GNU parallel is not the same as, and not compatible with, parallel as shipped with moreutils. These changes don't distinguish between the two and thus we could have some weird and unexpected failures.

With both in mind, I think we need a guard really early in the backup and restore steps (can bung it inshare/github-backup-utils/ghe-backup-config along with the other early checks) that checks for the GNU parallel and fails early if it can't be found.

This PR is now using moreutils parallel instead of GNU parallel and if both are installed the code will check the path for the presence of parallel and choose the appropriate executable.

@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch from 66ae7f5 to 7561b14 Compare April 4, 2020 11:57
@ryansimmen
Copy link
Member Author

Here is a thread describing why gawk is needed over mawk https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=593504

@ryansimmen ryansimmen mentioned this pull request Apr 5, 2020
Merged
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch from af61628 to 6a47bf3 Compare April 5, 2020 17:11
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch from 6a47bf3 to d263c8a Compare April 5, 2020 17:12
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch 4 times, most recently from 01cabcd to 53912d8 Compare April 6, 2020 01:10
@ryansimmen ryansimmen force-pushed the ryansimmen/parallel-backup branch from 53912d8 to c8eccf8 Compare April 6, 2020 02:25
lildude
lildude reviewed Apr 6, 2020
View reviewed changes
cainejette
cainejette reviewed Apr 6, 2020
View reviewed changes
cainejette
cainejette reviewed Apr 6, 2020
View reviewed changes
cainejette
cainejette reviewed Apr 6, 2020
View reviewed changes
cainejette
cainejette reviewed Apr 6, 2020
View reviewed changes
@ryansimmen ryansimmen merged commit 9449255 into master Apr 7, 2020
This was referenced Jun 9, 2020
Closed
Merged
@randyr505
Copy link

Glad to see my POC has come to fruition! I can't wait to test this out in our environment! @ryansimmen is there a limit for the parallel rsyncs? In my POC I ran 7000 at a time, lol. Thanks for your great work on this!

@ryansimmen
Copy link
Member Author

Glad to see my POC has come to fruition! I can't wait to test this out in our environment! @ryansimmen is there a limit for the parallel rsyncs? In my POC I ran 7000 at a time, lol. Thanks for your great work on this!

@randyr505 yes, you may simply set GHE_PARALLEL_RSYNC_MAX_JOBS. For reference please see https://github.com/github/backup-utils/blob/master/backup.config-example#L72

timreimherr pushed a commit that referenced this pull request Oct 24, 2023
@jianghao0718 @chuckp22
Manual backport of #597 - resolved conflict
d357bfb
timreimherr pushed a commit that referenced this pull request Oct 24, 2023
@jianghao0718 @chuckp22
Manual backport of #597 - resolved conflict
739a56b
amildahl pushed a commit that referenced this pull request Nov 28, 2023
@jianghao0718
Merge pull request #597 from github/hap/set-enterprise2-branch-env-var
eebe329 
Set enterprise2 branch as env var
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jianghao0718 jianghao0718 jianghao0718 left review comments

@dbussink dbussink Awaiting requested review from dbussink

+3 more reviewers

@lildude lildude lildude approved these changes

@snh snh snh approved these changes

@cainejette cainejette cainejette approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees

@ryansimmen ryansimmen

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@ryansimmen @lildude @randyr505 @snh @cainejette @jianghao0718