Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Solve a few issue with binary backup #548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Feb 11, 2020
Merged

Solve a few issue with binary backup #548

merged 20 commits into from
Feb 11, 2020

Conversation

jianghao0718
Copy link
Contributor

@jianghao0718 jianghao0718 commented Feb 5, 2020 •
edited
Loading

In this PR, we are trying to solve a few issues when backup and restore with binary backups

  1. Use pigz instead of gzip in backup-util
    Backup-util uses gzip which is super slow to compress large database files. It becomes the performance bottleneck when we backup binary backup. We will use pigz instead which is multi-threaded and way faster.

  2. Backup-util tool should always pointing to sql master for binary backup restore
    This is a bug in current binary backup restore. Backup-util restores mysqldump from any instance in cluster because it connects to sql master through connection. Binary backup restore has different mechanism. We need to connect to the Sql master node and copy files there instead. Customer does not necessarily restore on the MySql Master node.

/cc @github/ghes-infrastructure

@djdefi
Copy link
Member

djdefi commented Feb 5, 2020

I believe this introduces a new requirement to backup-utils, so we need to carefully consider this change. It would also need to be added to the requirements documentation here.

@jianghao0718 jianghao0718 changed the title Use pigz instead of gzip in backup-util [IN PROGRESS] Use pigz instead of gzip in backup-util Feb 6, 2020
@jianghao0718
Copy link
Contributor Author

@djdefi Right. And the tricky thing is changing the way we compress /decompress will have back-compat issue. I will continue working on this today to make sure we have least impact possible when improving performance

@djdefi
Copy link
Member

djdefi commented Feb 6, 2020

@snh corrected me, the pigz happens on the GHES instance, so this does not add a new dependency to the backup-utilities script after all.

@jianghao0718
Copy link
Contributor Author

@djdefi Correct. For the two code changes I had here, it is all running on GHE instances which has pigz already.

@jianghao0718 jianghao0718 changed the title [IN PROGRESS] Use pigz instead of gzip in backup-util Use pigz instead of gzip in backup-util Feb 6, 2020
lildude
lildude approved these changes Feb 6, 2020
View reviewed changes
Copy link
Member

@lildude lildude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me

@jianghao0718 jianghao0718 changed the title Use pigz instead of gzip in backup-util Use pigz instead of gzip in backup-util / Make sure MySql restore is on sql master for xtrabackup Feb 7, 2020
@jianghao0718 jianghao0718 changed the title Use pigz instead of gzip in backup-util / Make sure MySql restore is on sql master for xtrabackup Use pigz instead of gzip in backup-util Feb 7, 2020
@jianghao0718 jianghao0718 changed the title Use pigz instead of gzip in backup-util Solve a few issue Apple encounter with binary backup Feb 7, 2020
@jianghao0718
Copy link
Contributor Author

Hi @lildude and @djdefi , I added additional changes to backup-util for the issue we discovered yesterday, please help to review this additional change. Thanks a lot!

@jianghao0718 jianghao0718 changed the title Solve a few issue Apple encounter with binary backup Solve a few issue with binary backup Feb 7, 2020
@jianghao0718
Copy link
Contributor Author

jianghao0718 commented Feb 10, 2020 •
edited
Loading

@djdefi / @snh -- Do we always ask customer to run ghe-restore on their MySql master? @lildude told me he had the impression we did that. I have seen in the field people run ghe-restore on some other appliance without Mysql service and they hit the issue 2 I mentioned in description. Although the old logical backup restore could work on the other appliances because it connects to master and run scripts via MySql client.

lildude
lildude reviewed Feb 10, 2020
View reviewed changes
@oakeyc oakeyc mentioned this pull request Feb 10, 2020
Closed
@oakeyc oakeyc changed the base branch from master to rc2.19.3 February 10, 2020 23:12
@oakeyc oakeyc changed the base branch from rc2.19.3 to master February 10, 2020 23:16
@oakeyc oakeyc mentioned this pull request Feb 11, 2020
Merged
@oakeyc oakeyc merged commit 473d64b into master Feb 11, 2020
@oakeyc oakeyc deleted the hao/use-pigz-backuputil branch February 11, 2020 21:20
This was referenced Feb 13, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djdefi djdefi djdefi approved these changes

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach

+2 more reviewers

@lildude lildude lildude approved these changes

@oakeyc oakeyc oakeyc approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jianghao0718 @djdefi @lildude @oakeyc