Add support for xtrabackup, xtrabackup-stream, mydumper, mysqldump seed methods #22
Conversation
|
@shlomi-noach Hi Shlomi. Have you got time to look at this changes? |
|
My sincere apologies, not yet. Made sure to make myself a "reviewer" - it will now keep bugging me until resolved! Hope to get to this soon. |
|
@shlomi-noach thanks a lot! I will be on vacations for next two weeks, so if there will be any comments - will be glad to answer after them. |
|
I am finally getting to review this. |
build.sh
Outdated
| @@ -53,7 +53,7 @@ function precheck() { | |||
| ok=1 | |||
| fi | |||
|
|
|||
| if [[ $(go version | egrep "go1[.][01234]") ]]; then | |||
| if [[ $(go version | cut -d " " -f 3 | cut -d "." -f 2) -lt 5 ]]; then | |||
build.sh
Outdated
| @@ -103,8 +103,8 @@ function package() { | |||
| tar -C $builddir/orchestrator-agent -czf $TOPDIR/orchestrator-agent-"${RELEASE_VERSION}"-$target-$arch.tar.gz ./ | |||
|
|
|||
| echo "Creating Distro full packages" | |||
| fpm -v "${RELEASE_VERSION}" --epoch 1 -f -s dir -t rpm -n orchestrator-agent -C $builddir/orchestrator-agent --prefix=/ . | |||
| fpm -v "${RELEASE_VERSION}" --epoch 1 -f -s dir -t deb -n orchestrator-agent -C $builddir/orchestrator-agent --prefix=/ --deb-no-default-config-files . | |||
| fpm -v "${RELEASE_VERSION}" --rpm-os $target --architecture $arch --epoch 1 -f -d nc -s dir -t rpm -n orchestrator-agent -C $builddir/orchestrator-agent --prefix=/ . | |||
go/cmd/orchestrator-agent/main.go
Outdated
| } | ||
|
|
||
| config.Read([]string{"/etc/my.cnf", "/etc/mysql/my.cnf", "/usr/etc/my.cnf"}, "mysqlconfig") |
There was a problem hiding this comment.
What does "mysqlconfig" stand for?
There was a problem hiding this comment.
It's for reading MySQL configuration files in config.go
func Read(fileNames []string, configtype string) *Configuration {
if configtype == "appconfig" {
for _, fileName := range fileNames {
readJSON(fileName)
}
}
if configtype == "mysqlconfig" {
for _, fileName := range fileNames {
readINI(fileName)
}
}
return Config
}
go/config/config.go
Outdated
| // If the file does exist, then it is expected to be in valid INI format or the function bails out. | ||
| func readINI(fileName string) (*Configuration, error) { | ||
| cfg, err := ini.LoadSources(ini.LoadOptions{AllowBooleanKeys: true}, fileName) | ||
| if err == nil { |
There was a problem hiding this comment.
to have less indentation and clearer code, please use
if err != nil {
return Config, err
}
go/config/config.go
Outdated
| // AddKeyToMySQLConfig add new key with value to my.cnf | ||
| func AddKeyToMySQLConfig(key string, value string) error { | ||
| cfg, err := ini.LoadSources(ini.LoadOptions{AllowBooleanKeys: true, AllowShadows: true}, confFiles["MySQL"]) | ||
| if err == nil { |
There was a problem hiding this comment.
Same as above. Use if err != nil {...return...} and avoid indentations.
go/osagent/osagent.go
Outdated
| } | ||
| } | ||
| err = commandRun( | ||
| fmt.Sprintf(cmd), |
go/osagent/osagent.go
Outdated
| return BackupFolder, err | ||
| } | ||
|
|
||
| func backupMySQLUsers(seedId string) (err error) { |
There was a problem hiding this comment.
Please explain the purpose of backing up users?
There was a problem hiding this comment.
Sometimes there are tasks when we want to provision for example a read-only replica for some reporting purposes. And we want our reporting team to connect only for this replica, so when provisioning server and mysql we will create a specific login for this purposes. If we then will start provisioning backup from master using xtrabackup, we will lose this login. So this option is added to prevent such cases and backup specific logins before starting backup provisioning and restore them after it will be completed. There is a special config parameter for such cases called "MySQLBackupUsersOnTargetHost": ["'root'@'localhost'","'user_1'@'localhost'","'slave_user_1'@'localhost'","'not_exs_user'@'localhost'"]
go/osagent/osagent.go
Outdated
| } | ||
| } else { | ||
| cmd := fmt.Sprintf("mysqldump --master-data=2 --set-gtid-purged=OFF --user=%s --password=%s --port=%d --single-transaction mysql %s > %s", | ||
| config.Config.MySQLTopologyUser, config.Config.MySQLTopologyPassword, config.Config.MySQLPort, strings.Join(mysqlUsersTables[:], " "), path.Join(config.Config.MySQLBackupDir, mysqlUserBackupFileName)) |
There was a problem hiding this comment.
mysqlUsersTables[:] == mysqlUsersTables?
go/osagent/osagent.go
Outdated
| } | ||
| cmd := fmt.Sprintf("echo 'FLUSH PRIVILEGES;' >> %s", path.Join(config.Config.MySQLBackupDir, mysqlUserBackupFileName)) | ||
| err = commandRun( | ||
| fmt.Sprintf(cmd), |
go/osagent/osagent.go
Outdated
| } | ||
| cmd := fmt.Sprintf("tar zcfp %s -C %s .", path.Join(config.Config.MySQLBackupDir, mysqlBackupDatadirName), config.Config.MySQLDataDir) | ||
| err := commandRun( | ||
| fmt.Sprintf(cmd), |
There was a problem hiding this comment.
Taking a break. Review still not complete. TODO Marking this for next iteration.
|
My review is incomplete. I will continue at another time. I reviewed the PR linearly and got to about halfway. |
|
@MaxFedotov @shlomi-noach just curious, is this branch dead or is there any possibility that it will resume? I just found this project and thought it looks really cool, specially with the xtrabackup/mydymper/mysqldump seed methods. |
|
Hi @erkie, I will now working on redesign this and make different backup methods as plugins. Hope first draft will be ready in March |
|
@MaxFedotov excellent! Thanks for your awesome work. |
see #21 for description