Move most of initialization into health check #21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Because we set the "binlog isn't stale" flag at the end of a backup, a newly started cluster will spawn many `create_snapshot` processes even though we're locking the BACKUP_TTL in Consul. This adds a local file lock that prevents the node from running multiples. Also removed an unnecessary reliance on the repl user having access to the end-user's data DB just to run health checks.
If we don't make sure we create the file here we'll throw an IOError and catch it in the except block. This is ok in `is_backup_running` because if the file doesn't even exist we're obviously not running the backup either. But we need to make sure we create the file in `create_snapshot` or we end up bailing out too early.
MySQL must be running in order to execute most of our setup behavior so we're just going to make sure the directory structures are in place and then let the first health check handler take it from there. The happy path on the health check is to check a lock file against the node state and immediately return if we discover the lock exists. Otherwise, we bootstrap the instance.
- Corrected hostname vs service name confusion in get_state() - Added some extra debug logging - Fixed two dumb bugs in the locking for backups
Restoring from snapshot has to happen before we start mysqld, so if we have a clean start then we need to check in with Consul during startup to see if there's a snapshot available. Added debug trace logging and environment variable cleanup functions to assist with development and otherwise make life sane.
Contributor
|
🏡 🚶 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Having the setup for the database into the
preStarthandler has lead to a number of hard-to-debug races because we need to startup a temporary DB and then shut it down to run the "real" DB. This PR moves most of the initialization logic into the health check and drops a file lock on disk to make sure we only do it once per instance.While I was working on this I added some tooling to help future development:
./setup.sh test. I'd like to flesh these out further in the future but this is a good starting framework. (Props to @dissipate for encouraging me to finally start on this.)Also:
_envfile in./setup.shwill get parsed as values if you don't remove them.cc @misterbisson and @moretea @tianon as an FYI