Failover improvements #55

tgross · 2016-09-08T18:42:30Z

This PR changes the failover mechanism to be coordinated by mysqlrpladmin failover, which ensures that the transaction state is properly synced for the new master (at the expense of write availability during failover, which is part of our design anyways).

In order to make this project sanely testable, this work has included a refactoring to split the 1000+ lines of code into modules and classes that can have dependencies injected. ~~A new unit test suite includes using sys.settrace hooks to single-step thru simulated separate processes.~~ (Turns out this was totally unnecessary!)

@misterbisson as an FYI but the work isn't yet complete (note this has been rebased a bunch of times so the commit dates are fubar).

The MySQLUtilities package uses the Connector/Python library, which has a namespace collision with the PyMySQL library unless we install them in separate virtualenvs (which will complicate and bloat the container more). manage.py can use Connector/Python with minimal changes, mostly just working around bugs in sending multiple statements in a single `execute()` call.

Also, remove extra timestamp from manage.py logs

The existing code base has serious testability problems because it grew organically around a lot of global state. This refactoring moves most of the logic into separate classes that we can configure via DI and splits the classes out to their own modules for readability.

…esses

tgross · 2016-09-09T20:50:07Z

----------------------------------------------------------------------
Ran 37 tests in 7.137s

OK

@misterbisson at this point I have a passing unit test suite that gives us solid coverage of our configuration loading, pre_start, health, on_change, and snapshot_task, including a bunch of different failover scenarios. The idea behind this test suite is that it tests the algorithm we're using without worrying about the success of execution of MySQL commands, which we'll leave for the integration tests.

Starting next week I'll make sure this all works in the integration test suite and hands-on testing.

tgross · 2016-09-12T20:16:56Z

After fixing a couple dumb mistakes in my Python module layout, and a few real bugs, I now have successful failovers. The next step is to make sure the Shippable integration tests still work and update the README with the design changes.

misterbisson · 2016-09-12T20:37:37Z

etc/containerpilot.json

    {
      "name": "snapshot_check",
      "command": "python /usr/local/bin/manage.py snapshot_task",
-      "frequency": "10s",


Marking to come back to: why the increase from 10s to 5m?

At the end of the first pass thru the health check for the primary we do an initial snapshot so we can bootstrap replication (at the end of run_as_primary). Before we moved the snapshot into its own task we also checked if we needed a snapshot at the end of the health check. This worked fine because we'd already completed the run_as_primary steps. But when we moved it into its own task it overlaps with the health check, which means it can start to run before we've completed run_as_primary and this creates a ton of logging noise and errors. By moving it to 5 min we know that the initial setup has been completed without having to recheck it every time.

Also: at some point I'd like to look into improving this so we do incremental backups rather than full snapshots.

tgross · 2016-09-14T14:14:19Z

At this point I've got both unit tests and integration tests working on local Docker:

# make unit-test
.....................................
----------------------------------------------------------------------
Ran 37 tests in 1.100s

OK

# make test-local-docker
----------------------------------------------------------------------
MySQLStackTest.test_replication_and_failover
----------------------------------------------------------------------
elapsed  | task
1.817438 | docker-compose -f local-compose.yml -p my up -d
0.274307 | docker-compose -f local-compose.yml -p my ps
0.012382 | docker inspect my_consul_1
58.39401 | wait_for_service: mysql-primary 1
0.262109 | docker-compose -f local-compose.yml -p my ps -q mysql
0.138656 | docker exec d7f17f4770702404c912e2cb4edd0b5870792b453ae299040dd2dc0567e5b528 ip -o ad
0.405336 | assert_consul_correctness:
1.370994 | docker-compose -f local-compose.yml -p my scale mysql=3
26.12946 | wait_for_service: mysql 2
0.474732 | docker-compose -f local-compose.yml -p my ps -q mysql
0.186038 | docker exec d7f17f4770702404c912e2cb4edd0b5870792b453ae299040dd2dc0567e5b528 ip -o ad
0.076802 | docker exec 284f4a18a53fd665eabfe9f9b05c626527cba31d3c4fdc6221adb8886c443044 ip -o ad
0.072800 | docker exec 6383f672f14e586cde650932f3778e4fa76b94446aec43237e779a4a28563d58 ip -o ad
0.815884 | assert_consul_correctness:
0.242205 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e CREATE TABLE tbl1 (
0.124025 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1 (f
0.106777 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1 (f
0.103360 | docker exec 284f4a18a53f mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl
0.115371 | docker exec 6383f672f14e mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl
3.884255 | docker stop my_mysql_1
9.044485 | wait_for_service: mysql-primary 1
0.299285 | docker-compose -f local-compose.yml -p my ps -q mysql
0.013245 | docker exec d7f17f4770702404c912e2cb4edd0b5870792b453ae299040dd2dc0567e5b528 ip -o ad
0.104511 | docker exec 284f4a18a53fd665eabfe9f9b05c626527cba31d3c4fdc6221adb8886c443044 ip -o ad
0.077443 | docker exec 6383f672f14e586cde650932f3778e4fa76b94446aec43237e779a4a28563d58 ip -o ad
0.499566 | assert_consul_correctness:
0.002784 | wait_for_service: mysql 1
0.312183 | docker-compose -f local-compose.yml -p my ps -q mysql
0.013305 | docker exec d7f17f4770702404c912e2cb4edd0b5870792b453ae299040dd2dc0567e5b528 ip -o ad
0.103166 | docker exec 284f4a18a53fd665eabfe9f9b05c626527cba31d3c4fdc6221adb8886c443044 ip -o ad
0.120489 | docker exec 6383f672f14e586cde650932f3778e4fa76b94446aec43237e779a4a28563d58 ip -o ad
0.553040 | assert_consul_correctness:
0.116433 | docker exec 284f4a18a53f mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1
0.073758 | docker exec 6383f672f14e mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl
.
----------------------------------------------------------------------
Ran 1 test in 108.592s

OK

My tests aren't working on Triton right now but that's because of a setup problem (something to do with my credentials in the test environment... digging into it) and not a problem with the application.

tgross · 2016-09-14T14:49:23Z

@misterbisson I've pushed a big update to the README in this branch, which describes the new failover process and also outlines some of the guarantees and limitations of our setup.

misterbisson · 2016-09-14T15:09:26Z

README.md

+
+It's very important to note that the failover process described above prevents data corruption by ensuring that all replicas have the same set of transactions before continuing. But because MySQL replication is asynchronous it cannot protect against data *loss*. It's entirely possible for the primary to fail without any replica having received its last transactions. This is an inherent limitation of MySQL asynchronous replication and you must architect your application to take this into account.
+
+Also note that during failover, the MySQL cluster is unavailable for writes. Any client application should be using ContainerPilot or some other means to watch for changes to the `mysql-primary` service and halt writes until the failover is completed. Writes sent to a failed primary during failover will be lost!


Writes sent to a failed primary during failover will be lost!

Clarify: the primary will already be removed from Consul at that point, right? There is a clearly a race condition around the moment of failure, but once a primary is identified as failed, Consul won't report it as a primary anymore.

I think you're right to raise the warning here, perhaps I'm being defensive about making sure we know where the problem is.

misterbisson · 2016-09-14T15:17:37Z

This is looking solid all around.

I didn't see any changes here that would affect the configuration in https://github.com/autopilotpattern/wordpress. Am I missing anything? Is this gh39_use_standby-e8972e5 in https://hub.docker.com/r/autopilotpattern/mysql/tags/? If so, I should test it in the context of the WP implementation, yes?

tgross · 2016-09-14T15:20:34Z

I didn't see any changes here that would affect the configuration in https://github.com/autopilotpattern/wordpress. Am I missing anything?

Configuration should be the same.

Is this gh39_use_standby-e8972e5 in https://hub.docker.com/r/autopilotpattern/mysql/tags/? If so, I should test it in the context of the WP implementation, yes?

That tag is on the Hub and it sounds like a swell idea to test WP with it.

Still trying to figure out why make test-local-triton (which runs the test container locally but runs MySQL on Triton in us-sw-1) is giving me credentials-related errors.

tgross · 2016-09-14T15:42:30Z

Passing integration test suite on Triton:

----------------------------------------------------------------------
MySQLStackTest.test_replication_and_failover
----------------------------------------------------------------------
elapsed  | task
35.53385 | docker-compose -f docker-compose.yml -p my up -d
2.447364 | docker-compose -f docker-compose.yml -p my ps
0.940669 | docker inspect my_consul_1
65.88641 | wait_for_service: mysql-primary 1
3.004218 | docker-compose -f docker-compose.yml -p my ps -q mysql
3.494657 | docker exec c84516de92f5403ba0da4c733ca949 ip -o addr
6.676664 | assert_consul_correctness:
39.38328 | docker-compose -f docker-compose.yml -p my scale mysql=3
10.98797 | wait_for_service: mysql 2
4.712621 | docker-compose -f docker-compose.yml -p my ps -q mysql
3.667386 | docker exec c84516de92f5403ba0da4c733ca949 ip -o addr
3.197954 | docker exec 92c023506b9d4000863c3aef4ba534 ip -o addr
3.228828 | docker exec f5e2ea5642354ba6a3114f13d9007a ip -o addr
14.97571 | assert_consul_correctness:
3.051047 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e CREATE TABLE tbl1 (field1 INT, demodb
2.951622 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1 (field1, fiel demodb
3.046466 | docker exec my_mysql_1 mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1 (field1, fiel demodb
3.009918 | docker exec 92c023506b9d mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl1 WHERE `fiel demodb
3.151301 | docker exec f5e2ea564235 mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl1 WHERE `fiel demodb
10.99436 | docker stop my_mysql_1
5.538805 | wait_for_service: mysql-primary 1
4.406038 | docker-compose -f docker-compose.yml -p my ps -q mysql
1.983925 | docker exec c84516de92f5403ba0da4c733ca949 ip -o addr
3.001627 | docker exec 92c023506b9d4000863c3aef4ba534 ip -o addr
3.110972 | docker exec f5e2ea5642354ba6a3114f13d9007a ip -o addr
12.68047 | assert_consul_correctness:
0.089648 | wait_for_service: mysql 1
4.462744 | docker-compose -f docker-compose.yml -p my ps -q mysql
6.202129 | docker exec c84516de92f5403ba0da4c733ca949 ip -o addr
3.279309 | docker exec 92c023506b9d4000863c3aef4ba534 ip -o addr
3.434999 | docker exec f5e2ea5642354ba6a3114f13d9007a ip -o addr
17.55658 | assert_consul_correctness:
3.603358 | docker exec f5e2ea564235 mysql -u dbuser -p<redacted> --vertical -e INSERT INTO tbl1 (field1, fiel demodb
3.511389 | docker exec 92c023506b9d mysql -u dbuser -p<redacted> --vertical -e SELECT * FROM tbl1 WHERE `fiel demodb
17.82098 | docker-compose -f docker-compose.yml -p my stop
15.41207 | docker-compose -f docker-compose.yml -p my rm -f
.
----------------------------------------------------------------------
Ran 1 test in 282.843s

OK

misterbisson · 2016-09-15T04:15:24Z

gh39_use_standby-e8972e5 works for https://github.com/autopilotpattern/wordpress, though I wasn't able to upgrade a running MySQL cluster to the new version. We probably need an upgrade note in the readme making that clear.

tgross · 2016-09-15T12:54:07Z

Added a section to the README about upgrades and also added a table of contents to the top of the README.

misterbisson · 2016-09-15T20:35:33Z

🏡 🚶

tgross added 19 commits September 7, 2016 13:02

Remove tty flag from unit-test run, for compat w/ Shippable

79f3ab4

Run create_snapshot in same process as manage.py snapshot_task

2d97fd9

Give snapshot task user-friendly name for logs

07f6d60

Also, remove extra timestamp from manage.py logs

Provide tool to get generated root password from initial setup

e907eb4

Remove USE_STANDBY option

be7d04e

unit tests for config parsing and some low-hanging fruit

2a7a868

Proof-of-concept for using tracing hooks to unit test concurrent proc…

a73f717

…esses

Remove unncessary assertions in __init__.py

d1c3496

Unit testing for pre_start

298a49f

Unit testing for snapshot_task

1d3dc9a

Cut down on setup for pre_start tests

559ec6f

Clean up health check state machine, plus unit tests

083539a

Clean up unnecessary imports and arguments

75152b3

No need to use TestNode for pre_start w/ mock.side_effect

9a93fb5

Make sure tests can't stomp on each other's config files

8d732cd

Clean up failover state machine, plus unit tests

4c96756

While very clever, didn't need to run tests in steppable threads

23ee690

tgross added 7 commits September 12, 2016 14:39

Move files for correct Python path packaging

000978b

Move snapshot check frequency to something reasonable

4b7029c

Clean up import errs with new layout, a couple integration test fixes

f0e0a6c

more minor fixes

bc7a0f8

Force local to use DEBUG logging for ease of development

f856e4d

Make Consul check for primary use IP not name

2c1068e

Return own IP for primary when 'show slave hosts' is valid

f446a1e

tgross changed the title ~~[WIP] Failover improvements~~ Failover improvements Sep 12, 2016

misterbisson reviewed Sep 12, 2016
View reviewed changes

tgross added 4 commits September 13, 2016 15:12

Add makefile tooling to snag logs

a591207

Fix debug logging decorator

47c8a26

Simplify test query parsing

41ca4ab

Move failover unlock to end of health check

e8972e5

Update README for new failover design

dcff6fe

Inject links to ContainerPilot docs (#1)

72bee30

misterbisson reviewed Sep 14, 2016
View reviewed changes

Make sure TRITON_PROFILE is set during testing on Triton

b51941b

Reorder guarantees paragraph in README for clarity

3046c76

tgross added 2 commits September 15, 2016 08:50

Add README section on upgrades

f023f15

TOC in README

0a5bccd

tgross merged commit 25a0b14 into autopilotpattern:master Sep 15, 2016


		It's very important to note that the failover process described above prevents data corruption by ensuring that all replicas have the same set of transactions before continuing. But because MySQL replication is asynchronous it cannot protect against data loss. It's entirely possible for the primary to fail without any replica having received its last transactions. This is an inherent limitation of MySQL asynchronous replication and you must architect your application to take this into account.

		Also note that during failover, the MySQL cluster is unavailable for writes. Any client application should be using ContainerPilot or some other means to watch for changes to the `mysql-primary` service and halt writes until the failover is completed. Writes sent to a failed primary during failover will be lost!

Failover improvements #55

Failover improvements #55

Uh oh!

Conversation

tgross commented Sep 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgross commented Sep 9, 2016

Uh oh!

tgross commented Sep 12, 2016

Uh oh!

misterbisson Sep 12, 2016

Choose a reason for hiding this comment

Uh oh!

tgross Sep 13, 2016

Choose a reason for hiding this comment

Uh oh!

tgross Sep 13, 2016

Choose a reason for hiding this comment

Uh oh!

tgross commented Sep 14, 2016

Uh oh!

tgross commented Sep 14, 2016

Uh oh!

misterbisson Sep 14, 2016

Choose a reason for hiding this comment

Uh oh!

misterbisson commented Sep 14, 2016

Uh oh!

tgross commented Sep 14, 2016

Uh oh!

tgross commented Sep 14, 2016

Uh oh!

misterbisson commented Sep 15, 2016

Uh oh!

tgross commented Sep 15, 2016

Uh oh!

misterbisson commented Sep 15, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tgross commented Sep 8, 2016 •

edited

Loading