Attach EBS volumes to etcd nodes for persistence #98
Conversation
If you ever lose your etcd cluster for whatever reason, or if you should ever need to restart it, you should be able to recover your state. Mentioned in this issue: kz8s#75
|
the snapshot module PR would be awesome too |
|
@rimusz Ok I've updated this to take automatic snapshots of ALL ebs volumes associated with the cluster at a rate of once every two hours. This includes the etcd volumes as well as any persistent volumes created by Kubernetes. Happy to split this into two PR's if you'd prefer to review it that way. For some background, originally I tried doing the snapshotting using only cloudwatch, but it wouldn't work without some manual configuration, so I ended up following the lambda approach described here. This also gives us the added benefit of backing up the dynamically generated volumes as well since it's based on tags. |
|
ty @adambom !! No need to split into two PRs - this one is fine. If you wanted to add a few notes to the readme, that would be great - np if not, I will add. |
|
@wellsie will do |
|
Done |
| resource "aws_volume_attachment" "etcd" { | ||
| count = "${ length( split(",", var.azs) ) }" | ||
|
|
||
| device_name = "/dev/xvdf" |
There was a problem hiding this comment.
@adambom - Where are you mounting xvdf ? Or am I missing something ?
There was a problem hiding this comment.
I forgot to include some changes to cloud-config.tf. On startup we format /dev/xvdf and mount it to /media/etcd2 and instruct etcd to use that location to persist state. I chose xvdf because that's the device we're using on worker nodes for ephemeral storage. See 8e73f42.
If you ever lose your etcd cluster for whatever reason, or if you should ever need to restart it, you should be able to recover your state. Mentioned in this issue: #75
I have a module that will automatically snapshot the drives. If that's at all of interest, happy to commit that as well.