versioning of the [cipher text] on disk and to allow future schemes to be [added].

Done, I think "disk" was also misleading, changed to "in etcd".

Maybe this is just the pseudo code, but couldn't this use a struct to store these separate pieces instead of string munging/byte manipulation?

The stored data being a struct has been discussed - it comes with additional complexity in cost and perf. Prefixing is the cheapest option that requires no new serialization struct.

Our proto at rest format follows this pattern (magic prefix to make the bytes recognizable).

Added comment.

such that the first key is always the [key to use for encryption].

for persistent storage. [Within the cluster, it] only lives

s/--key-encryption-key-db-path/--encryption-provider-config/

Done.

Yes, there is a lag where some master may not be using the new write key, but as long as they all have it as a read key, it isn't important. Some will write with the new key, some will write with the old key, but all will be able to read with either key.

Done.

Is the race condition because not all masters will have the read key at the same time? If so, as long as no one is writing with that key, it doesn't matter. This step prepares all the masters for the safe transition to writing with the new key.

Yes. Make new key available everywhere before anyone writes.

Done.

Most closely matches the pattern that will be used for integrating with external encryption systems. Amazon KMS, Google KMS and HSM will serve the purpose of KEK storage rather than local disk.

Added.

Done, I think "disk" was also misleading, changed to "in etcd".

@smarterclayton did you want to switch to this? I got an internal crypto reviewer to look and he agreed with Diogo that XSalsa20 and Poly1305 should be OK.

Other than

1. pulling in a newer library (which has some small risk) which is not part of go standard crypto
2. requiring us to manage AEAD ourselves (bigger concern)
3. the cache attack is not much of a concern on x86 with AES-NI, but is more so on ARM

I'm not terribly concerned. I'm actually less concerned with having two impls than just having secretbox in the short term.

ack. Put this in an alternatives considered section.

Added comment.

Done.

Done.

Done.

Added.

typo

thanks, fixed.

Provide an example configuration?

needs more thinking, but yes.

Is this an API object? If so, please add it below with the rest of the objects, otherwise, you'll need to mention it before this section.

It is an API object but won't be exposed over REST. It would be like a component config or kubeconfig.

typo

thanks.

What's the expected volume of secrets here? Is there a performance impact of choosing only a single layer of keys vs. envelope encryption (DEK/KEK)?
A lot of customers I talk to are in the 100-1000 secrets range, so this should be fine, but worth considering.

that's a good question, 100-1000 seems about the right range to me. I wonder if the original secrets design had anything to say about this.

We routinely deal with clusters that have 100k and up secrets. In practice, it doesn't matter how many they are, they still have to be rotated. However, they don't have to be rotated synchronously - the whole point of having multiple keys is to allow that process to be done in the background. You have to execute a no-op PUT to every secret and verify you get a resource change (no resource change means the secret is already stored with the new key). Once you've completed that, you can remove the old key (it's zero cost to leave it, other than search order though).

(this is the same feature we use to move stored API resources to a new storage version)

ack. This seems like me to a push to envelope encryption model

Why does the rotation need to generate a new DEK? Or rather, in a non-emergency rotation (say, with automated rotation), this adds a lot of complexity.

Trying to reason through this (feel free to correct me):

1. Rotation of just the KEK time-limits exposure of the previous KEK where the attacker didn't also retrieve the DEKs.

2. DEKs need to be rotated every 2^32 writes somehow.

3. We need to rotate everything if DEKs and KEKs got exposed.

Given we need to write code for 2 and 3, is there much advantage shooting just for 1?

re: 2, ahh I see my confusion. Recommendation would be to generate a new DEK every write (not change on reads)
re: 3, yes, but this is (hopefully) a rare case. We still need to write code but it shouldn't be built into the 'default' rotation model.

My preferred end state is a combination of 2 & 3 - local DEK are generated and sent elsewhere to be encrypted with KEK. In this implementation, it may be a local KEK on disk, but longer term, there could be other backends and others. This also circumvents the KEK agreement bit because in 4 because they just need to agree on a single source which maintains the KEKs, rather than agree on the KEKs themselves.

yes, I think doing it like this only really makes sense if you use the external store. I'll reword this one when I get a chance.

This sounds like the ideal design to me as well. In the near term—if the initial implementation only supports storing KEKs as files on disk, is it correct we could:

a) make sure that the paths they're located at on disk (as defined in --encryption-provider-config=/path/to/config) is somewhere on our existing encrypted ramdisk? This is where we write secrets after we've fetched them from Vault, and;

b) automatically back them up as normal Vault secrets, and place them on new apiserver k8s hosts the way we place other secrets already?

These questions are probably a bit out of scope for a spec document, but I wanted to think through how we'd use this in a real world scenario.

Is this 1 with some sort of root key that unlocks all the disk keys?

I think? this is saying there's only one layer of keys, and they are retrieved from the HSM. So it's like option 1 with the keys in the HSM instead of on disk. @philips wrote this originally I believe.

Is the DEK forcibly cached in memory? Or do we want to put an external store in the serving path of a secret? (this likely also depends on the sensitivity of the workload, where it's hosted/ latency to external store, etc.). I would propose that if one of the DEK/KEK solutions is pursued, this should be a design option.

yes, the DEK is held in memory and we don't want to have to wait on decrypt by an external store for every secret. I think you're saying we should consider relying completely on the external store and not doing any caching?

I think that we have caching be the default, but might want to make this an option (not high priority).

It is an API object but won't be exposed over REST. It would be like a component config or kubeconfig.

Keys coming from local disk seems like a limitation to the design. Wouldn't this be difficult to manage by administrators of the cluster?

I think I should take the talk of key management out of this section (holdover from the previous doc). The crypto implementation doesn't care where the keys come from. Trade offs in terms of management complexity are covered under the options presented in the keys section.

I had this question on 454 also. Do you mind describing a little more by what it means "from the command line"?

This depends on which of the key management options we choose. For option 1 the key itself would be inside the config file. I reworded this to use a key retrieval interface that will be one of those options.

I just realized - we probably need to at least think about how to disable encryption. That means we need a config way to have decrypter keys in rotation, but ensure the write key is "none"

you should rather have a special none provider that has to be very explicitly configured, allowing "none" keys is generally dangerous.

That's in the implementation now.