Remove the epochs vector from HistoryTreeNode
#113
Conversation
…jority of operations only require the birth and current epochs. (Only audit & key history require the full list of epochs a node was mutated in) Resolves #112
|
I’m amazed at how fast you crunched this out. This all looks reasonable to me and is an elegant solution to the problem described in #112 🙂 Let's rebase this and let @kevinlewi have a chance to review before merging! |
| async fn get_epoch_lte_epoch( | ||
| &self, | ||
| node_label: crate::node_state::NodeLabel, | ||
| epoch_in_question: u64, | ||
| ) -> Result<u64, StorageError> { |
There was a problem hiding this comment.
So I guess the cool thing about doing this "linear search" for the correct epoch is that it can be done efficiently with a MySQL query, but may be less efficient if we are just relying on a simple set/get interface. Would it be better to find a middle-ground here with some pointer-based solution? Or perhaps we are not too worried about the performance impact of this function because it is not going to be called in lookup or publish?
There was a problem hiding this comment.
Yeah so the key history and audit pero queries do make this call, but we could probably figure out a nifty way to cache it but I think this is a good starting point at least.
I agree that it's not perfect but it completely mitigates the unbounded storage problems.
There was a problem hiding this comment.
Agreed, good point. Let's merge this for now then, and if we run into issues with performance on key history / audit, we can always revisit.
Thanks!!
There was a problem hiding this comment.
I believe most key-value storage systems provide some sort of key scan or match API that would make regex matching or prefix searching possible.
It would then be up to the storage layer implementer to use an encoding scheme that would allow for the most efficient searching. Perhaps still less efficient than an RDBMS like MySQL but might not be terrible if optimized to the specific system :)
This PR removes the the unbounded vector of epochs from
HistoryTreeNodein favor of just 2 u64's referring to the birth and death epochs.Resolves #112
Still need a merge with main to resolve merge failures.