Update tombstone API to minimize DB ops #429
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dillongeorge
approved these changes
Mar 26, 2024
| epoch: u64, | ||
| ) -> Result<(), StorageError> { | ||
| let key_data = self.get_user_data(username).await?; | ||
| let mut new_data = vec![]; |
Contributor
There was a problem hiding this comment.
Nit: I think you can write this a bit more cleanly (without a mutable variable) by doing something like so:
let new_data = key_data
.states
.into_iter()
.filter_map(|vs| {
if vs.epoch <= epoch && vs.value.0 != crate::TOMBSTONE {
DbRecord::ValueState(ValueState {
epoch: value_state.epoch,
label: value_state.label,
value: crate::AkdValue(crate::TOMBSTONE.to_vec()),
username: value_state.username,
version: value_state.version,
})
} else {
None
}
})
.collect();
Member
Author
There was a problem hiding this comment.
Ah, the rest of this file uses for loops and mutable collections, I'll keep it the current way for consistency for now. Thanks for the suggestion!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current
Our current tombstoning API
tombstone_value_statesaccepts a list ofValueStatekeys that tombstoning should be performed for. This means that the client flow will, more often than not, involve:ValueStateobjects for a given user in DBInternally, the tombstone API then does another lookup of
ValueStateobjects with the keys passed, modifies the objects to "tombstone" them, then another write to commit the now-tombstoned objects.Proposed
We can cut out the DB lookup pre-API call by having the API take in an epoch and a username, then tombstoning all
ValueStateobjects for the user with an epoch less than or equal to the passed epoch. Clients will not need any knowledge of whatValueStateobjects a user has, and just need to inform the API which epoch to stop tombstoning at.In addition, before we write the tombstoned records, we should filter out any objects that are already tombstoned, to avoid unnecessary writes.