Adding lookup / history proof efficiency optimization #444
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #444 +/- ##
==========================================
- Coverage 88.61% 88.04% -0.57%
==========================================
Files 39 38 -1
Lines 9109 8282 -827
==========================================
- Hits 8072 7292 -780
+ Misses 1037 990 -47 ☔ View full report in Codecov by Sentry. |
akd_core/src/utils.rs
Outdated
| let mut future_marker_versions: Vec<u64> = Vec::new(); | ||
|
|
||
| let end_version_length = get_bit_length(end_version); | ||
| let mut future: u64 = end_version; |
There was a problem hiding this comment.
Very small nit pick: shall we consider renaming this to something that isn't easily mistaken for a future (in the async sense).
There was a problem hiding this comment.
Renamed to future_version (and the other variable from past -> past_version)
akd_core/src/utils.rs
Outdated
| // | ||
| // Note that if the input is less than the smallest element of the array, or the | ||
| // array is of length 0, then this function will panic. | ||
| fn find_max_index_in_array(input: u64, arr: &[u64]) -> usize { |
There was a problem hiding this comment.
To ensure I'm not missing anything, the arr in this case is usually a small sorted array; right? I ask because we can technically do a binary search on the arr (since it's sorted) to find the largest index. However, if the array is small, I imagine the linear search may actually be faster in practice. I don't think this matters too much in the grand scheme of things, but figured I'd mention it.
There was a problem hiding this comment.
Good call! Hmm... so I changed this function to actually be called find_max_index_in_skiplist() since we only ever call it on MARKER_VERSION_SKIPLIST. And I added a comment that we could optimize this to be a binary search instead of a linear search in the future, but that at the moment it doesn't make too much of a difference since the array is only 7 elements long.
| let small_jump = 10; | ||
| let medium_jump = 1000; |
There was a problem hiding this comment.
Think it makes sense to tie the numeric values to the enum entries as part of the type, or as a conversion into an i32 or something? Might make it a bit more explicit rather than putting those details in this fn. Feel free to ignore if you prefer this!
There was a problem hiding this comment.
There was some annoying math / edge cases to cover with these ranges that I ended up hardcoding with these as integers, and since it's test code, I figured not to bother too much in terms of over-engineering it. I think this is ok for now.
|
@dillonrg thanks for the feedback! |
| // between a binary search and a linear one. But if this ends up being problematic | ||
| // in the future, it could certainly be optimized. | ||
| // | ||
| // Note that if the input is less than the smallest element of the array, then this |
There was a problem hiding this comment.
Hmm, this actually will not panic either. This is another underflow area! See: https://play.rust-lang.org/?version=stable&mode=release&edition=2021&gist=d7a4ef00f8f4340ac911fe5cdeed68e5
There was a problem hiding this comment.
Ah, thanks! Fixed in the latest update by adding explicit checks that result in panics
37d08d2 to
3debd6a
Compare
Implements an efficiency optimization that changes the contents of a key history proof (reducing the number of total past marker versions and future marker versions) without compromising on security.
From the doc comments of get_marker_versions() (which is basically the only function that is updated):
The past marker versions are determined as follows:
largest power of 2 less than start_version, determined as follows: For each bit position i
in start_version, if the bit is 1, include the value of start_version with the ith bit set
to 0 and followed by trailing zeros.
As a concrete example, if start_version = 85, the past marker versions would be [16, 64, 80, 84].
Since:
01010101 => 85
01010100 => 84
01010000 => 80
01000000 => 64
And 16 comes from MARKER_VERSION_SKIPLIST.
The future marker versions are determined as follows:
MARKER_VERSION_SKIPLIST that is greater than start_version.
smallest power of 2 greater than start_version, determined as follows: For each bit position i
in start_version, if the bit is 0, include the value of start_version with the ith bit set
to 1 and followed by trailing zeros.
As a concrete example, if start_version = 85, the future marker versions would be
[86, 88, 96, 128, 256, 65536, 2^32] (potentially truncated depending on if any of these
numbers exceed epoch).
Since:
01010101 => 85
01010110 => 86
01011000 => 88
01100000 => 96
10000000 => 128
And the remainder of the list comes from MARKER_VERSION_SKIPLIST.
Note that the past marker versions do not contain start_version, as this would be redundant
in the history proof (since membership is already checked for start_version).