-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Better separation between MVStore and Filestore #3629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ck" with "len" in the footer
|
Are you sure about this? |
|
@grandinj Of course, I am not sure, at least not until it's done and tested.
Multi-file itself does not mean better or worse, faster or slower. Mutlti-file it's just efficient way to roll disk space. If file systems would provide trimming from the beginning of the file, we won't need multiple. And I am pretty sure that append-only is the most efficient writes possible. so I expect it to be more performant. Hopefully it'll be better in regards to disk space utilization.
This is unrelated issue, subject to debate. This PR is just giving us opportunity to develop pluggable alternative FileStore implementations, that's all it is. |
|
I like single-file approach, but for space compaction and defragmentation we need to rewrite everything and it is reasonable to use a separate file for it. If we have versions 1–100 in a file-1 and only few of them are currently used, online compaction will require creation of a new file-2 with only used versions in it and reordered data, but during this procedure we need a file-3 to store new versions 101… (modified data only) in it. After compaction file-1 can be removed and we'll have files 2 and 3. During the next compaction we can combine alive versions from 2 and 3 into new file-4 and use new file-5 for the new versions and so on. Offline defragmentation can combine all files into one, it doesn't need an additional file for parallel transactions. (It's just an idea.) Copyright years in |
|
Is LIFT more trouble than it is worth, should I disable it? |
|
Looks like it. but let's leave it on for a little while, and disable on case-by-case basis for now. |
This PR is an attempt to move all persistence-related code out of MVStore class into FileStore and more clearly define interface between the two without exposing details of Filestore implementation.
This would allow for swappable alternative FileStore specializations, i.e. multi-file one, or one which would not hold all chunks in memory.
It has non-functional stub for multi-file store, because that part is still work in progress.