Blockfile snapshotter: Preserve sparseness on linux#12956
Blockfile snapshotter: Preserve sparseness on linux#12956dcantah wants to merge 1 commit intocontainerd:mainfrom
Conversation
5bdf2dd to
6e7a376
Compare
|
@dcantah, please, see if there's anything useful in https://github.com/containerd/containerd/pull/12066/changes that you can borrow. |
|
Ha! @fidencio so we both ran into this LOL |
Yep. I closed my PR as I don't have time to follow-up on that, but if there's anything there that could be useful (either on the changes or on the discussions), feel free to use it in the best way possible. |
|
@fidencio The other thing I found that seems off in this sn is the Usage calculation just uses the apparent size of the file which feels... odd. If you had a 100MiB scratch file it seems like the diff between layer 1 and layer 2 would just be 100-100. Feels like that should really return the number of blocks (after this sparseness issue is fixed though) |
|
@fidencio The history here is funny, your change was based on another contributors also I see 😭. Third time's the charm ehh... By the way our attempts look mostly the same implementation wise barring some tests, structure, and an early return/fallback to non-sparse if the fs doesn't support it, so I'm going to add you as a co-author |
6e7a376 to
f106020
Compare
The Go stdlib has an enlightened ReaderFrom/WriterTo method on os.File on linux that goes through copy_file_range. However, it doesn't try and handle things correctly if the file is sparse. This change adds a SEEK_DATA/SEEK_HOLE based source to destination copy for the blockfile snapshotter. This will definitely be faster, but the bigger win should be disk space savings. Co-authored-by: Fabiano Fidêncio <[email protected]> Signed-off-by: Danny Canter <[email protected]>
f106020 to
1548700
Compare
| // | ||
| // If the filesystem does not support SEEK_DATA/SEEK_HOLE, it falls back | ||
| // to a plain io.Copy. | ||
| func copyFileWithSync(target, source string) error { |
There was a problem hiding this comment.
Wondering if this should be moved to continuity/fs.CopyFile
There was a problem hiding this comment.
Honestly, that seems like the right call to me. Want me to open there?
|
Drafting as we'll try and get the copyFile approach into the continuity repo and then get out a release and vendor it here |
The Go stdlib has an enlightened
ReaderFrom/WriterTomethod onos.Fileon Linux that goes throughcopy_file_range(2). However, it doesn't try and handle things "correctly" (preserving sparseness) if the file is sparse. This change adds aSEEK_DATA/SEEK_HOLEbased source to destination copy for the blockfile snapshotter. This should be faster, but the bigger win should be disk space savings.Edit:
Seems this has been attempted twice before LOL. Documenting for future github travelers:
#12066
#10197