Replace os filesystem operations with pkg/filesystem
#22
Conversation
os file operations with pkg/filesystemos filesystem operations with pkg/filesystem
1b29556 to
426b71f
Compare
3d92ec1 to
48336bc
Compare
| ) | ||
|
|
||
| // FS is the default filesystem used by the package. | ||
| var FS vfs.FS = vfs.OSFS |
There was a problem hiding this comment.
Should we protect this with a mutex?
There was a problem hiding this comment.
We could, but swapping out the filesystem after the lib has been used is a bit iffy. Would result in an inconsistent filesystem. 🤔
There was a problem hiding this comment.
I'm going to merge it like this for now, let's add it if we find a use-case 👍🏻. Most operations would need to change from filesystem.FS to filesystem.FS() though so it would mean another commit with many changes, unfortunately.
pkg/snapshot/snapshot.go
Outdated
| _, _, errno := syscall.Syscall(unix.SYS_SYNCFS, dir.Fd(), 0, 0) | ||
| d, ok := dir.(*os.File) | ||
| if !ok { | ||
| return nil, nil, errors.New("unable to convert directory to file") |
There was a problem hiding this comment.
Would it make more sense here to call sync on filepath.Dir(dir) instead of erroring?
There was a problem hiding this comment.
We need the file descriptor (.Fd()) for the sycall, so there's nothing we can do if this fails.
We could do an interface test for Fd() uintptr instead though as it's more flexible.
@mtojek I'll make a separate PR for that 👍🏻 |
This PR replaces most
osfilesystem operations withvfs.FSinvoked viapkg/filesystem.Part of coder/envbuilder#230.
This allows us to run Kaniko (ish) safely with read-only or memory filesystems. It also allows us to transparently decompress cache archives to support multi-stage build cache probing.
Kaniko still calls out to a few libraries (e.g.
godirwalk,otiai10/copy, etc.) that don't support swapping outos. That's not something we have to deal with for cache probing so it's not a high priority.For simplicity,
filesystem.CreateTempandfilesystem.MkdirTempjust call out to theirosequivalents for simplicity but kept in thefilesystempackage for consistency (oswrite operations).Note: Initially this PR used afero, but afero was replaced due to inconsistency with stdlib functions (e.g.
afero.Walk!=filepath.Walk) and convoluted API ((afero.Lstater).LstatIfPossible, among other).