Prevent JSON file corruption with atomic writes #3586
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3586 +/- ##
===========================================
- Coverage 93.17% 93.10% -0.07%
===========================================
Files 68 68
Lines 15411 15432 +21
===========================================
+ Hits 14359 14368 +9
- Misses 1052 1064 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| if hasattr(os, 'fchmod'): | ||
| os.fchmod(temp_fd, 0o600) |
There was a problem hiding this comment.
This shouldn't be necessary – mkstemp's documentation says "The file is readable and writable only by the creating user ID.".
| if hasattr(os, 'fchmod'): | ||
| os.fchmod(temp_fd, 0o600) | ||
| with os.fdopen(temp_fd, 'w') as f: | ||
| temp_fd = None |
There was a problem hiding this comment.
os.fdopen is an alias of open. Exiting the context manager (for example when unwinding from an exception) should close the underlying fd, so I'm not sure the temp_fd = None + cleanup dance in except Exception below is necessary?
| f.flush() | ||
| os.fsync(f.fileno()) |
There was a problem hiding this comment.
Is there a particular reason to add an explicit flush + fsync?
Description of changes:
Use
os.replace()for atomic file operations to eliminate race conditions during concurrent access to the file cache. This ensures that writes are either fully completed or not applied at all, preventing partial writes that could leave the JSON file in an invalid state when multiple processes access the cache simultaneously.Note: This change is based on the approach suggested in #3544 by @ranlz77. Thanks for identifying this issue and proposing the initial solution.
Tests:
I verified that all tests including the the newly added unit tests pass.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.