fix(connections): remote database file copy correctness and cleanup - #2842
Merged
Merged
Conversation
…opies on remote database files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five correctness and data-safety defects in the read-only "Remote Database File" copy path (shipped in #2553, v0.69.0). Found while investigating #2831; each was reproduced before fixing. Independent of the live-session feature, and independent of each other.
Fixes
1. Stale
-walreplayed over a fresh copy.DatabaseFileLayout.staleAfterReplaceSuffixesdocuments the sidecars that must be cleared after the main file is replaced, but nothing called it. After a force-quit leaves anapp.db-walbeside a WAL-mode copy, the next fetch renames a fresh main file into place and SQLite replays the stale log against bytes it no longer matches.RemoteDatabaseFileTransfer.clearStaleSidecarsnow runs after the replace: a snapshot (fully checkpointed) clears every stale sidecar, a direct copy keeps the ones it just fetched and clears the rest.2. Copy reused after a WAL commit that didn't grow the log.
RemoteFileFingerprintcompared main size/mtime and WAL size. On Linux with the defaultjournal_size_limit = -1, a commit plus checkpoint can leave all three unchanged and move only the log's mtime, so a busy database read as unchanged and the stale copy was reused. The fingerprint (and the manifest) now recordwriteAheadLogModifiedand compare it. Old manifests without the field decode as nil and refetch once to gain a full baseline.3. Killed
VACUUM INTOreported as success.libssh2_channel_get_exit_statusreturns the stored0for a process killed by a signal, so an OOM-killed or interrupted snapshot command looked successful and its half-written snapshot was adopted.RemoteCommandResultnow reads the exit signal vialibssh2_channel_get_exit_signal, andsucceededrequires both a zero status and no signal.4. Interrupted snapshot files left on the server. The
VACUUM INTOsnapshot is removed by adeferthat cannot run when the session dies mid-fetch, so a dropped connection left a full copy of the database behind. The snapshot command now sweeps prior*.tablepro-snapshot-*leftovers before it writes a new one.5. Local working copies never deleted.
RemoteDatabaseFileStore.discard(_:)had no caller, and theabandonedCopies()a comment referenced did not exist, so a deleted connection or a changed path left multi-GB copies forever.pruneAbandonednow runs at launch and removes copies unused for 30 days; a reusetouches the copy so a daily-used database stays fresh. A remote-file copy is read-only and re-fetched on next open, so a swept copy costs one download and never loses data.Security
World-readable snapshot.
VACUUM INTOcreated the snapshot with the server umask (0644underumask 022) beside a0600database, exposing a full copy to every account on the host. The snapshot command now runs underumask 077, so it is0600from creation.Tests
RemoteDatabaseFileCorrectnessTests: the WAL-mtime fingerprint change and the identical case, a manifest missing the new field decoding as nil,clearStaleSidecarsfor both plans,succeededfalse on a signal, andpruneAbandonedremoving only a stale copy. The existingRemoteDatabaseFileTestsstill pass.Verification
The
umask/sweep behaviour on the server was validated in Docker during the #2831 investigation; the fingerprint WAL-mtime miss was reproduced on Ubuntu 24.04.