Move edits to their own collection (do NOT merge until after v3)#4328
Move edits to their own collection (do NOT merge until after v3)#4328imnasnainaec wants to merge 5 commits into
Conversation
Store each Edit as its own document in a new EditsCollection, with the UserEdit document's edits array holding only ordered ObjectId refs, so no document grows toward MongoDB's 16 MB limit as edit history accumulates. Goal/step writes are now targeted atomic updates on the small per-edit documents instead of full-document replaces; appends insert the edit and push its ref in a transaction. The API contract is unchanged: reads assemble the same UserEdit shape as before. Add database/migrate-useredits-to-edits-collection.js, a one-shot mongosh script to move existing embedded edits into the new collection, and purge EditsCollection in rm_project.py. Fixes #4320 Co-Authored-By: Claude Fable 5 <[email protected]>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4328 +/- ##
==========================================
+ Coverage 75.96% 76.10% +0.13%
==========================================
Files 305 305
Lines 11384 11383 -1
Branches 1411 1407 -4
==========================================
+ Hits 8648 8663 +15
+ Misses 2332 2321 -11
+ Partials 404 399 -5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Make UpdateStepInEdit require the target step to exist in its filter, so the bounds check is atomic with the write and a concurrent shrink of stepData can no longer cause $set to pad the array with nulls; the service's read-based pre-check is now redundant and removed. Make the test mock filter by projectId in GetUserEdit, Delete, and DeleteAllUserEdits to match the real repository, and bounds-check UpdateStepInEdit like the real filter does. Also share one ephemeral mongod across repository integration test fixtures via a SetUpFixture: running two runner instances in one test process caused intermittent MongoNotPrimaryException failures. Co-Authored-By: Claude Fable 5 <[email protected]>
Replace the read-then-branch in AddGoalToUserEdit with try-ReplaceEdit-
else-AddEdit, so the add-vs-replace decision comes from the atomic
ReplaceEdit result rather than a separate stale read (removing the
TOCTOU race and a query).
Extend the EditsCollection migration index to
{ projectId, userEditId, guid } so the single-edit replace/step
queries are covered as well as the per-user-edit listing.
Align the test mock's ReplaceEdit lookup to FindLastIndex to match
the service, and add a controller test for updating an existing goal.
Co-Authored-By: Claude Fable 5 <[email protected]>
The comment implied the replace-or-add sequence was atomic; reword it to state each repo call is atomic on its own and that the narrow duplicate-guid window on concurrent first writes is tolerated. Co-Authored-By: Claude Fable 5 <[email protected]>
The real repository's ReplaceEdit uses UpdateOne, which matches the first document by filter; align the mock to FindIndex accordingly. The service no longer does a FindLastIndex lookup (it now dispatches on ReplaceEdit's result), so mirroring the real repo is the right fidelity target. Co-Authored-By: Claude Fable 5 <[email protected]>
Caution
DO NOT MERGE until after the v3 release. This PR is a breaking MongoDB schema change with a required data migration. It is intentionally held back so it doesn't land in the v3 release; merge only once v3 has shipped and the migration plan below has been agreed on.
Resolves #4320 (the "longer term" fix proposed there). Supersedes #4327.
Problem
A
UserEditdocument embeds its entireeditsarray, which grows without bound, and every goal/step write loaded the full document and rewrote it withReplaceOneAsync. Once a document neared MongoDB's 16 MB limit, every write failed withWriteError 17419and the user was permanently blocked from recording goal/step progress in that project (observed in production; see #4320).Fix
Each
Editnow lives in its own document in a newEditsCollection(StoredEdit:projectId,userEditId, plus the edit fields), and theUserEditdocument'seditsarray holds only ObjectId refs, preserving order (StoredUserEdit). No document now grows meaningfully with use, so there is no cap on edit history.UserEditRepository.Replace(full-document rewrite) is removed.StoredEditand pushes its ref inside a multi-document transaction (the DB always runs as a replica set), so a failed append can't leave orphans.UserEditfrom both collections, so the API contract is unchanged — no frontend or OpenAPI changes.rm_project.pynow also purgesEditsCollection.Mongo migration instructions (required)
database/migrate-useredits-to-edits-collection.jsmoves every embedded edit intoEditsCollection, replaces eacheditsarray with ordered refs (documents are modified in place — never deleted, since each user'sworkedProjectsreferences the document_id), creates the{ projectId, userEditId }index, and verifies that no old-format documents remain and every ref resolves. It is idempotent and safe to rerun.This is a breaking schema change — old backends can't read migrated documents and the new backend can't read unmigrated ones — so it must be coordinated with the deploy:
Back up — e.g.
maintenance/scripts/combine_backup.py, or at minimummongodump --db=CombineDatabase --collection=UserEditsCollection.Stop the backend (brief maintenance window), e.g.
kubectl -n thecombine scale deployment backend --replicas=0.Run the migration on the database pod:
Deploy this version of the backend (which also restores the replica count).
If a pre-migration backup is ever restored later, rerun the script.
Per the precedent of the GUID-subtype migration (#4179 / #4209), the script can be removed from the repo in a follow-up once it has been run everywhere.
Testing
dotnet build BackendFramework.sln: 0 warnings, 0 errors.dotnet test Backend.Tests: 1196/1196 passed, including 20 newUserEditRepositoryTestsintegration tests against a real (ephemeral) mongod replica set: create/read round-trip across both collections, assembly in ref order, transactional no-orphan behavior on failed appends, delete cascades, the targeted goal/step updates, and the atomic step bounds check (UpdateStepInEditrequires the target step to exist in its filter, so an out-of-bounds$setcan't null-padstepData).SetUpFixture; two runner instances in one test process caused intermittentMongoNotPrimaryExceptionflakiness.UserEditControllerTestspass unchanged, confirming the API behavior is untouched.editsrefs serialize as ObjectIds), so it matches what the migration script writes.npm run fmt-backend: clean.Notes for reviewers
EditsCollectionindex only via the migration script (the backend doesn't manage indexes anywhere today). The index is a perf optimization — correctness doesn't depend on it — but if we'd rather the backend ensure it at startup, that's a small follow-up.AddGoalToUserEditnow decides add-vs-replace from the atomicReplaceEditresult instead of a prior read, andUpdateStepInEditrequires the target step to exist in its filter, so neither the goal write nor an out-of-bounds step write can corrupt data under concurrency. One pre-existing, low-risk race remains and is intentionally out of scope:UserEditController.UpdateUserEditStepstill reads the step count to choose append-vs-overwrite, so concurrent step writes to the same goal could pick the wrong branch. Fixing it would require reshaping the wire contract (the client sendsStepIndex ?? append), and the atomic bounds guard already rules out the only data-corruption outcome, so this is left for a follow-up.EditsCollectionindex the migration creates is{ projectId, userEditId, guid }, covering both the per-user-edit listing and the single-edit replace/step queries.This change is