Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Move edits to their own collection (do NOT merge until after v3)#4328

Draft
imnasnainaec wants to merge 5 commits into
masterfrom
useredit-edits-collection
Draft

Move edits to their own collection (do NOT merge until after v3)#4328
imnasnainaec wants to merge 5 commits into
masterfrom
useredit-edits-collection

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Caution

DO NOT MERGE until after the v3 release. This PR is a breaking MongoDB schema change with a required data migration. It is intentionally held back so it doesn't land in the v3 release; merge only once v3 has shipped and the migration plan below has been agreed on.

Resolves #4320 (the "longer term" fix proposed there). Supersedes #4327.

Problem

A UserEdit document embeds its entire edits array, which grows without bound, and every goal/step write loaded the full document and rewrote it with ReplaceOneAsync. Once a document neared MongoDB's 16 MB limit, every write failed with WriteError 17419 and the user was permanently blocked from recording goal/step progress in that project (observed in production; see #4320).

Fix

Each Edit now lives in its own document in a new EditsCollection (StoredEdit: projectId, userEditId, plus the edit fields), and the UserEdit document's edits array holds only ObjectId refs, preserving order (StoredUserEdit). No document now grows meaningfully with use, so there is no cap on edit history.

  • Goal/step writes are targeted atomic updates on the small per-edit documents; UserEditRepository.Replace (full-document rewrite) is removed.
  • Appending a new edit inserts the StoredEdit and pushes its ref inside a multi-document transaction (the DB always runs as a replica set), so a failed append can't leave orphans.
  • Reads assemble the wire UserEdit from both collections, so the API contract is unchanged — no frontend or OpenAPI changes.
  • rm_project.py now also purges EditsCollection.

Mongo migration instructions (required)

database/migrate-useredits-to-edits-collection.js moves every embedded edit into EditsCollection, replaces each edits array with ordered refs (documents are modified in place — never deleted, since each user's workedProjects references the document _id), creates the { projectId, userEditId } index, and verifies that no old-format documents remain and every ref resolves. It is idempotent and safe to rerun.

This is a breaking schema change — old backends can't read migrated documents and the new backend can't read unmigrated ones — so it must be coordinated with the deploy:

  1. Back up — e.g. maintenance/scripts/combine_backup.py, or at minimum mongodump --db=CombineDatabase --collection=UserEditsCollection.

  2. Stop the backend (brief maintenance window), e.g. kubectl -n thecombine scale deployment backend --replicas=0.

  3. Run the migration on the database pod:

    kubectl -n thecombine cp database/migrate-useredits-to-edits-collection.js <database-pod>:/tmp/migrate-useredits-to-edits-collection.js
    kubectl -n thecombine exec <database-pod> -- mongosh CombineDatabase /tmp/migrate-useredits-to-edits-collection.js
  4. Deploy this version of the backend (which also restores the replica count).

  5. If a pre-migration backup is ever restored later, rerun the script.

Per the precedent of the GUID-subtype migration (#4179 / #4209), the script can be removed from the repo in a follow-up once it has been run everywhere.

Testing

  • dotnet build BackendFramework.sln: 0 warnings, 0 errors.
  • dotnet test Backend.Tests: 1196/1196 passed, including 20 new UserEditRepositoryTests integration tests against a real (ephemeral) mongod replica set: create/read round-trip across both collections, assembly in ref order, transactional no-orphan behavior on failed appends, delete cascades, the targeted goal/step updates, and the atomic step bounds check (UpdateStepInEdit requires the target step to exist in its filter, so an out-of-bounds $set can't null-pad stepData).
  • The repository integration fixtures now share one ephemeral mongod via a SetUpFixture; two runner instances in one test process caused intermittent MongoNotPrimaryException flakiness.
  • Existing UserEditControllerTests pass unchanged, confirming the API behavior is untouched.
  • The raw BSON layout is asserted in tests (the edits refs serialize as ObjectIds), so it matches what the migration script writes.
  • npm run fmt-backend: clean.

Notes for reviewers

  • Fresh installs get the EditsCollection index only via the migration script (the backend doesn't manage indexes anywhere today). The index is a perf optimization — correctness doesn't depend on it — but if we'd rather the backend ensure it at startup, that's a small follow-up.
  • Already-affected production documents are fully unblocked by the migration itself; their whole history is preserved (nothing is trimmed).
  • AddGoalToUserEdit now decides add-vs-replace from the atomic ReplaceEdit result instead of a prior read, and UpdateStepInEdit requires the target step to exist in its filter, so neither the goal write nor an out-of-bounds step write can corrupt data under concurrency. One pre-existing, low-risk race remains and is intentionally out of scope: UserEditController.UpdateUserEditStep still reads the step count to choose append-vs-overwrite, so concurrent step writes to the same goal could pick the wrong branch. Fixing it would require reshaping the wire contract (the client sends StepIndex ?? append), and the atomic bounds guard already rules out the only data-corruption outcome, so this is left for a follow-up.
  • The EditsCollection index the migration creates is { projectId, userEditId, guid }, covering both the per-user-edit listing and the single-edit replace/step queries.

This change is Reviewable

Store each Edit as its own document in a new EditsCollection, with the
UserEdit document's edits array holding only ordered ObjectId refs, so
no document grows toward MongoDB's 16 MB limit as edit history
accumulates. Goal/step writes are now targeted atomic updates on the
small per-edit documents instead of full-document replaces; appends
insert the edit and push its ref in a transaction. The API contract is
unchanged: reads assemble the same UserEdit shape as before.

Add database/migrate-useredits-to-edits-collection.js, a one-shot
mongosh script to move existing embedded edits into the new collection,
and purge EditsCollection in rm_project.py.

Fixes #4320

Co-Authored-By: Claude Fable 5 <[email protected]>
@imnasnainaec imnasnainaec added bug Something isn't working backend wip Work In Progress labels Jul 9, 2026
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 434f2cae-a3e3-4058-8285-3c8ecda1b188

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch useredit-edits-collection

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.10%. Comparing base (713f8de) to head (fe4fa1e).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4328      +/-   ##
==========================================
+ Coverage   75.96%   76.10%   +0.13%     
==========================================
  Files         305      305              
  Lines       11384    11383       -1     
  Branches     1411     1407       -4     
==========================================
+ Hits         8648     8663      +15     
+ Misses       2332     2321      -11     
+ Partials      404      399       -5     
Flag Coverage Δ
backend 87.53% <100.00%> (+0.31%) ⬆️
frontend 66.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@imnasnainaec imnasnainaec self-assigned this Jul 9, 2026
imnasnainaec and others added 4 commits July 9, 2026 13:37
Make UpdateStepInEdit require the target step to exist in its filter,
so the bounds check is atomic with the write and a concurrent shrink
of stepData can no longer cause $set to pad the array with nulls; the
service's read-based pre-check is now redundant and removed.

Make the test mock filter by projectId in GetUserEdit, Delete, and
DeleteAllUserEdits to match the real repository, and bounds-check
UpdateStepInEdit like the real filter does.

Also share one ephemeral mongod across repository integration test
fixtures via a SetUpFixture: running two runner instances in one test
process caused intermittent MongoNotPrimaryException failures.

Co-Authored-By: Claude Fable 5 <[email protected]>
Replace the read-then-branch in AddGoalToUserEdit with try-ReplaceEdit-
else-AddEdit, so the add-vs-replace decision comes from the atomic
ReplaceEdit result rather than a separate stale read (removing the
TOCTOU race and a query).

Extend the EditsCollection migration index to
{ projectId, userEditId, guid } so the single-edit replace/step
queries are covered as well as the per-user-edit listing.

Align the test mock's ReplaceEdit lookup to FindLastIndex to match
the service, and add a controller test for updating an existing goal.

Co-Authored-By: Claude Fable 5 <[email protected]>
The comment implied the replace-or-add sequence was atomic; reword it
to state each repo call is atomic on its own and that the narrow
duplicate-guid window on concurrent first writes is tolerated.

Co-Authored-By: Claude Fable 5 <[email protected]>
The real repository's ReplaceEdit uses UpdateOne, which matches the
first document by filter; align the mock to FindIndex accordingly.
The service no longer does a FindLastIndex lookup (it now dispatches
on ReplaceEdit's result), so mirroring the real repo is the right
fidelity target.

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend bug Something isn't working deployment goal python test wip Work In Progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UserEdit document can exceed MongoDB's 16 MB limit, permanently blocking goal/step writes

1 participant