Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Cap UserEdit edits to fix 16 MB limit (do NOT merge until after v3)#4327

Closed
imnasnainaec wants to merge 1 commit into
masterfrom
fix-useredit-16mb-cap
Closed

Cap UserEdit edits to fix 16 MB limit (do NOT merge until after v3)#4327
imnasnainaec wants to merge 1 commit into
masterfrom
fix-useredit-16mb-cap

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Caution

DO NOT MERGE until after the v3 release. This PR changes the UserEdit write path and includes a data migration that trims production data. It is intentionally held back so it doesn't land in the v3 release; merge only once v3 has shipped and the migration plan below has been agreed on.

Resolves #4320

Problem

A UserEdit document's edits array is unbounded, and every goal/step write loaded the full document and rewrote it with ReplaceOneAsync. Once a document neared MongoDB's 16 MB limit, every write failed with WriteError 17419 and the user was permanently blocked from recording goal/step progress in that project (observed in production; see #4320).

Fix

  • Targeted atomic updates instead of full-document replaces. UserEditRepository.Replace is removed; goal and step writes now use $push/$set on just the affected edits element.
  • Server-side cap on edits. Appending a new edit uses $push + $slice: -UserEdit.MaxEdits, so the array is atomically trimmed to the most recent MaxEdits entries on every append. Growth is bounded going forward.
  • MaxEdits = 250: production documents that hit the limit held 387–976 edits at 14–16 MB (up to ~40 kB per edit), so 250 keeps worst-case documents around 10 MB. Value is easy to tune if we want to keep more history.

Because the capped $push produces a smaller resulting document, deploying this backend also unblocks already-affected users on their next goal write — the migration below is still needed to shrink oversized documents proactively (they slow down reads such as GetProjectUserEdits).

Frontend requires no changes: edits are addressed by GUID everywhere; nothing persists an index into the edits array.

Mongo migration instructions

database/trim-user-edits.js trims every UserEdit document with more than 250 edits down to its most recent 250, in place (documents are never deleted — each user's workedProjects references the document _id). Run it once against production (and qa) after this PR is deployed:

  1. Back up first — trimmed entries are unrecoverable without a backup:

    # full backup via the maintenance pod, or at minimum:
    mongodump --db=CombineDatabase --collection=UserEditsCollection
  2. Run the script on the database pod:

    kubectl -n thecombine cp database/trim-user-edits.js <database-pod>:/tmp/trim-user-edits.js
    kubectl -n thecombine exec <database-pod> -- mongosh CombineDatabase /tmp/trim-user-edits.js

    The script prints each affected document (_id, project, edit count, size) before trimming and verifies afterward that no document exceeds the cap.

  3. Per the precedent of the GUID-subtype migration (Migrate MongoDB GUIDs from subtype 3 to 4 #4179 / Remove one-shot db guid migration script #4209), the script can be removed from the repo in a follow-up once it has been run everywhere.

Testing

  • dotnet build BackendFramework.sln: 0 warnings, 0 errors.
  • dotnet test Backend.Tests: 1186/1186 passed, including 12 new UserEditRepositoryTests integration tests that run the new atomic updates against a real (ephemeral) mongod — append, cap-trim (oldest dropped, newest last), replace-by-guid, step append/overwrite, and false-on-missing cases — plus a controller-level trim test via the mock.
  • npm run fmt-backend: clean.
  • Step writes previously stamped Edit.Modified; the new $push/$set updates preserve that (the goal history UI displays it).

Notes for reviewers

  • Replace had no callers outside UserEditService, so it was removed from IUserEditRepository, the repository, and the mock rather than left as dead code that reintroduces the full-document-rewrite pattern.
  • Each Edit.changes blob is still unbounded (a single huge edit could in theory exceed limits on its own); trimming what's stored per edit and/or moving edits to their own collection remain the longer-term options discussed in UserEdit document can exceed MongoDB's 16 MB limit, permanently blocking goal/step writes #4320.

This change is Reviewable

Replace full-document UserEdit writes with targeted atomic updates
and cap the edits array at the most recent 250 entries via $push +
$slice, so goal/step writes can no longer fail once a document
nears MongoDB's 16 MB limit.

Add database/trim-user-edits.js, a one-shot mongosh script to trim
existing oversized documents in place.

Fixes #4320

Co-Authored-By: Claude Fable 5 <[email protected]>
@imnasnainaec imnasnainaec added bug Something isn't working backend wip Work In Progress labels Jul 9, 2026
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7b1cb2da-a2ae-41eb-befc-990800cb9c17

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-useredit-16mb-cap

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.01%. Comparing base (713f8de) to head (9d336d4).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4327      +/-   ##
==========================================
+ Coverage   75.96%   76.01%   +0.04%     
==========================================
  Files         305      305              
  Lines       11384    11366      -18     
  Branches     1411     1408       -3     
==========================================
- Hits         8648     8640       -8     
+ Misses       2332     2325       -7     
+ Partials      404      401       -3     
Flag Coverage Δ
backend 87.37% <100.00%> (+0.15%) ⬆️
frontend 66.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@imnasnainaec imnasnainaec self-assigned this Jul 9, 2026
@imnasnainaec

Copy link
Copy Markdown
Collaborator Author

Superseded by a rework that will move edits into their own collection (one document per edit) instead of capping the array.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend bug Something isn't working goal test wip Work In Progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UserEdit document can exceed MongoDB's 16 MB limit, permanently blocking goal/step writes

1 participant