Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(bulk-delete): incremental delete-by-pattern (client-side SCANDEL)#308

Merged
KIvanow merged 8 commits into
masterfrom
feat/bulk-delete-by-pattern
Jul 8, 2026
Merged

feat(bulk-delete): incremental delete-by-pattern (client-side SCANDEL)#308
KIvanow merged 8 commits into
masterfrom
feat/bulk-delete-by-pattern

Conversation

@KIvanow

@KIvanow KIvanow commented Jul 8, 2026

Copy link
Copy Markdown
Member

Summary

Adds a safe, client-driven bulk key deletion by pattern — the SCANDEL use case from valkey/valkey#2623. Upstream deferred the dedicated command in favour of Lua, so the monitor drives SCAN + UNLINK itself. That gives us dry-run, COUNT-style pacing, cancellation, cluster fan-out and an audit trail today — non-blocking and observable, which a one-shot Lua EVAL can't be.

Pro-tier, gated behind a new bulkDelete feature and a required connection header.

Changes

  • Engine (proprietary/bulk-delete): pure SCAN (MATCH/COUNT/TYPE) + UNLINK walk over a ScanTarget abstraction. Safety rails: mandatory pattern, catch-all (*) requires explicit confirmation, maxKeys cap with truncated reporting, inter-batch pacing, cooperative cancel.
  • Cluster-safe deletes: keys are removed via a pipeline of per-key UNLINKs, not one variadic command — a SCAN batch spans many hash slots and a multi-key command would otherwise hit CROSSSLOT on a cluster node.
  • Jobs: both preview (dry-run) and execute run as bounded background jobs, polled via GET /bulk-delete/jobs/:id, so a large/sparse keyspace can't block a request. Jobs are connection-scoped and cancellable.
  • Cluster fan-out across every discovered primary; unreachable primaries are skipped and recorded rather than failing the whole run.
  • API: POST /preview, POST /execute (202 + jobId), GET /jobs/:id, POST /jobs/:id/cancel, GET /audits.
  • Audit: one row per execute run (bulk_delete_audits) across sqlite/postgres/memory; runs left running by a crashed/restarted process are reconciled to failed on startup.
  • Web: Bulk Delete page — pattern/type/scope/count/pause/maxKeys inputs, preview with sample keys, typed DELETE confirmation, live progress with cancel, skipped-node warnings, recent-runs table.

Checklist

  • Unit / integration tests added
  • Docs added / updated
  • Roborev review passed — run roborev review --branch or /roborev-review-branch in Claude Code (internal)
  • Competitive analysis done / discussed (internal)
  • Blog post about it discussed (internal)

Notes

  • bulk_delete_audits is new in this PR, so no migration is needed for the added skipped_nodes column.
  • Startup reconciliation marks all running audit rows failed; in a multi-instance Postgres deployment this would also touch another instance's in-flight runs (fine for self-hosted / sqlite).

Note

High Risk
Irreversible key deletion against live Valkey/Redis with cluster fan-out; mistakes or overly broad patterns can cause data loss despite preview and confirmations.

Overview
Adds a Pro-tier bulk delete by pattern capability: the monitor runs incremental SCAN + UNLINK (dry-run preview and execute) instead of a one-shot server command, with license gating via new bulkDelete / Feature.BULK_DELETE.

The proprietary BulkDeleteModule exposes async jobs (POST /preview, POST /execute → 202 + job id), polling, cancel, and audit listing—all connection-scoped and behind LicenseGuard. The engine validates patterns (catch-all * needs confirmDeleteAll), caps preview size, supports pacing/cancel, fans out to cluster primaries (skips unreachable nodes), and deletes via per-key pipelined UNLINK to avoid cluster CROSSSLOT. One concurrent job per connection is enforced (409 on overlap).

Storage gains bulk_delete_audits (sqlite/postgres/memory) with upserted execute-run rows and startup reconciliation of stale running rows to failed.

The web adds a Bulk Delete page (pattern/type/scope, preview samples, typed DELETE confirm, live progress/cancel, audit table) plus API client and tests for stale-preview UX when the form changes mid-request.

Reviewed by Cursor Bugbot for commit 641ec84. Bugbot is set up for automated code reviews on this repo. Configure here.

Adds a safe, client-driven bulk key deletion by pattern — the SCANDEL
use case from valkey/valkey#2623, which upstream deferred in favour of
Lua. The monitor drives SCAN + UNLINK itself, which gives dry-run,
COUNT-style pacing, cancellation, cluster fan-out and an audit trail
without a dedicated server command.

Backend (proprietary/bulk-delete):
- Pure engine over a ScanTarget abstraction: MATCH/COUNT[/TYPE] walk with
  dry-run/execute modes, mandatory-pattern and catch-all guards, a
  maxKeys cap, batch pacing and cooperative cancel.
- Deletes are issued as a pipeline of per-key UNLINKs rather than one
  variadic command, so a batch spanning multiple hash slots does not hit
  CROSSSLOT on a cluster node.
- Preview and execute both run as bounded background jobs (polled via
  GET /jobs/:id) so a large/sparse keyspace can't block a request; jobs
  are connection-scoped and cancellable.
- Cluster fan-out skips unreachable primaries and records them instead
  of failing the whole run.
- Gated behind a new Pro `bulkDelete` feature and a required connection
  header. One audit row per execute run across sqlite/postgres/memory;
  interrupted (crash-left) runs are reconciled to failed on startup.

Web:
- Bulk Delete page: pattern/type/scope/count/pause/maxKeys inputs,
  preview with sample keys, typed DELETE confirmation, live progress with
  cancel, skipped-node warnings and a recent-runs table.

Tests: engine, scan-target (per-key UNLINK), service (scoping, cluster
fan-out, node-skip, startup sweep) and the web api client.
Comment thread apps/web/src/pages/BulkDelete.tsx Outdated
The Preview card rendered from the live polled dry-run job, while only
the confirm-dialog `preview` state was cleared when match/type/scope/
maxKeys changed. So after a dry-run finished, editing the target left a
stale card showing counts and sample keys for the previous target next
to the edited form. Gate the card on a `previewStale` flag that is set
when the target changes and cleared when a fresh preview starts.
Comment thread proprietary/bulk-delete/bulk-delete.service.ts
For cluster scope, if every discovered primary failed to connect,
buildTargets returned an empty target list and runJob still ran the
engine over zero nodes and marked the job 'completed' — reporting
success for a run that scanned and deleted nothing. Empty targets only
occurs in that all-unreachable case (node scope and the no-primaries
fallback always yield one target), so treat it as a failure with an
informative error; the skipped-node list is still recorded.
Comment thread proprietary/bulk-delete/bulk-delete.service.ts
Comment thread proprietary/bulk-delete/bulk-delete.service.ts
Comment thread proprietary/bulk-delete/bulk-delete.controller.ts
…allback

Addresses three review findings:

- GET /audits clamped `limit` to 500 but not to a minimum of 1, so a
  negative value reached storage (LIMIT -1 means "no limit" in SQLite,
  errors in Postgres). Clamp to [1, 500].
- Execute jobs fired the initial "running" audit write with `void` and
  wrote the final row in the finally block; a slow initial write could
  land after the final one and revert the row to running/0/null. Keep
  the initial write's promise and await it before the final write so the
  completed row always wins.
- Cluster scope threw and failed the whole job when discoverNodes errored
  on a non-cluster connection (CLUSTER NODES unsupported). Tolerate the
  error and fall back to the single connected node, matching the existing
  no-primaries behaviour.
Comment thread apps/web/src/pages/BulkDelete.tsx Outdated
…et edit

The previewStale flag hid the preview card whenever the target changed,
but the card also carries a running dry-run's live progress and Cancel
control — and the target inputs stay editable during a run. Editing them
mid-run therefore hid the progress card and Cancel while the server job
kept going. Only hide a *finished* preview once the target changed; a
running preview always renders. Extracted showPreviewCard() and unit
tested it.
Comment thread apps/web/src/pages/BulkDelete.tsx Outdated
Comment thread apps/web/src/pages/BulkDelete.tsx Outdated
… too

Two review findings:

- The web sends the same body (including batchPauseMs) for preview and
  execute, but batchPauseMs was declared only on the execute DTO. Under
  the global forbidNonWhitelisted pipe, a preview with a pause value
  returned 400. Move batchPauseMs onto the shared preview DTO so both
  accept it (and the dry-run walk honours it).
- The run-progress card was only gated on job.mode, so a finished execute
  run stayed on screen for an old pattern after the target was edited.
  Generalise the staleness rule (extracted as jobCardVisible) to both
  cards: a running job always shows; a finished card hides once the
  target changes. The flag now resets whenever any job starts.
Comment thread proprietary/bulk-delete/bulk-delete.service.ts

@jamby77 jamby77 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

startJob launched a new background run on every call without checking
whether the same connection already had one in flight, so repeated
preview/execute API calls could stack overlapping SCAN/UNLINK walks
against one node.

Guard startJob with findActiveJob(): if the connection already has a
running job, throw ConflictException (409) naming it and asking the
caller to wait or cancel first. The check-then-register path is
synchronous, so two rapid calls can't both pass the guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01KiqY4sKyXLfBG9rhgh1Zj7
Comment thread apps/web/src/pages/BulkDelete.tsx
…e-preview race

If the match/type/scope/maxKeys changed after Preview was clicked but before
the job id returned, startJob cleared the targetChanged flag, so a finished
dry-run for the OLD pattern could be shown (card + confirm dialog) while Delete
used the current form values.

Snapshot the request at click time and pass it as the mutation variable, and
derive the job's target signature from that same request. Staleness is now a
comparison of the pinned signature against the current form, immune to the form
changing between the click and the id returning (onMutate/mutationFn run in a
microtask and would otherwise capture the edited value). Drops the boolean flag,
the stored preview state, and the capture effect; the confirm-dialog count now
reads off the live job only when it is a finished dry-run matching the form.

Adds a component test reproducing the in-flight edit and a unit test pinning
which fields define the target signature.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 641ec84. Configure here.

`A bulk-delete job (${active.id}) is already running for connection '${connectionId}'. ` +
`Wait for it to finish or cancel it before starting another.`,
);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In-memory job lock only

Medium Severity

Concurrent bulk-delete protection uses an in-process Map via findActiveJob, so only one running job per connection is enforced on a single API instance. With multiple monitor replicas, two execute requests for the same connection can run overlapping SCAN/UNLINK walks because no shared lock exists.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 641ec84. Configure here.

// Pace only between real work, never after the final batch of a walk.
if (params.batchPauseMs > 0 && cursor !== '0') {
await sleep(params.batchPauseMs);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancel ignored during batch pause

Low Severity

Cooperative cancellation is checked only at the start of each scan iteration, not during batchPauseMs sleeps between batches. After a user cancels, the engine can remain in running for up to the configured pause (up to 60s) before stopping and reporting cancelled.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 641ec84. Configure here.

@KIvanow KIvanow merged commit b854946 into master Jul 8, 2026
3 checks passed
@KIvanow KIvanow deleted the feat/bulk-delete-by-pattern branch July 8, 2026 16:11
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants