feat(bulk-delete): incremental delete-by-pattern (client-side SCANDEL)#308
Conversation
Adds a safe, client-driven bulk key deletion by pattern — the SCANDEL use case from valkey/valkey#2623, which upstream deferred in favour of Lua. The monitor drives SCAN + UNLINK itself, which gives dry-run, COUNT-style pacing, cancellation, cluster fan-out and an audit trail without a dedicated server command. Backend (proprietary/bulk-delete): - Pure engine over a ScanTarget abstraction: MATCH/COUNT[/TYPE] walk with dry-run/execute modes, mandatory-pattern and catch-all guards, a maxKeys cap, batch pacing and cooperative cancel. - Deletes are issued as a pipeline of per-key UNLINKs rather than one variadic command, so a batch spanning multiple hash slots does not hit CROSSSLOT on a cluster node. - Preview and execute both run as bounded background jobs (polled via GET /jobs/:id) so a large/sparse keyspace can't block a request; jobs are connection-scoped and cancellable. - Cluster fan-out skips unreachable primaries and records them instead of failing the whole run. - Gated behind a new Pro `bulkDelete` feature and a required connection header. One audit row per execute run across sqlite/postgres/memory; interrupted (crash-left) runs are reconciled to failed on startup. Web: - Bulk Delete page: pattern/type/scope/count/pause/maxKeys inputs, preview with sample keys, typed DELETE confirmation, live progress with cancel, skipped-node warnings and a recent-runs table. Tests: engine, scan-target (per-key UNLINK), service (scoping, cluster fan-out, node-skip, startup sweep) and the web api client.
The Preview card rendered from the live polled dry-run job, while only the confirm-dialog `preview` state was cleared when match/type/scope/ maxKeys changed. So after a dry-run finished, editing the target left a stale card showing counts and sample keys for the previous target next to the edited form. Gate the card on a `previewStale` flag that is set when the target changes and cleared when a fresh preview starts.
For cluster scope, if every discovered primary failed to connect, buildTargets returned an empty target list and runJob still ran the engine over zero nodes and marked the job 'completed' — reporting success for a run that scanned and deleted nothing. Empty targets only occurs in that all-unreachable case (node scope and the no-primaries fallback always yield one target), so treat it as a failure with an informative error; the skipped-node list is still recorded.
…allback Addresses three review findings: - GET /audits clamped `limit` to 500 but not to a minimum of 1, so a negative value reached storage (LIMIT -1 means "no limit" in SQLite, errors in Postgres). Clamp to [1, 500]. - Execute jobs fired the initial "running" audit write with `void` and wrote the final row in the finally block; a slow initial write could land after the final one and revert the row to running/0/null. Keep the initial write's promise and await it before the final write so the completed row always wins. - Cluster scope threw and failed the whole job when discoverNodes errored on a non-cluster connection (CLUSTER NODES unsupported). Tolerate the error and fall back to the single connected node, matching the existing no-primaries behaviour.
…et edit The previewStale flag hid the preview card whenever the target changed, but the card also carries a running dry-run's live progress and Cancel control — and the target inputs stay editable during a run. Editing them mid-run therefore hid the progress card and Cancel while the server job kept going. Only hide a *finished* preview once the target changed; a running preview always renders. Extracted showPreviewCard() and unit tested it.
… too Two review findings: - The web sends the same body (including batchPauseMs) for preview and execute, but batchPauseMs was declared only on the execute DTO. Under the global forbidNonWhitelisted pipe, a preview with a pause value returned 400. Move batchPauseMs onto the shared preview DTO so both accept it (and the dry-run walk honours it). - The run-progress card was only gated on job.mode, so a finished execute run stayed on screen for an old pattern after the target was edited. Generalise the staleness rule (extracted as jobCardVisible) to both cards: a running job always shows; a finished card hides once the target changes. The flag now resets whenever any job starts.
startJob launched a new background run on every call without checking whether the same connection already had one in flight, so repeated preview/execute API calls could stack overlapping SCAN/UNLINK walks against one node. Guard startJob with findActiveJob(): if the connection already has a running job, throw ConflictException (409) naming it and asking the caller to wait or cancel first. The check-then-register path is synchronous, so two rapid calls can't both pass the guard. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01KiqY4sKyXLfBG9rhgh1Zj7
…e-preview race If the match/type/scope/maxKeys changed after Preview was clicked but before the job id returned, startJob cleared the targetChanged flag, so a finished dry-run for the OLD pattern could be shown (card + confirm dialog) while Delete used the current form values. Snapshot the request at click time and pass it as the mutation variable, and derive the job's target signature from that same request. Staleness is now a comparison of the pinned signature against the current form, immune to the form changing between the click and the id returning (onMutate/mutationFn run in a microtask and would otherwise capture the edited value). Drops the boolean flag, the stored preview state, and the capture effect; the confirm-dialog count now reads off the live job only when it is a finished dry-run matching the form. Adds a component test reproducing the in-flight edit and a unit test pinning which fields define the target signature.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 641ec84. Configure here.
| `A bulk-delete job (${active.id}) is already running for connection '${connectionId}'. ` + | ||
| `Wait for it to finish or cancel it before starting another.`, | ||
| ); | ||
| } |
There was a problem hiding this comment.
In-memory job lock only
Medium Severity
Concurrent bulk-delete protection uses an in-process Map via findActiveJob, so only one running job per connection is enforced on a single API instance. With multiple monitor replicas, two execute requests for the same connection can run overlapping SCAN/UNLINK walks because no shared lock exists.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 641ec84. Configure here.
| // Pace only between real work, never after the final batch of a walk. | ||
| if (params.batchPauseMs > 0 && cursor !== '0') { | ||
| await sleep(params.batchPauseMs); | ||
| } |
There was a problem hiding this comment.
Cancel ignored during batch pause
Low Severity
Cooperative cancellation is checked only at the start of each scan iteration, not during batchPauseMs sleeps between batches. After a user cancels, the engine can remain in running for up to the configured pause (up to 60s) before stopping and reporting cancelled.
Reviewed by Cursor Bugbot for commit 641ec84. Configure here.


Summary
Adds a safe, client-driven bulk key deletion by pattern — the SCANDEL use case from valkey/valkey#2623. Upstream deferred the dedicated command in favour of Lua, so the monitor drives
SCAN+UNLINKitself. That gives us dry-run,COUNT-style pacing, cancellation, cluster fan-out and an audit trail today — non-blocking and observable, which a one-shot LuaEVALcan't be.Pro-tier, gated behind a new
bulkDeletefeature and a required connection header.Changes
proprietary/bulk-delete): pureSCAN(MATCH/COUNT/TYPE) +UNLINKwalk over aScanTargetabstraction. Safety rails: mandatory pattern, catch-all (*) requires explicit confirmation,maxKeyscap withtruncatedreporting, inter-batch pacing, cooperative cancel.UNLINKs, not one variadic command — a SCAN batch spans many hash slots and a multi-key command would otherwise hitCROSSSLOTon a cluster node.GET /bulk-delete/jobs/:id, so a large/sparse keyspace can't block a request. Jobs are connection-scoped and cancellable.POST /preview,POST /execute(202 + jobId),GET /jobs/:id,POST /jobs/:id/cancel,GET /audits.bulk_delete_audits) across sqlite/postgres/memory; runs leftrunningby a crashed/restarted process are reconciled tofailedon startup.DELETEconfirmation, live progress with cancel, skipped-node warnings, recent-runs table.Checklist
roborev review --branchor/roborev-review-branchin Claude Code (internal)Notes
bulk_delete_auditsis new in this PR, so no migration is needed for the addedskipped_nodescolumn.runningaudit rows failed; in a multi-instance Postgres deployment this would also touch another instance's in-flight runs (fine for self-hosted / sqlite).Note
High Risk
Irreversible key deletion against live Valkey/Redis with cluster fan-out; mistakes or overly broad patterns can cause data loss despite preview and confirmations.
Overview
Adds a Pro-tier bulk delete by pattern capability: the monitor runs incremental
SCAN+UNLINK(dry-run preview and execute) instead of a one-shot server command, with license gating via newbulkDelete/Feature.BULK_DELETE.The proprietary
BulkDeleteModuleexposes async jobs (POST /preview,POST /execute→ 202 + job id), polling, cancel, and audit listing—all connection-scoped and behindLicenseGuard. The engine validates patterns (catch-all*needsconfirmDeleteAll), caps preview size, supports pacing/cancel, fans out to cluster primaries (skips unreachable nodes), and deletes via per-key pipelinedUNLINKto avoid clusterCROSSSLOT. One concurrent job per connection is enforced (409 on overlap).Storage gains
bulk_delete_audits(sqlite/postgres/memory) with upserted execute-run rows and startup reconciliation of stalerunningrows tofailed.The web adds a Bulk Delete page (pattern/type/scope, preview samples, typed
DELETEconfirm, live progress/cancel, audit table) plus API client and tests for stale-preview UX when the form changes mid-request.Reviewed by Cursor Bugbot for commit 641ec84. Bugbot is set up for automated code reviews on this repo. Configure here.