Part of #3654.

Background

magda-minion-framework powers TypeScript metadata enhancement minions exposed through @magda/minion-sdk. The public minion model is intentionally generic: a minion listens to registry record changes, derives metadata, and writes results back to registry, usually as one or more owned aspects.

Currently the framework registers webhooks for create and patch style events and invokes onRecordFound(record, registry) for records included in the webhook payload. It does not subscribe to DeleteRecord events by default and does not expose a common deletion callback.

This means metadata minions have no shared way to clean up derived output when source registry records are deleted. It also means any future delete handling needs to avoid the stale-delete race where a delayed old delete event removes output for a record that has already been recreated.

Problem

A minion can be behind the webhook event stream:

1. Record A exists and a minion has written derived output for it.
2. Record A is deleted and a DeleteRecord event is queued.
3. Record A is recreated before the minion catches up.
4. The minion eventually processes the old delete event.
5. Blindly deleting derived output would be wrong if the latest record is valid and in scope.

At the same time, simply skipping every delete event when the record exists is not enough. The recreated record might exist but no longer match the minion's scope. In that case, the minion may still need to clean up output it owns.

Proposed design

Add opt-in delete-event support to magda-minion-framework.

Proposed MinionOptions additions:

type LatestRecordStatus = "exists" | "notFound";

type DeleteDecision = "processDelete" | "skipDelete";

type ShouldProcessDeleteEvent = (params: {
    event: RegistryEvent;
    recordId: string;
    tenantId: number;
    latestRecordStatus: LatestRecordStatus;
    latestRecord?: Record;
    registry: AuthorizedRegistryClient;
}) => Promise<DeleteDecision> | DeleteDecision;

type OnRecordDeleted = (params: {
    event: RegistryEvent;
    recordId: string;
    tenantId: number;
    latestRecordStatus: LatestRecordStatus;
    latestRecord?: Record;
    registry: AuthorizedRegistryClient;
}) => Promise<void>;

Extend options with:

onRecordDeleted?: OnRecordDeleted;
shouldProcessDeleteEvent?: ShouldProcessDeleteEvent;
handleDeleteEvents?: boolean;
deleteConcurrency?: number;

Default behavior must remain unchanged:

• Existing minions do not subscribe to delete events.
• Existing minions keep current includeEvents behavior unless explicitly configured otherwise.
• Existing onRecordFound(record, registry) behavior is unchanged.

Delete handling should be enabled only when handleDeleteEvents, onRecordDeleted, or shouldProcessDeleteEvent is supplied.

When enabled:

• webhook registration includes DeleteRecord
• webhook config sets includeEvents: true
• setupWebhookEndpoint continues to process payload.records through onRecordFound
• setupWebhookEndpoint also extracts DeleteRecord events from payload.events
• delete events are deduplicated by tenantId + recordId
• latest registry state is fetched before deciding whether to call onRecordDeleted

Default decision:

• latest record not found -> processDelete
• latest record exists -> skipDelete

Advanced minions can override this via shouldProcessDeleteEvent.

Why the framework should not infer cleanup from watched aspects

A minion's aspects and optionalAspects determine webhook relevance, but they do not fully describe the minion's output ownership or cleanup semantics.

A minion may:

• monitor one aspect and write another
• inspect optional aspects or external resources
• write no output for some matching records
• aggregate output onto a parent record
• write multiple owned aspects
• intentionally preserve output after input disappears

Therefore, the framework should expose latest-state context and callback hooks, but the minion must explicitly decide what owned output to clean up.

Error handling

• 404 latest-record lookup means the record is absent and delete processing can proceed.
• 5xx/network/transient lookup failures should fail webhook processing so registry retries.
• malformed delete events without recordId should fail webhook processing.
• errors from shouldProcessDeleteEvent or onRecordDeleted should fail webhook processing.
• async webhook acknowledgement should report success only after all record and delete work succeeds.

Acceptance criteria

• Existing minion tests pass unchanged when delete options are absent.
• Default webhook registration does not include DeleteRecord.
• Opt-in webhook registration includes DeleteRecord and includeEvents: true.
• Existing onRecordFound behavior remains unchanged.
• Delete events are deduplicated by tenantId + recordId.
• Latest-state 404 calls onRecordDeleted by default.
• Latest-state existing record skips delete by default.
• Custom shouldProcessDeleteEvent can process delete even when latest record exists.
• Transient latest-state lookup failure prevents webhook acknowledgement.
• onRecordDeleted failure prevents webhook acknowledgement.
• @magda/minion-sdk exports any new public types needed by minion authors.

Relevant code

• magda-minion-framework/src/MinionOptions.ts
• magda-minion-framework/src/registerWebhook.ts
• magda-minion-framework/src/buildWebhookConfig.ts
• magda-minion-framework/src/setupWebhookEndpoint.ts
• packages/minion-sdk/src/index.ts
• docs/docs/how-to-build-your-own-connectors-minions.md