Track Vercel request IDs on workflow events for observability#997
Draft
Track Vercel request IDs on workflow events for observability#997
Conversation
🦋 Changeset detectedLatest commit: e37e4fd The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (2 failed)astro (1 failed):
fastify (1 failed):
🌍 Community Worlds (43 failed)turso (43 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end propagation and persistence of Vercel request IDs (x-vercel-id) so workflow/step events can be correlated with request-level logs in observability tooling.
Changes:
- Extends event and queue metadata types to carry an optional
requestId, and threads it through core runtime event creation calls. - Extracts
x-vercel-idin Vercel and local queue handlers and propagates it into queue callback metadata (viaAsyncLocalStoragefor@vercel/queue). - Persists
requestIdon events in Postgres and local file event storage, and forwards it through the Vercel API client.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/queue.ts | Adds requestId?: string to queue handler metadata type. |
| packages/world/src/events.ts | Adds requestId?: string to CreateEventParams and to EventSchema. |
| packages/world-vercel/src/queue.ts | Captures x-vercel-id from incoming requests and propagates via AsyncLocalStorage. |
| packages/world-vercel/src/events.ts | Forwards requestId in v2 event creation requests. |
| packages/world-postgres/src/storage.ts | Writes requestId into event inserts and includes it in returned event objects. |
| packages/world-postgres/src/drizzle/schema.ts | Adds request_id column to the Drizzle events table schema. |
| packages/world-local/src/storage/events-storage.ts | Includes requestId when writing local event JSON. |
| packages/world-local/src/queue.ts | Extracts x-vercel-id and passes it into handler metadata. |
| packages/core/src/runtime/suspension-handler.ts | Propagates requestId into suspension-created events. |
| packages/core/src/runtime/step-handler.ts | Propagates requestId into step lifecycle events. |
| packages/core/src/runtime/start.ts | Adds StartOptions.requestId and passes it into run_created event creation. |
| packages/core/src/runtime.ts | Propagates requestId into workflow lifecycle events and suspension handling. |
| .changeset/request-id-tracking.md | Declares patch releases for the affected packages. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+106
to
+108
| specVersion: integer('spec_version'), | ||
| /** The Vercel request ID (x-vercel-id header) that created this event */ | ||
| requestId: varchar('request_id'), |
There was a problem hiding this comment.
A Drizzle migration needs to be added for this new request_id column. The package’s db:push script runs drizzle-orm migrations (see src/cli.ts), so without a new migrations/*.sql entry that ALTERs workflow.workflow_events to add request_id, installs/upgrades will fail at runtime when inserts include requestId.
packages/world-vercel/src/queue.ts
Outdated
| // Wrap the VQS handler to extract x-vercel-id from the incoming request | ||
| // and propagate it via AsyncLocalStorage into the handler callback | ||
| return async (req: Request) => { | ||
| const requestId = req.headers.get(VERCEL_REQUEST_ID_HEADER) ?? undefined; |
There was a problem hiding this comment.
Header extraction uses req.headers.get(...) ?? undefined, which preserves an empty header value as ''. That then propagates as a (falsy) requestId through metadata and can lead to inconsistent persistence/serialization downstream. Consider normalizing here (e.g., trim and convert '' to undefined) so the rest of the pipeline can treat “missing/blank” consistently.
Suggested change
| const requestId = req.headers.get(VERCEL_REQUEST_ID_HEADER) ?? undefined; | |
| const rawRequestId = req.headers.get(VERCEL_REQUEST_ID_HEADER); | |
| const requestId = rawRequestId?.trim() || undefined; |
packages/world-local/src/queue.ts
Outdated
Comment on lines
+193
to
+207
| // Extract x-vercel-id for request ID tracking (available on Vercel deployments) | ||
| const requestId = req.headers.get('x-vercel-id') ?? undefined; | ||
|
|
||
| if (!queueName.startsWith(prefix)) { | ||
| return Response.json({ error: 'Unhandled queue' }, { status: 400 }); | ||
| } | ||
|
|
||
| const body = await new JsonTransport().deserialize(req.body); | ||
| try { | ||
| const result = await handler(body, { attempt, queueName, messageId }); | ||
| const result = await handler(body, { | ||
| attempt, | ||
| queueName, | ||
| messageId, | ||
| requestId, | ||
| }); |
There was a problem hiding this comment.
Header extraction uses req.headers.get('x-vercel-id') ?? undefined, which keeps an empty header value as ''. Since downstream code often gates on truthiness, this can result in the requestId being dropped from emitted events even though it was “present”. Consider normalizing (trim + convert blank to undefined) before passing it into the handler metadata.
Comment on lines
393
to
397
| eventType: data.eventType, | ||
| eventData: 'eventData' in data ? data.eventData : undefined, | ||
| specVersion: effectiveSpecVersion, | ||
| requestId: params?.requestId ?? null, | ||
| }) |
There was a problem hiding this comment.
This change adds new persisted/returned behavior (requestId stored on events). There are extensive integration tests in packages/world-postgres/test/storage.test.ts, but none currently assert requestId round-trips. Adding at least one test that calls events.create(..., ..., { requestId }) and verifies the returned event (and/or a subsequent list/get) includes the same requestId would prevent regressions.
Comment on lines
+396
to
+406
| requestId: params?.requestId ?? null, | ||
| }) | ||
| .returning({ createdAt: Schema.events.createdAt }); | ||
|
|
||
| const result = { ...data, ...value, runId: effectiveRunId, eventId }; | ||
| const result = { | ||
| ...data, | ||
| ...value, | ||
| runId: effectiveRunId, | ||
| eventId, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), | ||
| }; |
There was a problem hiding this comment.
params?.requestId is written to the DB using ?? null, but the returned event object only includes requestId when it’s truthy (params?.requestId ? ...). This can create inconsistencies (e.g., an empty-string requestId is persisted but omitted from the returned event) and makes behavior dependent on truthiness. Consider normalizing requestId (treat blank as undefined) and using an explicit undefined/null check when deciding whether to include it in the result.
Comment on lines
858
to
878
| specVersion: effectiveSpecVersion, | ||
| requestId: params?.requestId ?? null, | ||
| }) | ||
| .returning({ createdAt: events.createdAt }); | ||
|
|
||
| if (!conflictValue) { | ||
| throw new WorkflowAPIError( | ||
| `Event ${eventId} could not be created`, | ||
| { status: 409 } | ||
| ); | ||
| } | ||
|
|
||
| const conflictResult = { | ||
| eventType: 'hook_conflict' as const, | ||
| correlationId: data.correlationId, | ||
| eventData: conflictEventData, | ||
| ...conflictValue, | ||
| runId: effectiveRunId, | ||
| eventId, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), | ||
| }; |
There was a problem hiding this comment.
Same truthiness issue as other result constructions: requestId is persisted via ?? null, but then only included in the returned conflictResult when params?.requestId is truthy. Prefer an explicit undefined/null check (or normalize blank to undefined earlier) so storage and returned event stay consistent.
| ...value, | ||
| runId: effectiveRunId, | ||
| eventId, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), |
There was a problem hiding this comment.
requestId is always inserted (as null when missing), but the returned result only includes it when truthy. This can drop valid-but-falsy values (e.g., empty string) and makes behavior inconsistent across code paths. Consider normalizing the input and switching to an explicit undefined/null check when adding requestId to the returned event object.
Suggested change
| ...(params?.requestId ? { requestId: params.requestId } : {}), | |
| ...(params?.requestId !== undefined | |
| ? { requestId: params.requestId ?? null } | |
| : {}), |
Comment on lines
136
to
149
| const result = await world.events.create( | ||
| runId, | ||
| { | ||
| eventType: 'run_created', | ||
| specVersion, | ||
| eventData: { | ||
| deploymentId: deploymentId, | ||
| workflowName: workflowName, | ||
| input: workflowArguments, | ||
| executionContext: { traceCarrier, workflowCoreVersion }, | ||
| }, | ||
| }, | ||
| { v1Compat } | ||
| { v1Compat, requestId: opts?.requestId } | ||
| ); |
There was a problem hiding this comment.
StartOptions adds requestId and threads it into world.events.create(...). There are existing unit tests for start() in packages/core/src/runtime/start.test.ts, but none currently assert that options are forwarded into the create-event params. Adding a test that passes requestId and verifies the mocked world.events.create receives it would lock in this behavior.
| eventId, | ||
| createdAt: now, | ||
| specVersion: effectiveSpecVersion, | ||
| ...(params?.requestId ? { requestId: params.requestId } : {}), |
There was a problem hiding this comment.
requestId is added to the stored event only when it’s truthy. If an empty-string requestId slips through (e.g., a present-but-blank header), it will be silently dropped, making behavior differ from “explicitly provided but blank”. Consider normalizing requestId earlier (trim + convert blank to undefined) and/or switching this condition to an explicit undefined/null check for consistency with other storages.
Suggested change
| ...(params?.requestId ? { requestId: params.requestId } : {}), | |
| ...(params?.requestId !== undefined && params?.requestId !== null | |
| ? { requestId: params.requestId } | |
| : {}), |
Collaborator
Author
|
gonna spend a bit more time thinking through this flow properly before re-opening it out of draft |
bd96bc3 to
d011706
Compare
Extract x-vercel-id header and VQS message ID from incoming requests in queue handlers and pass them through metadata. The step handler forwards both IDs via CreateEventParams when creating step_started events. The Vercel world wires these fields to the workflow-server API, which will store them on the step entity for correlating request logs with step execution attempts in observability. Changes: - Add requestId/messageId to queue handler metadata (world, world-vercel, world-local) - Add requestId/messageId to CreateEventParams (world) - Pass both IDs in step_started event creation (core) - Wire requestId/messageId to POST body in Vercel events API (world-vercel) https://claude.ai/code/session_01TqomNcQoixA1HzxvFFBsu2
d011706 to
e37e4fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for tracking Vercel request IDs (
x-vercel-idheader) throughout the workflow execution lifecycle. Request IDs are now captured from incoming HTTP requests and propagated to all workflow and step events, enabling better correlation between request logs and workflow executions in observability systems.Backend PR needed for store these IDs: https://github.com/vercel/workflow-server/pull/251
Key Changes
requestIdparameter to theCreateEventParamsinterface and threaded it through all event creation calls in the workflow runtime, step handler, and suspension handler@vercel/queuehandler wrapper to extract thex-vercel-idheader from incoming requests usingAsyncLocalStoragerequestIdto the queue handler metadata interfacerequestIdfield on eventsrequestIdin event creation requestsrequestIdfield toEventSchemaand documented it in relevant interfacesrequestIdcolumn to PostgreSQL events tableImplementation Details
AsyncLocalStorageto propagate request IDs through the async call chain in the Vercel queue handler, since the@vercel/queuelibrary abstracts away the raw Request objecthttps://claude.ai/code/session_01TqomNcQoixA1HzxvFFBsu2