Should we account for function execution time limits specifically in the case of Vercel world? If the serverless fuction is already close to the end of it's limit and the workflow server throws a 429, adding a 10 sec sleep could potentially exceed the function execution limit and the function could get SIGKILLd midway.

The workflow layer should never take much more than a few seconds, so I think it's highly unlikely that we'd run into timeouts, so I'm not too worried about this, but technically a concern

The comment says "Safe to retry the entire handler because 429 is sent from server middleware before the request is processed — no server state has changed."

This holds if the 429 always hits on the first API call in the handler. But the handler makes multiple world API calls sequentially (e.g. runs.get → events.create(run_started) → replay → events.create(run_completed)). If a later call gets 429'd, the retry re-executes everything from the top.

For the workflow handler this is probably fine since replay is deterministic and events are idempotent.

For the step handler this is more concerning — the retry would re-execute user step code, which may not be idempotent. Is the assumption that the workflow-server's rate limiting middleware rejects at the connection level (so all calls in one handler invocation either succeed or all fail)? If so, worth documenting that assumption.

Good catch, fixed

The third step is 7200 seconds = 2 hours. Is that intentional? That's a very long time for a workflow to be stalled on a server error. If the 500 was truly transient, 2 hours feels excessive. If it's a sustained outage, you'd probably want to fail rather than retry after 2h when the server might have moved on.

Maybe something like [5, 30, 120] (5s, 30s, 2min) would be more practical?

Changed to [5, 30, 120]. Realistically if we keep getting 500s, it'd be nice to re-try the run much later so we have a change to incident-mitigate, but I think it's fine to fail runs instead too

cool. this was fully AI generated review btw. idk where it came up with 5s, 30, 120 - but I do agree i we should have some shorter ones before getting to 7200 anyway

The step handler gets withThrottleRetry for 429s but no equivalent 5xx retry logic like the workflow handler has. If a world API call (e.g. events.create for step_started or step_completed) returns a 5xx, it'll be caught by the step's error handling and burn a step retry attempt — re-executing potentially expensive user code for what was a transient infrastructure error.

Is this intentional for this PR's scope, or should step handlers also get 5xx backoff retry?

500s are handled via regular step re-try. 429s are only special-cased because they allow the step-retry mechanism to use the retry-after header instead of guessing at the retry

right but the comment is about not burning a full step attempt for transient 5xx. doesn't need to be addressed in this PR necessarily but good to think about. happy to move in any direction for now since they're all better than nothing and then come back to this in the future with real experience

Makes sense that steps rely on their existing retry mechanism. One concern though: with withThrottleRetry removed, a 429 from the world API during step execution (e.g. on step_started or step_completed) will now burn a step retry attempt — potentially exhausting maxRetries without the user's code ever being at fault.

Ideally 429s should be retried transparently without consuming an attempt. A targeted retry around just the world API calls (not wrapping the user code execution) could thread that needle. Worth tracking as a follow-up.

Good that this was promoted from the vercel-specific type to the shared interface. Note that world-local currently ignores delaySeconds entirely — so 5xx retries in local dev will fire immediately instead of with backoff. Not a blocker but worth a follow-up or at minimum a comment.

Seems fine to ignore for local world, since there are no 429s

although we probably do actually want to support delaySeconds in local world anyway as we start using this option in queue more often in the future for more use cases. even if it's not implemented in this PR, I think we should leave an explicit comment that local world ignores this since it's a nuance. We should later have an e2e test that checks for this behaviour and would fail on local world without proper queue delaySeconds implementation (cc @TooTallNate )

Redundant check: this is already inside if (WorkflowAPIError.is(err)), so the second WorkflowAPIError.is(err) on this line is always true. Should just be if (err.status === 429).

The step handler has 429 handling here inside the step_started catch block, but unlike the workflow handler, it does not use withThrottleRetry and does not have the short-wait in-process retry path. This means a brief 429 (e.g., retryAfter=2s) will always defer to the queue rather than sleeping in-process. Is that intentional? If steps should also get the in-process retry for short waits, consider wrapping with withThrottleRetry similarly to the workflow handler.

Nit: if retryAfter is undefined (header missing), this defaults to 1 second. For 429 responses without a Retry-After header, 1 second might be too aggressive. A slightly higher default (e.g., 3-5s) would be more conservative and reduce the chance of hitting the server again immediately while it's under load.

The entire workflow handler body is now wrapped in withThrottleRetry. This means if a 429 is thrown by any world call during workflow execution (e.g., world.runs.get, world.events.create, runWorkflow), the entire handler will be retried from scratch.

1. If run_started event was already created successfully and then a later call throws 429, the retry will call world.runs.get again and find the run already running, which is handled correctly.
2. However, if run_completed event creation throws 429, the retry would re-run the entire workflow (replay + execution). The workflow replay should be deterministic, but this could be expensive. Worth documenting this tradeoff in a comment.

The 500 retry re-enqueues with a new requestedAt and incremented serverErrorRetryCount. The traceCarrier is serialized fresh via serializeTraceCarrier(). If the trace context matters for correlating retries, this creates a new trace for each retry attempt. Consider whether preserving the original traceContext from the message would be preferable for observability (linking all retry attempts to the same parent trace).

Note: the HTTP Retry-After header can also be an HTTP-date (e.g., Wed, 21 Oct 2015 07:28:00 GMT). The parseInt approach correctly returns NaN for date values, falling back to undefined. This is fine since the workflow-server likely only sends numeric values, but a comment noting this would be helpful.