Postmortem: Missing task logs and traces

Date: 2026-09-03
Duration: ~67 minutes
Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dro...

We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes

A small percentage of streams experienced issues but it's resolved now.

The root cause was some server crashes for our provider which they're working on a permanent fix for now.

The dashboard is now available. If you are still experiencing issues please hard reload the dashboard application.

The issue is now resolved, the dashboard and API are operating normally.

SSO login access has been restored.

Root cause identified and a fix applied. We're provisioning additional ClickHouse resources to prevent this occurring again.