Long Scheduler iteration times #31594

MPickfield · 2025-08-06T18:53:22Z

MPickfield
Aug 6, 2025

Hello,

We are experiencing some performance issues with the Dagster scheduler. This appears to be a re-emergence of an issue we mitigated by increasing the available resources, but this time we do not seem resource-constrained.

Details

Our daemon scheduler thread is taking ~15 seconds to process an individual run request. We have an hourly schedule that launches ~550 partitions on its busiest tick, meaning that the scheduler takes around 2 hours and 20 minutes just to pass our run requests to the queued run coordinator.

The individual partitions take between 20 minutes and 3 hours to complete.

Here is an example of our long schedule tick evaluation

and logs showing RunRequests are being processed about 15s apart

Resources

The dagster daemon's container seems to have ample resources

Similarly, dagsters database seems to have plenty of resources

Dagster DB resource screenshots

Logs & pyspy dumps

Here are the first 10 minutes of logs from part of the relevant timerange, 3:00AM EDT - 3:10AM
downloaded-logs-20250806-141211.json

And some pyspy dumps taken while this is happening, as suggested in the previous discussion.

PySpy Dump 1

py-spy dump --pid 1
Process 1: /usr/local/bin/python /usr/local/bin/dagster-daemon run -w /dagster-workspace/workspace.yaml
Python v3.10.14 (/usr/local/bin/python3.10)
Thread 1 (idle): "MainThread"
    check_daemon_loop (dagster/_daemon/controller.py:296)
    _daemon_run_command (dagster/_daemon/cli/__init__.py:106)
    wrap (dagster/_core/telemetry.py:168)
    run_command (dagster/_daemon/cli/__init__.py:83)
    invoke (click/core.py:782)
    invoke (click/core.py:1434)
    invoke (click/core.py:1688)
    main (click/core.py:1078)
    __call__ (click/core.py:1157)
    main (dagster/_daemon/cli/__init__.py:178)
    <module> (dagster-daemon:8)
Thread 8 (idle): "grpc-server-registry-cleanup"
    wait (threading.py:324)
    wait (threading.py:607)
    _clear_old_processes (dagster/_core/remote_representation/grpc_server_registry.py:243)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 43 (idle): "dagster-daemon-SENSOR"
    wait (threading.py:324)
    wait (threading.py:607)
    execute_sensor_iteration_loop (dagster/_daemon/sensor.py:289)
    core_loop (dagster/_daemon/daemon.py:329)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 44 (idle): "dagster-daemon-BACKFILL"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/daemon.py:262)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 45 (idle): "dagster-daemon-SCHEDULER"
    wait (threading.py:324)
    wait (threading.py:607)
    execute_scheduler_iteration_loop (dagster/_scheduler/scheduler.py:254)
    core_loop (dagster/_daemon/daemon.py:284)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 46 (idle): "dagster-daemon-QUEUED_RUN_COORDINATOR"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/daemon.py:262)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 47 (active): "dagster-daemon-MONITORING"
    connect (psycopg2/__init__.py:122)
    connect (sqlalchemy/engine/default.py:620)
    connect (sqlalchemy/engine/create.py:643)
    __connect (sqlalchemy/pool/base.py:896)
    __init__ (sqlalchemy/pool/base.py:674)
    _create_connection (sqlalchemy/pool/base.py:390)
    _do_get (sqlalchemy/pool/impl.py:308)
    checkout (sqlalchemy/pool/base.py:712)
    _checkout (sqlalchemy/pool/base.py:1263)
    connect (sqlalchemy/pool/base.py:449)
    raw_connection (sqlalchemy/engine/base.py:3300)
    __init__ (sqlalchemy/engine/base.py:146)
    connect (sqlalchemy/engine/base.py:3276)
    retry_pg_connection_fn (dagster_postgres/utils.py:117)
    create_pg_connection (dagster_postgres/utils.py:165)
    __enter__ (contextlib.py:135)
    get_records_for_run (dagster/_core/storage/event_log/sql_event_log.py:532)
    get_logs_for_run (dagster/_core/storage/event_log/base.py:189)
    all_logs (dagster/_core/instance/__init__.py:1889)
    inner (dagster/_utils/__init__.py:694)
    count_resume_run_attempts (dagster/_daemon/monitoring/run_monitoring.py:98)
    count_resume_run_attempts (dagster/_core/instance/__init__.py:2724)
    check_run_worker_health (dagster_k8s/launcher.py:415)
    monitor_started_run (dagster/_daemon/monitoring/run_monitoring.py:111)
    execute_run_monitoring_iteration (dagster/_daemon/monitoring/run_monitoring.py:187)
    run_iteration (dagster/_daemon/daemon.py:359)
    core_loop (dagster/_daemon/daemon.py:255)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 48 (idle): "dagster-daemon-ASSET"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/asset_daemon.py:427)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 55 (idle): "sensor_daemon_worker_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 56 (idle): "schedule_daemon_worker_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 57 (idle): "schedule_daemon_worker_1"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 58 (idle): "schedule_daemon_worker_2"
    connect (psycopg2/__init__.py:122)
    connect (sqlalchemy/engine/default.py:620)
    connect (sqlalchemy/engine/create.py:643)
    __connect (sqlalchemy/pool/base.py:896)
    __init__ (sqlalchemy/pool/base.py:674)
    _create_connection (sqlalchemy/pool/base.py:390)
    _do_get (sqlalchemy/pool/impl.py:308)
    checkout (sqlalchemy/pool/base.py:712)
    _checkout (sqlalchemy/pool/base.py:1263)
    connect (sqlalchemy/pool/base.py:449)
    raw_connection (sqlalchemy/engine/base.py:3300)
    __init__ (sqlalchemy/engine/base.py:146)
    connect (sqlalchemy/engine/base.py:3276)
    retry_pg_connection_fn (dagster_postgres/utils.py:117)
    create_pg_connection (dagster_postgres/utils.py:165)
    __enter__ (contextlib.py:135)
    store_asset_event (dagster_postgres/event_log/event_log.py:269)
    store_event (dagster_postgres/event_log/event_log.py:198)
    handle_new_event (dagster/_core/instance/__init__.py:2404)
    report_dagster_event (dagster/_core/instance/__init__.py:2502)
    _log_materialization_planned_event_for_asset (dagster/_core/instance/__init__.py:1415)
    _log_asset_planned_events (dagster/_core/instance/__init__.py:1432)
    create_run (dagster/_core/instance/__init__.py:1612)
    _create_scheduler_run (dagster/_scheduler/scheduler.py:967)
    _submit_run_request (dagster/_scheduler/scheduler.py:740)
    <lambda> (dagster/_scheduler/scheduler.py:856)
    _schedule_runs_at_time (dagster/_scheduler/scheduler.py:870)
    launch_scheduled_runs_for_schedule_iterator (dagster/_scheduler/scheduler.py:618)
    launch_scheduled_runs_for_schedule (dagster/_scheduler/scheduler.py:464)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 59 (idle): "schedule_daemon_worker_3"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 4003 (idle): "run_dequeue_worker_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 4004 (idle): "run_dequeue_worker_1"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 34375 (idle): "run_dequeue_worker_2"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 184192 (idle): "grpc-server-watch"
    wait (threading.py:324)
    wait (threading.py:607)
    watch_for_changes (dagster/_grpc/server_watcher.py:89)
    watch_grpc_server_thread (dagster/_grpc/server_watcher.py:119)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

PySpy Dump 2

py-spy dump --pid 1
Process 1: /usr/local/bin/python /usr/local/bin/dagster-daemon run -w /dagster-workspace/workspace.yaml
Python v3.10.14 (/usr/local/bin/python3.10)
Thread 1 (idle): "MainThread"
    check_daemon_loop (dagster/_daemon/controller.py:296)
    _daemon_run_command (dagster/_daemon/cli/__init__.py:106)
    wrap (dagster/_core/telemetry.py:168)
    run_command (dagster/_daemon/cli/__init__.py:83)
    invoke (click/core.py:782)
    invoke (click/core.py:1434)
    invoke (click/core.py:1688)
    main (click/core.py:1078)
    __call__ (click/core.py:1157)
    main (dagster/_daemon/cli/__init__.py:178)
    <module> (dagster-daemon:8)
Thread 8 (idle): "grpc-server-registry-cleanup"
    wait (threading.py:324)
    wait (threading.py:607)
    _clear_old_processes (dagster/_core/remote_representation/grpc_server_registry.py:243)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 43 (idle): "dagster-daemon-SENSOR"
    wait (threading.py:324)
    wait (threading.py:607)
    execute_sensor_iteration_loop (dagster/_daemon/sensor.py:289)
    core_loop (dagster/_daemon/daemon.py:329)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 44 (idle): "dagster-daemon-BACKFILL"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/daemon.py:262)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 45 (idle): "dagster-daemon-SCHEDULER"
    wait (threading.py:324)
    wait (threading.py:607)
    execute_scheduler_iteration_loop (dagster/_scheduler/scheduler.py:254)
    core_loop (dagster/_daemon/daemon.py:284)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 46 (idle): "dagster-daemon-QUEUED_RUN_COORDINATOR"
    wait (threading.py:320)
    wait (threading.py:607)
    as_completed (concurrent/futures/_base.py:245)
    _dequeue_runs_iter_threaded (dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py:147)
    _dequeue_runs_iter (dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py:107)
    run_iteration (dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py:90)
    core_loop (dagster/_daemon/daemon.py:255)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 47 (idle): "dagster-daemon-MONITORING"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/daemon.py:262)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 48 (idle): "dagster-daemon-ASSET"
    wait (threading.py:324)
    wait (threading.py:607)
    core_loop (dagster/_daemon/asset_daemon.py:427)
    run_daemon_loop (dagster/_daemon/daemon.py:121)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 55 (idle): "sensor_daemon_worker_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 56 (idle): "schedule_daemon_worker_0"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 57 (idle): "schedule_daemon_worker_1"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 58 (active): "schedule_daemon_worker_2"
    connect (psycopg2/__init__.py:122)
    connect (sqlalchemy/engine/default.py:620)
    connect (sqlalchemy/engine/create.py:643)
    __connect (sqlalchemy/pool/base.py:896)
    __init__ (sqlalchemy/pool/base.py:674)
    _create_connection (sqlalchemy/pool/base.py:390)
    _do_get (sqlalchemy/pool/impl.py:308)
    checkout (sqlalchemy/pool/base.py:712)
    _checkout (sqlalchemy/pool/base.py:1263)
    connect (sqlalchemy/pool/base.py:449)
    raw_connection (sqlalchemy/engine/base.py:3300)
    __init__ (sqlalchemy/engine/base.py:146)
    connect (sqlalchemy/engine/base.py:3276)
    retry_pg_connection_fn (dagster_postgres/utils.py:117)
    create_pg_connection (dagster_postgres/utils.py:165)
    __enter__ (contextlib.py:135)
    store_event (dagster_postgres/event_log/event_log.py:177)
    handle_new_event (dagster/_core/instance/__init__.py:2404)
    report_dagster_event (dagster/_core/instance/__init__.py:2502)
    _log_materialization_planned_event_for_asset (dagster/_core/instance/__init__.py:1415)
    _log_asset_planned_events (dagster/_core/instance/__init__.py:1432)
    create_run (dagster/_core/instance/__init__.py:1612)
    _create_scheduler_run (dagster/_scheduler/scheduler.py:967)
    _submit_run_request (dagster/_scheduler/scheduler.py:740)
    <lambda> (dagster/_scheduler/scheduler.py:856)
    _schedule_runs_at_time (dagster/_scheduler/scheduler.py:870)
    launch_scheduled_runs_for_schedule_iterator (dagster/_scheduler/scheduler.py:618)
    launch_scheduled_runs_for_schedule (dagster/_scheduler/scheduler.py:464)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 59 (idle): "schedule_daemon_worker_3"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 4003 (active): "run_dequeue_worker_0"
    connect (psycopg2/__init__.py:122)
    connect (sqlalchemy/engine/default.py:620)
    connect (sqlalchemy/engine/create.py:643)
    __connect (sqlalchemy/pool/base.py:896)
    __init__ (sqlalchemy/pool/base.py:674)
    _create_connection (sqlalchemy/pool/base.py:390)
    _do_get (sqlalchemy/pool/impl.py:308)
    checkout (sqlalchemy/pool/base.py:712)
    _checkout (sqlalchemy/pool/base.py:1263)
    connect (sqlalchemy/pool/base.py:449)
    raw_connection (sqlalchemy/engine/base.py:3300)
    __init__ (sqlalchemy/engine/base.py:146)
    connect (sqlalchemy/engine/base.py:3276)
    retry_pg_connection_fn (dagster_postgres/utils.py:117)
    create_pg_connection (dagster_postgres/utils.py:165)
    __enter__ (contextlib.py:135)
    fetchall (dagster/_core/storage/runs/sql_run_storage.py:117)
    _get_run_by_id (dagster/_core/storage/runs/sql_run_storage.py:391)
    add_run_tags (dagster/_core/storage/runs/sql_run_storage.py:467)
    add_run_tags (dagster/_core/instance/__init__.py:1790)
    inner (dagster/_utils/__init__.py:694)
    _launch_k8s_job_with_args (dagster_k8s/launcher.py:230)
    launch_run (dagster_k8s/launcher.py:294)
    _dequeue_run (dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py:383)
    _dequeue_run_thread (dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py:129)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 4004 (idle): "run_dequeue_worker_1"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 34375 (idle): "run_dequeue_worker_2"
    _worker (concurrent/futures/thread.py:81)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 184739 (idle): "grpc-server-watch"
    wait (threading.py:324)
    wait (threading.py:607)
    watch_for_changes (dagster/_grpc/server_watcher.py:89)
    watch_grpc_server_thread (dagster/_grpc/server_watcher.py:119)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

RunRequests

Our actual run requests are fairly lightweight, our schedule looks like this

ACCOUNT_PARTITION_NAME = "account_ids"
account_id_partitions = DynamicPartitionsDefinition(name=ACCOUNT_PARTITION_NAME)


@schedule(job=replicator, cron_schedule="0 * * * *")
def hourly(
    context: ScheduleEvaluationContext,
) -> SkipReason | list[RunRequest]:
    db_accounts: Dict[str, MinimalSnowflakeAccount] = {}
    with UnsafeAllOrgSessionLocal() as unsafe_all_org_session:
        for account in get_all_accounts(unsafe_all_org_session):
            db_accounts[account.uuid] = account

    account_ids = context.instance.get_dynamic_partitions(ACCOUNT_PARTITION_NAME)
    account_ids = [
        account_id
        for account_id in account_ids
        if account_id in db_accounts
        and eligible_to_run(
            db_accounts[account_id].replicator_cron_schedule,
            context.scheduled_execution_time,
        )
    ]
    if not account_ids:
        return SkipReason("No account IDs in DB")
    run_requests: list[RunRequest] = [
        RunRequest(
            partition_key=account_id,
            run_key=f"{account_id}-{context.scheduled_execution_time}",
        )
        for account_id in account_ids
    ]

    logger.info(f"Hourly schedule run requests yielded {len(run_requests)} requests")
    return run_requests


schedule_defs: list[ScheduleDefinition] = [hourly]

Our run coordinator configuration looks like this

    runCoordinator:
      type: QueuedRunCoordinator
      config:
        queuedRunCoordinator:
          maxConcurrentRuns: 200
          # Tag based concurrency limits. See https://docs.dagster.io/deployment/run-coordinator#usage
          # Limits runs to one per partition (preventing multiple customer replicator runs at once)
          tagConcurrencyLimits:
            - key: dagster/partition
              limit: 1
              value:
                applyLimitPerUniqueValue: true

Naively, I would expect that all 550 RunRequests would be queued reasonably quickly, and then started as resources become available to process them.

What is the expected behaviour here? Is there some setting or other metric we should check?

MPickfield · 2025-08-06T19:03:59Z

MPickfield
Aug 6, 2025
Author

Just noticed that in the previous discussion, we were asked to run pyspy record, not pyspy dump! We'll collect that info on its next run and share it here.

0 replies

NiallRees · 2025-08-07T04:47:46Z

NiallRees
Aug 7, 2025

1-2025-08-07T04:42:48Z.json

here is the output of pyspy record. Of note is that all but one of the four daemon works is idle - here is the flamegraph of the third showing the 15 seconds between run requests. Looks like all the time is spent handling events? Not sure what that means exactly/how to action an improvement. Hoping @gibsondan might have a magical solve 🤞 thanks in advance.

0 replies

NiallRees · 2025-08-19T16:31:27Z

NiallRees
Aug 19, 2025

We implemented pgbouncer which has reduced the time between run requests to 4s. That still seems slow but is a good improvement.

We're also noticing that all but 1 of the daemon workers is idle, so we'll look into that next.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Long Scheduler iteration times #31594

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Long Scheduler iteration times #31594

Uh oh!

Uh oh!

MPickfield Aug 6, 2025

Details

Resources

Logs & pyspy dumps

RunRequests

Replies: 3 comments

Uh oh!

MPickfield Aug 6, 2025 Author

Uh oh!

Uh oh!

NiallRees Aug 7, 2025

Uh oh!

NiallRees Aug 19, 2025

MPickfield
Aug 6, 2025

MPickfield
Aug 6, 2025
Author

NiallRees
Aug 7, 2025

NiallRees
Aug 19, 2025