feat: implement DagTaskGroupsExistence and DagTasksExistence endpoints - #67832
feat: implement DagTaskGroupsExistence and DagTasksExistence endpoints#67832dzpan0 wants to merge 2 commits into
Conversation
d17ea3d to
6b834a8
Compare
Adds two GET endpoints to the Execution API for batched existence checks against a Dag's tasks and task groups. Each takes a list of ids and returns them partitioned into 'existing' and 'missing' with 200 or 404 only when the Dag is missing. Passing an empty list works as a Dag existence probe. These allow the clients to get the actual information of a Dag, returning correct information even when the Dag hasn’t been ran once related: apache#40745. Co-authored-by: Diogo Callado <[email protected]>
The new existence check and tests for ExternalTaskSensor uses endpoints that were newly added, so these will be excluded for lower versions.
6b834a8 to
4ce2687
Compare
|
Hi @ashb @kaxil @amoghrajesh. Would appreciate a review when you have a chance. |
|
That should be some kind of HEAD request I think. But more importantly, I don't know why we need a separate request for this. We make a normal request to get the task status don't we? So couldn't we make a 404 from the get status request encode that info already? |
Dag Versioning makes this problematic. Until a run exists, there is no gurantee that the task will exist in any given run. |
I thought making the endpoint separate the given tasks into two lists would allow more liberty and diverse use cases than just checking weather the tasks exist. On the other note, I thought using
Yeah, that makes sense. I'll close this for now and have another think about it. |
related: #40745
Description
Follow-up to #64394, which fixed
ExternalTaskSensor'scheck_existence=Truefor the AF2. On AF3, the deferrable path still defers silently when the external Dag, task, or task group does not exist, because the existence check has no access to the metadata database.This PR adds two Execution API endpoints that let the sensor verify external references in a single round trip, and wires the sensor to use them before deferring:
GET /dags/{dag_id}/tasks/existence?task_ids=...GET /dags/{dag_id}/task-groups/existence?task_group_ids=...Each returns its input partitioned into
existingandmissing. Passing an emptytask_idslist works as a Dag-existence probe (a missing Dag surfaces asDAG_NOT_FOUND, which the sensor translates toExternalDagNotFoundError). Missing tasks or task groups surface asExternalTaskNotFoundErrorandExternalTaskGroupNotFoundError, respectively, matching the AF2 path's behavior.The Task SDK exposes them via two new
RuntimeTaskInstanceaccessors:get_dag_tasks_existenceandget_dag_task_groups_existence.Why
Existing dag-level Execution API routes can't distinguish a Dag that doesn't exist from a Dag that exists, but has no DagRuns:
GET /dag_runs/countreturns 0 andGET /dag_runs/previousreturns None in both cases.GET /dags/{dag_id}settles the existence question on its own, but says nothing about specific tasks or task groups. These new endpoints fill that gap by combining the existence check with a per-id partition.These endpoints can answer that cheaply and explicitly, which is useful for any future operator or sensor that needs to validate references against an external dag before deferring or scheduling work.
Testing
Unit tests added across the API route, datamodels, SDK client, supervisor handler, runtime accessor and sensor layers.
Was generative AI tooling used to co-author this PR?
Claude (Opus 4.7) following the project guidelines.
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.