Add best-effort cleanup to EmrCreateJobFlowOperator on post-creation failure#61010
Merged
shahar1 merged 2 commits intoFeb 10, 2026
Merged
Conversation
Contributor
Author
2 tasks
Contributor
Author
vincbeck
approved these changes
Jan 26, 2026
uranusjr
approved these changes
Jan 27, 2026
9968272 to
e494b8b
Compare
Closed
2 tasks
Attempt best-effort termination of EMR clusters when failures occur after successful job flow creation. Cleanup does not mask the original exception.
e494b8b to
3cab55f
Compare
vincbeck
approved these changes
Jan 29, 2026
Contributor
|
@SameerMesiah97 Could you please resolve conflicts? Edit: I took care of it so I could include it in the upcoming release (used Gen. AI, GitHub Copilot + Claude Sonnet 4.6) |
# Conflicts: # providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py # providers/amazon/tests/unit/amazon/aws/operators/test_emr_create_job_flow.py
Contributor
Author
So you no longer want me to fix it? |
Contributor
No need, I took care of it as I'm starting the release process very soon :) |
81 tasks
Alok-kumar-priyadarshi
pushed a commit
to Alok-kumar-priyadarshi/airflow
that referenced
this pull request
Feb 11, 2026
Ratasa143
pushed a commit
to Ratasa143/airflow
that referenced
this pull request
Feb 15, 2026
choo121600
pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
AkshayArali
pushed a commit
to AkshayArali/airflow_630
that referenced
this pull request
Feb 27, 2026
AkshayArali
pushed a commit
to AkshayArali/airflow_630
that referenced
this pull request
Feb 27, 2026
55 tasks
Subham-KRLX
pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
dominikhei
pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
Ankurdeewan
pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
radhwene
pushed a commit
to radhwene/airflow
that referenced
this pull request
Mar 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Added best-effort cleanup to
EmrCreateJobFlowOperatorto terminate EMR clusters when failures occur after successful cluster creation. Cleanup behavior is guarded by a flag and is opted in by default.In certain failure modes, the operator could previously create a cluster via
create_job_flowand then fail during later execution steps (for example, while waiting for completion whenDescribeClusterpermissions are missing). In these cases, the task failed while leaving the cluster running. The operator now attempts to terminate the created job flow if an exception is raised after creation. Cleanup is best-effort and does not override or mask the original exception.This change applies a similar failure-handling approach recently introduced for
EC2CreateInstanceOperatorin PR #60904. But cleanup is only triggered for post-start EMR job flow failures (including waiter-related errors), ensuring termination is attempted only when a job flow was successfully created and avoiding interception of non-AWS exceptions.Rationale
EmrCreateJobFlowOperatoris responsible for provisioning and coordinating an external, stateful service whose lifecycle extends beyond task execution. If the task fails after cluster creation, Airflow can no longer reliably manage or observe the cluster’s state. Adding opportunistic cleanup in these scenarios reduces the risk of orphaned EMR clusters and unexpected infrastructure costs, while preserving existing failure semantics. Cleanup errors are logged and do not affect the task’s final failure state.Restricting cleanup to post-creation EMR job flow failures prevents unintended termination in unrelated failure paths while still addressing orphaned job flows created during execution.
Tests
Documentation
The docstring for
EmrCreateJobFlowOperatorhas been updated with a brief description of the new flagterminate_job_flow_on_failure.Backwards Compatibility
A new flag called
terminate_job_flow_on_failurehas been added toEmrCreateJobFlowOperatorwith a default setting ofTrue. Cleanup will now be attempted on a best-effort basis ifWaiterErroris encountered.Reproduciblity
The failure scenario could not be reproduced directly due to personal AWS account permissions. However, based on the current control flow of
EmrCreateJobFlowOperator, it is possible for cluster creation to succeed while a later step fails, leaving the EMR cluster running without cleanup. This change defensively addresses that case. Contributors reading this PR are free to provide a reproduction for the aforementioned failure mode if they can.