Retry KubernetesPodOperator pod-status reads on a transient 404 - #73106
Closed
hitenshKharva wants to merge 2 commits into
Closed
hitenshKharva wants to merge 2 commits into
hitenshKharva wants to merge 2 commits into
Conversation
A task pod can 404 briefly when a higher-priority daemonset pod (commonly scheduled by node autoscalers like Karpenter on newly created nodes) preempts it before Kubernetes reschedules it elsewhere. Reading pod status during this window previously failed the task outright since 404 was excluded from the operator's existing transient-error retry set, even though the failure is transient infrastructure churn rather than a real problem with the task. Closes: apache#59626 Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SaCoAHZUXWqg6eAX9Godp9
hitenshKharva
requested review from
hussein-awala,
jedcunningham and
jscheffl
as code owners
September 13, 2026 21:01
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
The chart set GIT_SYNC_BRANCH, GIT_SYNC_REV, and GITSYNC_REF unconditionally, using v2-2-stable/HEAD as the legacy defaults for the first two. git-sync v4 gives the legacy branch/rev flags precedence over ref whenever both are present, so a user who only customized dags.gitSync.ref (the documented v4 way to pin a revision) got silently overridden back to the chart's default branch instead - most visibly when node autoscaling or pod rescheduling caused an unexpected checkout of the wrong ref. branch and rev now default to unset and are only emitted when a user opts into them explicitly (e.g. to pin an older git-sync v3 image), so ref is authoritative by default as the git-sync v4 docs intend. Note: the lint-helm-chart and kubeconform prek hooks were skipped for this commit because the sandbox this was authored in blocks get.helm.sh (needed by `breeze k8s setup-env` to provision the helm binary those hooks shell out to). Run them locally or rely on CI before merging. Closes: apache#42918 Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SaCoAHZUXWqg6eAX9Godp9
Author
|
This issue seems to be already fixed. Apologies for the rework and noise |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A task pod can 404 briefly when a higher-priority daemonset pod (commonly scheduled by node autoscalers like Karpenter on newly created nodes) preempts it before Kubernetes reschedules it elsewhere. Reading pod status during this window previously failed the task outright since 404 was excluded from the operator's existing transient-error retry set, even though the failure is transient infrastructure churn rather than a real problem with the task.
Closes: #59626
Claude-Session: https://claude.ai/code/session_01SaCoAHZUXWqg6eAX9Godp9
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.