Fix minio-make-bucket-job lingering after failed helm upgrade - #842
Conversation
The stable/minio chart (v5.0.33) uses old-style Helm labels (release: <name>, app: minio-make-bucket-job) and has hook-delete-policy: hook-succeeded only. When the job fails, it is not deleted by Helm on rollback. clean_jobs was filtering by app.kubernetes.io/instance which doesn't exist on old-style chart resources, so the stuck job was never cleaned up. The second helm upgrade attempt would then fail with "failed to deploy codacy-minio-make-bucket-job" because the job already existed. Add a second kubectl delete with the legacy release=<name> label selector so clean_jobs catches both new- and old-style Helm-labelled jobs. AI-Generated: true
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull Request Overview
The pull request successfully implements the cleanup of lingering Kubernetes jobs by expanding the label selectors to include legacy labels. This addresses a specific failure point in the CI pipeline where failed Helm hooks prevented subsequent upgrades.
Codacy analysis indicates that the changes are up to standards with no new quality issues. However, there are no automated tests provided in this PR to verify the label selector logic or ensure idempotency. While the code changes are straightforward, adopting the suggested shell quoting best practices will improve the robustness of the CI scripts.
Test suggestions
- Verify clean_jobs successfully selects and deletes jobs with the 'release' label matching RELEASE_NAME.
- Verify clean_jobs ignores cases where no jobs match the 'release' label (idempotency).
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify clean_jobs successfully selects and deletes jobs with the 'release' label matching RELEASE_NAME.
2. Verify clean_jobs ignores cases where no jobs match the 'release' label (idempotency).
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| .PHONY: clean_jobs | ||
| clean_jobs: | ||
| -kubectl delete jobs -l app.kubernetes.io/instance=${RELEASE_NAME} --namespace ${NAMESPACE} --ignore-not-found=true | ||
| -kubectl delete jobs -l release=${RELEASE_NAME} --namespace ${NAMESPACE} --ignore-not-found=true |
There was a problem hiding this comment.
⚪ LOW RISK
Suggestion: Wrap the variable expansions in double quotes to ensure the command is robust against special characters in the release name or namespace.
Problem
Job 31121 (
deploy_to_doks_release) failed at the Install Codacy step with:Root cause
The
stable/miniochart (v5.0.33) uses old-style Helm labels (release: <name>,app: minio-make-bucket-job) instead of the modernapp.kubernetes.io/instancelabels. It also setshook-delete-policy: hook-succeeded(notbefore-hook-creation), so a failed minio-make-bucket-job is not deleted by Helm on rollback.The retry flow in
deploy_to_doks_from_chartmuseumrunsclean_jobsbetween attempts, butclean_jobswas only filtering byapp.kubernetes.io/instance=${RELEASE_NAME}— a label that doesn't exist on old-style chart resources. This is why the cleanup step reported "No resources found" despite the stuck job being present.On the second
helm upgradeattempt, Helm tried to create the minio-make-bucket-job as a post-upgrade hook but found it already existed, reporting "failed to deploy".Fix
Add a second
kubectl deleteinclean_jobsusing the legacyrelease=<name>label selector so that both new- and old-style Helm-labelled jobs are cleaned up before a retry.clean_jobs: -kubectl delete jobs -l app.kubernetes.io/instance=${RELEASE_NAME} --namespace ${NAMESPACE} --ignore-not-found=true -kubectl delete jobs -l release=${RELEASE_NAME} --namespace ${NAMESPACE} --ignore-not-found=truehttps://app.circleci.com/agents/gh/codacy/chat/1a4c9694-264f-4dd6-a821-5943532c253a