BugFix: scale-up workload stuck pending not triggering preemption. #6973

ichekrygin · 2025-09-23T23:39:30Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes bug when scaled-up workload stuck pending not triggering preemption.

Which issue(s) this PR fixes:

Fixes #6969

Special notes for your reviewer:

Does this PR introduce a user-facing change?

ElasticJobs: workloads correctly trigger workload preemption in response to a scale-up event.

…bsViaWorkloadSlices and PartialAdmission These two features are not designed to work together. While admission webhooks already prevent `PartialAdmission` from being combined with `ElasticJobsViaWorkloadSlices`, we add this safety check in the scheduler as well to guard against misconfiguration. This ensures that both features cannot be enabled at the same time, avoiding unexpected scheduling behavior. Signed-off-by: Illya Chekrygin <[email protected]>

…estFor Ensures that when a workload slice is replaced, its resource requests are properly reflected in `TotalRequestFor` during flavor assignment and subsequent preemption targets list generation. Signed-off-by: Illya Chekrygin <[email protected]>

netlify · 2025-09-23T23:39:36Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`e443d93`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/68d4212d0b3ba2000985aa8f

ichekrygin · 2025-09-24T04:22:48Z

/test pull-kueue-test-integration-baseline-main
/test pull-kueue-test-integration-extended-main

tenzen-y

I didn't check all, but left a few comments

tenzen-y · 2025-09-24T13:33:31Z

test/integration/singlecluster/scheduler/suite_test.go

 	}
 	cfg = fwk.Init()
 	ctx, k8sClient = fwk.SetupClient(cfg)
+	gomega.Expect(utilfeature.DefaultMutableFeatureGate.SetFromMap(map[string]bool{string(features.ElasticJobsViaWorkloadSlices): true})).Should(gomega.Succeed())


We should avoid enabling such Alpha feature at global level. We should set the FG only in the affected case.
You can learn HOW in

kueue/test/integration/multikueue/dispatcher_test.go

Line 169 in c87e0c7

features.SetFeatureGateDuringTest(ginkgo.GinkgoTB(), features.MultiKueueBatchJobWithManagedBy, true)

Ah... yes. Good catch! For some reason I thought I need "feature" activation at the manager level.
Reverted.

mimowo · 2025-09-24T13:37:33Z

pkg/scheduler/flavorassigner/flavorassigner.go

+		if a.replaceWorkloadSlice != nil {
+			ps = *ps.ScaledTo(ps.Count - a.replaceWorkloadSlice.TotalRequests[i].Count)
+		} else {
+			aps := a.PodSets[i]
+			if aps.Count != ps.Count {
+				ps = *ps.ScaledTo(aps.Count)
+			}
 		}


Instead of doing two independent branches could we just decrement the value we are scaling?

Something like:

newCount := a.PodSets[i].Count if a.replaceWorkloadSlice != nil { newCount -= a.replaceWorkloadSlice.TotalRequests[i].Count } ps = *ps.ScaledTo(newCount)

wdyt?

mimowo · 2025-09-24T13:40:16Z

test/integration/singlecluster/scheduler/preemption_test.go

+			util.ExpectObjectToBeDeleted(ctx, k8sClient, cq, true)
+		})
+
+		ginkgo.It("Should preempt on-create Workloads with lower priority when there is not enough quota", func() {


Does this "It" introduce additional coverage?

It’s updated (PTAL), with feature activation moved into the test-case scope.
This test is dedicated to validating that general workload preemption works — specifically, that an elastic workload can preempt another job outside of the scale-up context. In other words, it serves more as a sanity check.

Happy to remove it if we think it’s redundant.

I see the motivation. OTOH we already have a test exercising priority based preemption (in the simple case) for ElasticJobs here:

kueue/test/integration/singlecluster/controller/jobs/job/job_controller_test.go

Lines 3491 to 3498 in 785114f

highPriorityJob := testingjob.MakeJob("high", ns.Name).

SetAnnotation(workloadslicing.EnabledAnnotationKey, workloadslicing.EnabledAnnotationValue).

Queue(kueue.LocalQueueName(localQueue.Name)).

Request(corev1.ResourceCPU, "1000m").

Parallelism(3).

Completions(3).

WorkloadPriorityClass(highPriorityClass.Name).

Obj()

I know the pre-existing test is a bit more involving, but I'm unclear adding the new small test makes it easier to understand.

I would focus only on the issue at hand and drop the extra test. I would be happy to keep it if we haven't had the other preexisting test already.

mimowo · 2025-09-24T13:51:42Z

pkg/scheduler/scheduler.go

 	}

-	if features.Enabled(features.PartialAdmission) && wl.CanBePartiallyAdmitted() {
+	if features.Enabled(features.PartialAdmission) && wl.CanBePartiallyAdmitted() && replaceableWorkloadSlice == nil {


Do we have some test for this? IIUC the new integration tests (or the reported issue) also show up with the "PartialAdmission" disabled? So, I think this code change may not be exercised by the integration tests

So, IIUC this code change is fixing a different issue. Also, I'm wondering if we shouldn't just disable PartialAdmission for ElasticJobs at the validation level.

Let me know if I'm missing something.

So, IIUC this code change is fixing a different issue. Also, I'm wondering if we shouldn't just disable PartialAdmission for ElasticJobs at the validation level.

We already enforce that at the validation level. The purpose of this change is to add a sanity check, ensuring that PartialAdmission (PA) is never combined with ElasticJobs (EJ). It also serves as an explicit reminder at the scheduler level that PA is not compatible with EJ.

Please let me know if this addition is a "block", I am happy to revert this change.

Let's cleanup the code. We can rely on validation, this is the common practice in k8s, to avoid checking against invariants which are satisfied by validation.

…case scoped feature activation. Signed-off-by: Illya Chekrygin <[email protected]>

Signed-off-by: Illya Chekrygin <[email protected]>

mimowo · 2025-09-24T16:54:32Z

Thank you 👍
/lgtm
/approve
/cherry-pick release-0.13

k8s-infra-cherrypick-robot · 2025-09-24T16:54:36Z

@mimowo: once the present PR merges, I will cherry-pick it on top of release-0.13 in a new PR and assign it to you.

In response to this:

Thank you 👍
/lgtm
/approve
/cherry-pick release-0.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-09-24T16:54:40Z

LGTM label has been added.

Git tree hash: 52172b576dc766e81a53b27176d90af8834b2e91

k8s-ci-robot · 2025-09-24T16:54:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ichekrygin, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-cherrypick-robot · 2025-09-24T17:14:53Z

@mimowo: #6973 failed to apply on top of branch "release-0.13":

Applying: Scheduler: add safety check to ensure mutual exclusivity of ElasticJobsViaWorkloadSlices and PartialAdmission
Applying: Fix: account for replaced workload slice in assigned flavor TotalRequestFor
Using index info to reconstruct a base tree...
M	pkg/scheduler/flavorassigner/flavorassigner.go
M	pkg/scheduler/flavorassigner/flavorassigner_test.go
M	test/integration/singlecluster/scheduler/preemption_test.go
M	test/integration/singlecluster/scheduler/suite_test.go
Falling back to patching base and 3-way merge...
Auto-merging test/integration/singlecluster/scheduler/suite_test.go
CONFLICT (content): Merge conflict in test/integration/singlecluster/scheduler/suite_test.go
Auto-merging test/integration/singlecluster/scheduler/preemption_test.go
Auto-merging pkg/scheduler/flavorassigner/flavorassigner_test.go
Auto-merging pkg/scheduler/flavorassigner/flavorassigner.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0002 Fix: account for replaced workload slice in assigned flavor TotalRequestFor

In response to this:

Thank you 👍
/lgtm
/approve
/cherry-pick release-0.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mimowo · 2025-09-25T07:56:49Z

@ichekrygin can you maybe prepare the cherry-pick? The ./hack/cherry_pick_pull.sh script is pretty useful for that.

…ubernetes-sigs#6973) * Scheduler: add safety check to ensure mutual exclusivity of ElasticJobsViaWorkloadSlices and PartialAdmission These two features are not designed to work together. While admission webhooks already prevent `PartialAdmission` from being combined with `ElasticJobsViaWorkloadSlices`, we add this safety check in the scheduler as well to guard against misconfiguration. This ensures that both features cannot be enabled at the same time, avoiding unexpected scheduling behavior. Signed-off-by: Illya Chekrygin <[email protected]> * Fix: account for replaced workload slice in assigned flavor TotalRequestFor Ensures that when a workload slice is replaced, its resource requests are properly reflected in `TotalRequestFor` during flavor assignment and subsequent preemption targets list generation. Signed-off-by: Illya Chekrygin <[email protected]> * Revert feature enablement at the suite level, replacing it with test-case scoped feature activation. Signed-off-by: Illya Chekrygin <[email protected]> * Update TotalRequestsFor calculation per PR feedback. Signed-off-by: Illya Chekrygin <[email protected]> * Revert change in scheduler removing check for ElasticWorkload. Signed-off-by: Illya Chekrygin <[email protected]> * Remove "redundant" integration test. Signed-off-by: Illya Chekrygin <[email protected]> --------- Signed-off-by: Illya Chekrygin <[email protected]> (cherry picked from commit 19ebce0)

…6973) (#7013) * Scheduler: add safety check to ensure mutual exclusivity of ElasticJobsViaWorkloadSlices and PartialAdmission These two features are not designed to work together. While admission webhooks already prevent `PartialAdmission` from being combined with `ElasticJobsViaWorkloadSlices`, we add this safety check in the scheduler as well to guard against misconfiguration. This ensures that both features cannot be enabled at the same time, avoiding unexpected scheduling behavior. * Fix: account for replaced workload slice in assigned flavor TotalRequestFor Ensures that when a workload slice is replaced, its resource requests are properly reflected in `TotalRequestFor` during flavor assignment and subsequent preemption targets list generation. * Revert feature enablement at the suite level, replacing it with test-case scoped feature activation. * Update TotalRequestsFor calculation per PR feedback. * Revert change in scheduler removing check for ElasticWorkload. * Remove "redundant" integration test. --------- (cherry picked from commit 19ebce0) Signed-off-by: Illya Chekrygin <[email protected]>

tenzen-y · 2025-09-30T13:38:48Z

/release-note-edit

ElasticJobs: workloads correctly trigger workload preemption in response to a scale-up event.

ichekrygin added 2 commits September 23, 2025 10:24

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Sep 23, 2025

k8s-ci-robot requested review from kannon92 and tenzen-y September 23, 2025 23:39

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 23, 2025

tenzen-y reviewed Sep 24, 2025

View reviewed changes

mimowo reviewed Sep 24, 2025

View reviewed changes

ichekrygin added 4 commits September 24, 2025 08:59

Revert feature enablement at the suite level, replacing it with test-…

79c39b0

…case scoped feature activation. Signed-off-by: Illya Chekrygin <[email protected]>

Update TotalRequestsFor calculation per PR feedback.

4154e81

Signed-off-by: Illya Chekrygin <[email protected]>

Revert change in scheduler removing check for ElasticWorkload.

e4aee80

Signed-off-by: Illya Chekrygin <[email protected]>

Remove "redundant" integration test.

e443d93

Signed-off-by: Illya Chekrygin <[email protected]>

k8s-ci-robot assigned mimowo Sep 24, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 24, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 24, 2025

k8s-ci-robot merged commit 19ebce0 into kubernetes-sigs:main Sep 24, 2025
23 checks passed

k8s-ci-robot added this to the v0.14 milestone Sep 24, 2025

ichekrygin deleted the wl-slice-preemption branch September 24, 2025 22:50

mimowo mentioned this pull request Sep 30, 2025

Release v0.14.0 #6756

Closed

36 tasks

	highPriorityJob := testingjob.MakeJob("high", ns.Name).
	SetAnnotation(workloadslicing.EnabledAnnotationKey, workloadslicing.EnabledAnnotationValue).
	Queue(kueue.LocalQueueName(localQueue.Name)).
	Request(corev1.ResourceCPU, "1000m").
	Parallelism(3).
	Completions(3).
	WorkloadPriorityClass(highPriorityClass.Name).
	Obj()

BugFix: scale-up workload stuck pending not triggering preemption. #6973

BugFix: scale-up workload stuck pending not triggering preemption. #6973

Uh oh!

Conversation

ichekrygin commented Sep 23, 2025 • edited by k8s-ci-robot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

netlify bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

ichekrygin commented Sep 24, 2025

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo commented Sep 24, 2025

Uh oh!

k8s-infra-cherrypick-robot commented Sep 24, 2025

Uh oh!

k8s-ci-robot commented Sep 24, 2025

Uh oh!

k8s-ci-robot commented Sep 24, 2025

Uh oh!

Uh oh!

k8s-infra-cherrypick-robot commented Sep 24, 2025

Uh oh!

mimowo commented Sep 25, 2025

Uh oh!

tenzen-y commented Sep 30, 2025

Uh oh!

Uh oh!

ichekrygin commented Sep 23, 2025 •

edited by k8s-ci-robot

Loading

netlify bot commented Sep 23, 2025 •

edited

Loading

mimowo Sep 24, 2025 •

edited

Loading

mimowo Sep 24, 2025 •

edited

Loading