Please add some timeout, I also noticed they are missing in the old tests, so you could specify them too

This looks good, but let me use this as an occasion to get understanding on how scale down works.

When a user requests scale down:

1. Do we release some quota during the scaledown before the pods are actually deleted, resulting in temporarily pods running over quota?
2. Do we keep more quota than needed until the scale up is finished?

IIUC ideally we would gradually release the quota (as the pods terminate and are accounted in reclaimablePods) until the scale down is finished. Once finished we cleanup reclaimablePods and update the podsets count, so that the scaled down workload looks as if it was created new. This workload property would make debugging easier.

Let me know if I'm potentially missing something. If you agree, but assess this is more work I can lgtm / approve this one, so that we fix one issue at the time, let me know.

Currently, batch/v1.Job is the only framework that supports the ElasticJobsViaWorkloadSlices feature.
A scale-down is triggered when the user (or automation) updates the Job's spec.parallelism to a lower value. Kueue’s handling of this event is entirely reactionary. Once the spec.parallelism field is decreased, the Kubernetes Job controller proceeds to complete and remove the excess pods. Kueue, in parallel, observes the change and updates the associated Workload’s spec.podSets[].count to reflect the new, lower value.

Since both the K8s Job controller and the Kueue controller respond independently to the same job change, the pod removal and quota release happen concurrently. In practice, pod deletion often completes slightly before the Workload update and subsequent quota release. However, there are no guardrails to enforce strict ordering, nor is there coordination between the two controllers to ensure that quota is only released after the pods are gone.

By the same token, there's no concept of gradual scale-down in this context. When a Job’s parallelism is reduced, for example, from 10 to 2, the Kubernetes Job controller will proceed to delete the 8 excess pods almost immediately. The deletions happen as fast as the controller and API server allow, and there's no stepwise or progressive reduction. So in practice, the scale-down is effectively instantaneous from the controller's point of view.

I see, actually the reclaimablePods are only decremented when the terminated pods are Succeeded. We should not make the assumption for pods deleted due to scale down. They may end in 0 or non zero exit code. So, I think it is correct not to rely on reclaimablePods.

I see this approach allows for pods running at the same beyond quota for a brief time. Since this is temporary only, I'm not very concerned about it. It would be ideal to release quota gradually, but it might be hard in practice, so we can leave it as a follow up feature on its own.