It's really weird that sound of the instances are stuck.

The issue stalling issue is resolved when pytest-xdist is disabled.

I just reactivated the pytest-xdist to check if the problem went away on its own.

Let's wait for the run with pytest-xdist to complete.

Let's wait for the run with pytest-xdist to complete.

I have been experimenting on my fork. I suspect we will need to disable pytest-xdist everywhere, since the instance that stalls is not predictable. Let's see what happens here.

Comparing a previous run where stalled builds were canceled and the most recent run with pytest-xdist enabled the jobs that stalled are not consistent.

Edit: Here is a run on my fork with verbose on + pytest-xdist trying to debug the issue.

Is there any new library version triggering this?

Edit: Here is a run on my fork with verbose on + pytest-xdist trying to debug the issue.

On your verbose fork, the last test that PASSED on gw1 is inspection/tests/test_partial_dependence.py::test_partial_dependence_pipeline. Then there are plenty of tests running on gw0 only.

At the end:

utils/tests/test_weight_vector.py::test_type_invariance[float64-WeightVector64]

is started but never PASSED while the previous utils/tests/test_weight_vector.py::test_type_invariance[float32-WeightVector32] was PASSED on gw0.

Not sure what to make of this though...

The last build seems stalled even with pytest-xdist disabled. We need to activate the verbose mode.

It looks good, but maybe we should disable verbosity here before merging.

The tests are passing. We need to investigate more but at least PRs will become greener.

BTW this stalling is kind of painful. Every PR that does not sync with upstream will need to wait +1 hours for jobs to stall out. This means PRs are taking up resources that prevents the CI from running on other PRs.

True and we observe with @ogrisel that Azure was indicating that we were consuming too many credits.
I would propose to make a push in the PR for the latest PR when we see that contributors are around. It is a bit invasive but I don't see how to solve the problem otherwise.