Here is an idea to reduce our Azure Pipelines CI usage while trying to make it give informative feedback as possible in case of typos committed in PRs.

The suggestion would be to change the Azure Pipelines config to have 2 top level jobs:

• 1 linux quick-failing build that runs the tests with pytest --maxfail=3
• 1 linting job that runs flake8, mypy and black

Then if both those jobs complete successfully, run the other jobs to cover all the supported platforms.

The goal of this design would be to:

• to not waste too much resources for PRs that receive many sequential commits as most test failures are expected to be highly correlated across platforms,
• to still get informative informative test failure message to iterate, even if the linter fails for an unused import (already the case with the current config, this would be preserved),
• to not waste time in the first run with a very long failure report when half of the tests to fail for the same typo.

The downside it that when one expects the PR to be green, the total time will be "first test run + longest test run of the remainder" instead of just "linter time + longest test run" as is the case now with more parallelism.