CI: configure gotestsum for flaky test retries and early failure#52069
Open
veeceey wants to merge 2 commits intomoby:masterfrom
Open
CI: configure gotestsum for flaky test retries and early failure#52069veeceey wants to merge 2 commits intomoby:masterfrom
veeceey wants to merge 2 commits intomoby:masterfrom
Conversation
Add gotestsum flags across all test scripts (unit, integration, and Windows) to automatically retry flaky tests and fail early when too many tests fail: - --rerun-fails=2: retry failed tests up to 2 times before reporting failure (existing libnetwork flaky tests keep --rerun-fails=4) - --rerun-fails-max-failures=10: skip retries if initial run has more than 10 failures (indicates a real problem, not flakiness) - --rerun-fails-abort-on-data-race: never mask data race failures with retries - --rerun-fails-report: write a report of retried tests for post-analysis - --max-fails=20: abort the test run early after 20 failures to avoid waiting for long CI runs that will never pass For unit tests, packages are now passed via --packages= instead of as positional arguments, which is required for --rerun-fails to correctly identify and rerun only the specific failed tests rather than the entire suite. For integration tests using --raw-command (pre-compiled test binaries), only --max-fails is added since --rerun-fails is not compatible with --raw-command mode. Closes moby#50732 Signed-off-by: Varun Chawla <[email protected]>
3439182 to
320f133
Compare
gotestsum requires packages to be specified via --packages= when --rerun-fails is used with go test args. Fix the unit test invocation to use --packages= instead of positional args, and add --packages=./... to the integration test invocation. Signed-off-by: Varun Chawla <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #50732 by configuring
gotestsumflags across all test scripts to automatically retry flaky tests and fail early when too many tests fail.Changes
hack/test/unit: Added--rerun-fails=2,--rerun-fails-max-failures=10,--rerun-fails-abort-on-data-race,--rerun-fails-report, and--max-fails=20to all gotestsum invocations (api, client, main packages, and libnetwork). Packages are now passed via--packages=instead of positional args, which is required for--rerun-failsto correctly rerun only the specific failed tests. The existing libnetwork flaky tests retain their--rerun-fails=4setting and also gain the new safety flags.hack/make/.integration-test-helpers: Added--max-fails=20only, since--rerun-failsis not compatible with--raw-commandmode (used for pre-compiled test binaries).hack/make.ps1(Windows): Added the same flags to all gotestsum invocations for unit, integration, and integration-cli tests..github/workflows/buildkit.yml: Added the full set of flags to the BuildKit test step.Flag rationale
--rerun-fails--rerun-fails-max-failures--rerun-fails-abort-on-data-race--rerun-fails-reportbundles/*-rerun-report.txt--max-failsTest plan
--max-failscauses early abort when many tests fail simultaneouslyCloses #50732
🤖 Generated with Claude Code