fix(#6307): quarantine tests are being run but exit with status code 0 #6397

calvinbayer · 2025-09-08T09:16:45Z

What does this PR do?

If a test is quarantined, the test is executed but if it fails, it should not fail the pipeline (i.e., status should exit with code 0)
Previously, the test was skipped

Motivation

[BUG]: Quarantine causes playwright test to be skipped #6307

github-actions · 2025-09-08T09:17:20Z

Overall package size

Self size: 11.98 MB
Deduped: 112.06 MB
No deduping: 112.46 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.7.0 | 35.02 MB | 35.02 MB | | @datadog/native-appsec | 10.2.1 | 20.64 MB | 20.65 MB | | @datadog/native-iast-taint-tracking | 4.0.0 | 11.72 MB | 11.73 MB | | @datadog/pprof | 5.10.0 | 9.91 MB | 10.3 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.4 | 2.95 MB | 5.6 MB | | @datadog/wasm-js-rewriter | 4.0.1 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 10.3.0 | 617.18 kB | 1.08 MB | | import-in-the-middle | 1.14.2 | 122.36 kB | 850.93 kB | | lru-cache | 10.4.3 | 804.3 kB | 804.3 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | source-map | 0.7.6 | 185.63 kB | 185.63 kB | | pprof-format | 2.2.1 | 163.06 kB | 163.06 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 7.0.5 | 63.38 kB | 63.38 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.3 | 23.74 kB | 23.74 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2025-09-08T09:17:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.31%. Comparing base (820f59b) to head (7f8ccbb).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6397   +/-   ##
=======================================
  Coverage   84.31%   84.31%           
=======================================
  Files         477      477           
  Lines       20088    20088           
=======================================
  Hits        16938    16938           
  Misses       3150     3150

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2025-09-08T09:32:02Z

Benchmarks

Benchmark execution time: 2025-09-08 14:53:44

Comparing candidate commit 7f8ccbb in PR branch calvin.bayer/fix-6307/quarantined-tests-are-skipped with baseline commit 820f59b in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1688 metrics, 76 unstable metrics.

datadog-datadog-prod-us1 · 2025-09-08T09:35:22Z

⚠️ Tests

⚠️ Warnings

❄️ 1 New flaky test detected

cypress@latest commonJS libraries capabilities adds capabilities to tests from integration-tests/cypress/cypress.spec.js (Datadog)
Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/dd-trace-js/dd-trace-js/integration-tests/cypress/cypress.spec.js)

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7f8ccbb | Docs | Was this helpful? Give us feedback!}

juan-fernandez

nice job! Some minor comments

juan-fernandez · 2025-09-08T13:20:41Z

packages/datadog-instrumentations/src/playwright.js

          test.expectedStatus = 'skipped'
+        } else if (testProperties.quarantined) {
+          // Do not skip quarantined tests, let them run and overwrite results post-run if they fail
+          const fullname = getTestFullname(test)


we should probably extract this to a function in this file, as this logic is found in other places

I agree, refactored the code to extract that into a separate function

juan-fernandez · 2025-09-08T13:21:07Z

integration-tests/playwright/playwright.spec.js

              const events = payloads.flatMap(({ payload }) => payload.events)

              const testSession = events.find(event => event.type === 'test_session_end').content
-              if (isQuarantining) {


shouldn't we also check that the test session is passing? so that we know that the error is actually suppressed

actually, we should modify the test to assert that the test session fails or passes depending on whether we're quarantining or not

We check the exit code here: https://github.com/DataDog/dd-trace-js/pull/6397/files/e36043bda26333a45225dc1566cbe7e00ddc5990#diff-68b5e7860766c25caad41f6fbfce1920923567d3c88cbf7cf24894ce1bd8e305L1328-L1332

For the test session status: Right now it is reported as failed which is the same behavior as Jest. Let's follow up on that separately in case that's not the desired behavior

As discussed in Slack, created a separate task to make sure we report the suite/module/session as passed back to Datadog: https://datadoghq.atlassian.net/browse/SDTEST-2611

juan-fernandez

🚀

#6397) * fix(#6307): quarantine tests are being run but exit with status code 0 * refactor: extract function to get fully qualified name of test

calvinbayer closed this Sep 8, 2025

calvinbayer reopened this Sep 8, 2025

calvinbayer force-pushed the calvin.bayer/fix-6307/quarantined-tests-are-skipped branch from e336312 to 974f897 Compare September 8, 2025 09:21

calvinbayer added the semver-patch label Sep 8, 2025

calvinbayer force-pushed the calvin.bayer/fix-6307/quarantined-tests-are-skipped branch from 974f897 to e36043b Compare September 8, 2025 09:29

calvinbayer marked this pull request as ready for review September 8, 2025 10:59

calvinbayer requested a review from a team as a code owner September 8, 2025 10:59

calvinbayer requested review from juan-fernandez and Mariovido September 8, 2025 10:59

juan-fernandez reviewed Sep 8, 2025

View reviewed changes

calvinbayer added 2 commits September 8, 2025 16:45

fix(#6307): quarantine tests are being run but exit with status code 0

632fef2

refactor: extract function to get fully qualified name of test

7f8ccbb

calvinbayer force-pushed the calvin.bayer/fix-6307/quarantined-tests-are-skipped branch from 95fd801 to 7f8ccbb Compare September 8, 2025 14:45

calvinbayer requested a review from juan-fernandez September 9, 2025 09:57

juan-fernandez approved these changes Sep 9, 2025

View reviewed changes

calvinbayer merged commit 9d9dc3f into master Sep 9, 2025
1069 of 1075 checks passed

calvinbayer deleted the calvin.bayer/fix-6307/quarantined-tests-are-skipped branch September 9, 2025 17:57

dd-octo-sts bot mentioned this pull request Sep 9, 2025

v5.66.0 proposal #6378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(#6307): quarantine tests are being run but exit with status code 0 #6397

fix(#6307): quarantine tests are being run but exit with status code 0 #6397

calvinbayer commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 8, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 8, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Sep 8, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Sep 8, 2025 •

edited by datadog-official bot

Loading

Uh oh!

juan-fernandez left a comment

Uh oh!

juan-fernandez Sep 8, 2025 •

edited

Loading

Uh oh!

calvinbayer Sep 8, 2025

Uh oh!

juan-fernandez Sep 8, 2025

Uh oh!

juan-fernandez Sep 8, 2025

Uh oh!

calvinbayer Sep 8, 2025

Uh oh!

calvinbayer Sep 9, 2025

Uh oh!

juan-fernandez left a comment

Uh oh!

Uh oh!

Uh oh!

fix(#6307): quarantine tests are being run but exit with status code 0 #6397

fix(#6307): quarantine tests are being run but exit with status code 0 #6397

Conversation

calvinbayer commented Sep 8, 2025

What does this PR do?

Motivation

Uh oh!

github-actions bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

codecov bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

datadog-datadog-prod-us1 bot commented Sep 8, 2025 • edited by datadog-official bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

juan-fernandez left a comment

Choose a reason for hiding this comment

Uh oh!

juan-fernandez Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

calvinbayer Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

juan-fernandez Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

juan-fernandez Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

calvinbayer Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

calvinbayer Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

juan-fernandez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 8, 2025 •

edited

Loading

codecov bot commented Sep 8, 2025 •

edited

Loading

pr-commenter bot commented Sep 8, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Sep 8, 2025 •

edited by datadog-official bot

Loading

juan-fernandez Sep 8, 2025 •

edited

Loading