fix: prioritize xcresult output over stderr noise in test results#254
Merged
cameroncooke merged 6 commits intogetsentry:mainfrom Mar 3, 2026
Merged
Conversation
c6d1a9d to
ecf2288
Compare
ecf2288 to
14eaf6e
Compare
commit: |
cameroncooke
added a commit
that referenced
this pull request
Mar 2, 2026
…r output Make xcresult handling consistent across shared, macOS, and device test tools. Parse totalTestCount with strict numeric checks so empty/invalid values cannot suppress build-output fallback behavior. Filter stderr at line level instead of dropping entire text blocks. This preserves useful status and diagnostics while still removing stderr noise when xcresult summaries are authoritative. Extract shared stderr content filtering into a common utility to remove duplication and keep behavior aligned across test paths. Update device tool tests to assert summary-first ordering. Fixes #254
When xcresult parsing succeeds and tests actually ran (totalTestCount > 0), stderr lines are redundant noise (e.g. "multiple matching destinations") and are filtered out. The xcresult summary is placed first in the response. When xcresult reports 0 tests (build failed before tests could run), the xcresult is meaningless and stderr is preserved since it contains the actual compilation errors. Fixes getsentry#231
…r output Make xcresult handling consistent across shared, macOS, and device test tools. Parse totalTestCount with strict numeric checks so empty/invalid values cannot suppress build-output fallback behavior. Filter stderr at line level instead of dropping entire text blocks. This preserves useful status and diagnostics while still removing stderr noise when xcresult summaries are authoritative. Extract shared stderr content filtering into a common utility to remove duplication and keep behavior aligned across test paths. Update device tool tests to assert summary-first ordering. Fixes getsentry#254
050c4b7 to
314dc20
Compare
…tSummary Strip orphaned `---` separator lines left behind after filtering [stderr] lines from Claude Code consolidated output. Also move the XcresultSummary interface into the shared test-result-content module to eliminate duplication.
…o-warn coercers Three identical inline coerce functions in cli.ts, yargs-app.ts, and daemon.ts all mapped 'warning' to 'warn' for backwards compatibility. Extract a single coerceLogLevel export from logger.ts and reference it from all three call sites.
…ry boundary Move consolidation from individual call sites (build-utils, test-common) into the typed-tool-factory handlers so it applies once at the boundary. This eliminates scattered consolidation calls and simplifies logic functions. Supporting changes: - Cache isRunningUnderClaudeCode() result since it runs on every tool call - Accept optional EnvironmentDetector in consolidation for testability - Inject CommandExecutor/FileSystemExecutor into test-common for DI - Simplify filterStderrContent now that separator cleanup is unnecessary - Add unit tests for consolidation logic and factory wiring
- Reduce verbose type guards in test_macos formatTestSummary using direct casts and nullish coalescing - Remove unused JSDoc typedef for TestSummary in test_macos - Consolidate enforceInstallPolicy guard conditions into single check - Simplify init report agentsGuidance construction - Remove stale comments in test_device
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
cameroncooke
added a commit
that referenced
this pull request
Mar 3, 2026
…r output Make xcresult handling consistent across shared, macOS, and device test tools. Parse totalTestCount with strict numeric checks so empty/invalid values cannot suppress build-output fallback behavior. Filter stderr at line level instead of dropping entire text blocks. This preserves useful status and diagnostics while still removing stderr noise when xcresult summaries are authoritative. Extract shared stderr content filtering into a common utility to remove duplication and keep behavior aligned across test paths. Update device tool tests to assert summary-first ordering. Fixes #254
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When xcodebuild tests fail, stderr output (e.g. "multiple matching destinations" warnings) was being tagged as errors and displayed prominently, hiding the actual test failure details from the xcresult bundle.
This PR fixes the response assembly in all three test tool paths (
test_macos,test_device,test-common):totalTestCount > 0): xcresult is the authoritative source. Stderr lines are filtered out as redundant noise, and the xcresult summary is placed first in the response.totalTestCount == 0): xcresult is meaningless (empty summary). Stderr is preserved since it contains the actual compilation errors.No changes to
build-utils.tsor stderr capture/classification logic.Fixes #231
Test plan