BuildAndVerify.ps1 and Agent Improvements (Issue-Resolver, PR-Reviewer & Test-Repro)

Description of Change

This PR adds BuildAndVerify.ps1, a PowerShell script that enables automated build verification before finalizing changes, and improves multiple agents by consolidating duplicated content and enhancing their workflows. Also introduces a new specialized test-repro-agent for creating issue reproduction tests.

BuildAndVerify.ps1 Script

Purpose:

• Verify builds compile successfully before completing tasks
• Fallback verification tool when OS limitations prevent running full HostApp/Sandbox scripts
• Catch compilation errors and test failures early in the development cycle

Default Behavior:

• Builds TestCases.HostApp on all available TFMs (target frameworks)
• Builds each UI test project for available platforms
• Automatically detects OS and builds appropriate targets:
  • Linux: Android targets only (net10.0-android)
  • macOS: iOS, Android, and MacCatalyst targets
  • Windows: Windows and Android targets

Optional Features:

• -RunUnitTests flag runs all unit tests after building
• -Configuration parameter supports Debug/Release builds
• Clear colored output with success/failure reporting
• Non-zero exit codes on failures for CI integration

Parameters

• -RunUnitTests (switch) - Run all unit tests after building (default: false)
• -Configuration (string) - Build configuration: "Debug" or "Release" (default: Debug)

Usage Examples

# Default: Build HostApp and UI tests for current platform
pwsh .github/scripts/BuildAndVerify.ps1

# Build and run all unit tests
pwsh .github/scripts/BuildAndVerify.ps1 -RunUnitTests

# Build with Release configuration
pwsh .github/scripts/BuildAndVerify.ps1 -Configuration Release

# Build with Release and run unit tests
pwsh .github/scripts/BuildAndVerify.ps1 -Configuration Release -RunUnitTests

New: Test-Repro-Agent

Purpose: Specialized agent for creating issue reproduction tests ONLY (does NOT implement fixes)

Key Features:

• Minimal scope: Creates reproduction test only, explicitly prevented from fixing issues
• Prioritizes unit tests: Prefer unit tests (faster) over UI tests when possible
• Clear boundaries: Strong emphasis that this agent does NOT fix issues
• Validation focused: Ensures test FAILS (reproducing the bug) before stopping
• Lightweight workflow: 5-step process (fetch → assess → create → validate → report)
• Minimal tooling: Only has access to issue reading, test execution, and UI test instructions

When to use:

• "Create a reproduction test for issue Don't subscribe to ImmutableBrush #12345"
• "Write a test that reproduces #67890"
• When you need JUST the reproduction test without the full fix workflow

Workflow:

1. Fetch issue and read all comments
2. Create assessment with reproduction strategy
3. Create reproduction test (prefer unit tests, fallback to UI tests)
4. Validate test FAILS (reproducing bug)
5. Report results and STOP

Issue-Resolver Agent Improvements

Consolidation and Enhancements:

• Eliminated duplication: Removed ~95 lines of UI test structure that duplicated .github/instructions/uitests.instructions.md
• Referenced authoritative source: Now points to uitests.instructions.md for complete UI test details
• Prioritized unit tests: Updated workflow to prefer unit tests over UI tests (unit tests run faster)
• Enhanced PR tracking: Step 1 now specifically searches for the OPEN PR created by @copilot for the current issue, skipping closed PRs
• Improved state awareness: Initial assessment now determines current checkpoint stage by reading existing PR, enabling continuation from where previous work left off
• Better workflow continuity: Agent can resume work at any checkpoint by analyzing PR state and commits
• Streamlined workflow: Removed Step 10 (PR submission) as PRs are created/updated at checkpoints when validation is needed
• Added PR Standards: Defined standardized PR title format and description template for consistent commit messages (repository uses PR title/description as commit messages)
• Fixed test guidance: Corrected that handlers always require UI tests, not unit tests; standardized on using BuildAndVerify.ps1 for all test execution
• Reduced file size: From 713 lines to ~495 lines (-31%)
• Better maintainability: Single source of truth for UI test patterns

PR-Reviewer Agent Improvements

Consolidation and Enhancements:

• Removed checkpoint/pause: Agent now validates autonomously and reports findings directly without waiting for human approval
• Updated test validation: Supports both unit tests and UI tests, not just UI tests
• Prioritized unit tests: Prefer unit tests where appropriate (faster, no device needed)
• Clarified handler requirement: Handlers always require UI tests (device validation mandatory)
• Standardized test execution: Use BuildAndVerify.ps1 for running all tests consistently
• Platform limitation handling: Report upfront if unable to test on required platforms (e.g., Linux without iOS)
• Improved workflow: Streamlined from 7 steps to 6 steps (removed checkpoint/pause step)
• Autonomous validation: Agent attempts to validate tests on its own using BuildAndVerify.ps1 and reports platform limitations

Key Workflow Changes

Test-Repro-Agent (NEW):

1. Fetch issue and read all comments
2. Create assessment with reproduction strategy (prefer unit tests)
3. Create reproduction test
4. Validate test FAILS (confirming bug reproduction)
5. Report results and STOP (explicitly does NOT fix the issue)

Issue-Resolver:

1. Fetch issue and find Copilot PR (skip closed PRs)
2. Determine checkpoint stage from PR state
3. Create/update reproduction test (prefer unit tests except handlers which always need UI tests)
4. Validate test fails using BuildAndVerify.ps1
5. Submit PR at checkpoints following standardized format

PR-Reviewer:

1. Checkout PR
2. Review code and form independent opinion
3. Review tests (both unit and UI tests)
4. Deep analysis - identify edge cases
5. Validate tests autonomously using BuildAndVerify.ps1 (report platform limitations if unable to test)
6. Write review with findings

Validation & Testing

All permutations tested successfully on Linux:

• ✅ Default (no parameters): Builds HostApp (Android) + UI test project
• ✅ Configuration Release: Builds with Release configuration
• ✅ RunUnitTests: Builds and runs 4 unit test projects (Core, Controls.Core, Controls.Xaml, Essentials)
• ✅ Configuration Release + RunUnitTests: Builds and runs unit tests in Release mode

Error detection verified:

• ✅ Compile failures: Detects and reports build errors with file names, line numbers, and error messages
• ✅ Test failures: Detects and reports test failures with stack traces and failure details
• ✅ Exit codes: Returns non-zero exit codes when failures are detected (for CI integration)

Documentation Updates

.github/README-AI.md:

• Added BuildAndVerify.ps1 to "Shared Scripts" section
• Documented Linux OS limitation use case

.github/copilot-instructions.md:

• Added new "Build Verification" section in Development Workflow
• Explained when to use on Linux with OS limitations

.github/agents/issue-resolver.md:

• Consolidated and improved workflow
• Added PR Standards section
• Standardized all test execution to use BuildAndVerify.ps1

.github/agents/pr-reviewer.md:

• Removed checkpoint/pause workflow
• Updated to support both unit and UI test validation
• Standardized test execution using BuildAndVerify.ps1

.github/agents/test-repro-agent.md: (NEW)

• Complete specialized agent for creating reproduction tests ONLY
• Clear boundaries: does NOT fix issues
• Minimal tool requirements (issue reading, test execution, UI test instructions)
• Prioritizes unit tests over UI tests
• Validation-focused 5-step workflow

Files Changed

• ✅ .github/scripts/BuildAndVerify.ps1 - New build verification script (250+ lines)
• ✅ .github/README-AI.md - Added script documentation
• ✅ .github/copilot-instructions.md - Added Build Verification section
• ✅ .github/agents/issue-resolver.md - Consolidated workflow (~369 lines changed, 218 removed)
• ✅ .github/agents/pr-reviewer.md - Autonomous validation workflow (~185 lines changed)
• ✅ .github/agents/test-repro-agent.md - NEW: Specialized reproduction test creation agent (327 lines)

Issues Fixed

N/A - This PR provides build verification tooling and improves agent instructions for GitHub Copilot agents.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.