Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Sep 1, 2025

This PR enhances the skUnit API consistency and clarifies the test pattern by introducing clearer separation between assertion and system-under-test clients.

Key Improvements

1. Clear Two-Client Pattern

The framework now clearly distinguishes between:

  • Assertion Client: Used by ChatScenarioRunner constructor for semantic evaluations and assertions
  • System Under Test Client: Passed to RunAsync() - the client whose behavior is being tested

2. Constructor-Based Initialization

Test classes now use constructor-based initialization with clearer variable names:

public class GreetingTests
{
    private readonly ChatScenarioRunner ScenarioRunner;
    private readonly IChatClient systemUnderTestClient;

    public GreetingTests()
    {
        var assertionClient = /* assertion/evaluation model */;
        systemUnderTestClient = /* system under test model */;
        ScenarioRunner = new ChatScenarioRunner(assertionClient);
    }

    [Fact]
    public async Task TestGreeting()
    {
        var scenarios = ChatScenario.LoadFromText(markdown);
        await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
    }
}

3. Enhanced Documentation

  • Updated README with comprehensive examples showing the two-client pattern
  • Added XML documentation explaining the distinction between assertion and system-under-test clients
  • Updated all test framework examples (xUnit, MSTest, NUnit) to show the improved pattern

4. API Consistency

  • Continued support for ASSERT keyword alongside existing CHECK statements
  • ChatScenarioRunner constructor parameter renamed to assertionClient for clarity
  • All test classes updated to use the clearer naming convention
  • All documentation and test scenarios use original assertion names (SemanticCondition, JsonCheck, FunctionCall) for consistency

5. Updated Examples

  • All demo projects refactored to use the improved pattern
  • Test infrastructure updated to demonstrate best practices
  • Documentation examples consistently show the two-client separation

This change maintains full backward compatibility while promoting clearer, more maintainable test code that makes the distinction between evaluation and testing clients explicit.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI changed the title [WIP] Enhance API consistency by refactorings and renamings Enhance API consistency: Introduce ChatScenarioRunner, ASSERT keyword, and assertion synonyms Sep 1, 2025
Copilot AI requested a review from mehrandvd September 1, 2025 19:05
Copilot finished work on behalf of mehrandvd September 1, 2025 19:05
@mehrandvd
Copy link
Owner

@copilot The implementation is fine; update all the tests and documentation to use the latest recommended naming conventions. There's no need to include unit tests for backward compatibility anymore. Keep the existing tests and just update them to use the latest API.

Copy link
Contributor Author

Copilot AI commented Sep 1, 2025

@copilot The implementation is fine; update all the tests and documentation to use the latest recommended naming conventions. There's no need to include unit tests for backward compatibility anymore. ...

I've successfully updated all tests and documentation to use the latest recommended naming conventions. Here's what I've updated:

Tests Updated:

  • All test classes now use ChatScenarioRunner instead of ScenarioAssert
  • All test methods use RunAsync() instead of PassAsync()
  • Renamed test files to reflect new API (e.g., ChatScenarioRunnerLoggingTests.cs)
  • Updated test infrastructure base class to use new API

Documentation Updated:

  • Updated README.md with all new API examples
  • Enhanced check-statements-spec.md to document both ASSERT and CHECK keywords
  • Added comprehensive section on assertion name synonyms

Scenario Files Updated:

  • All test scenario markdown files now use ASSERT keyword
  • Used new assertion synonyms like Condition, JsonStructure, and ToolCall
  • Updated demo project scenarios to use latest conventions

Key Changes:

  • ScenarioAssert.PassAsync()ChatScenarioRunner.RunAsync()
  • CHECK SemanticConditionASSERT Condition
  • CHECK JsonCheckASSERT JsonStructure
  • CHECK FunctionCallASSERT ToolCall

All tests are passing and code formatting is consistent. The changes maintain full backward compatibility while promoting the new recommended API. Commit: 2288ec8

Copilot finished work on behalf of mehrandvd September 1, 2025 20:29
@mehrandvd
Copy link
Owner

@copilot Thanks for the PR—great improvements. I have a suggestion to make the test pattern and docs clearer, especially regarding the two different chat clients involved.

Current sample pattern:

var runner = new ChatScenarioRunner(myChatClient);
await runner.RunAsync(scenarios, myChatClient);

Issue:

There are actually two distinct roles but the sample uses the same variable name (myChatClient) for both:

  1. Assertion / evaluation model: the client passed into ChatScenarioRunner’s constructor, used internally for semantic/assertion evaluation.
  2. System-under-test model: the client passed into RunAsync, whose behavior the scenarios validate.

When both are called myChatClient and the runner is created inline inside each test, it’s easy to conflate them.

Proposal:

  • In each test class, instantiate ChatScenarioRunner once in the constructor (no fixture needed).
  • Use clearer variable names to show intent (e.g., assertionClient vs systemUnderTestClient).
  • In test methods, just call ScenarioRunner.RunAsync(…).

Example refactor:

public class GreetingTests
{
    private readonly ChatScenarioRunner ScenarioRunner;
    private readonly IChatClient systemUnderTestClient;

    public GreetingTests()
    {
        var assertionClient = /* build the model used for semantic assertions */;
        systemUnderTestClient = /* build or obtain the model under test */;
        ScenarioRunner = new ChatScenarioRunner(assertionClient);
    }

    [Fact]
    public async Task TestGreeting()
    {
        var markdown = File.ReadAllText("greeting.md");
        var scenarios = ChatScenario.LoadFromText(markdown);

        await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
    }
}

Documentation adjustment:

Before:

var runner = new ChatScenarioRunner(myChatClient);
await runner.RunAsync(scenarios, myChatClient);

After (clear separation):

// In your test class constructor:
var assertionClient = /* assertion/evaluation model */;
ScenarioRunner = new ChatScenarioRunner(assertionClient);

// In your test:
await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);

Suggested follow-up changes:

  1. Update test files to adopt constructor-based initialization and clearer variable names.
  2. Update README / samples to show only the simplified call (no per-test instantiation).
  3. Consider renaming the ChatScenarioRunner constructor parameter and its internal field to assertionClient or evaluationClient for clarity.
  4. Add XML doc comments explaining the two-client pattern to prevent accidental misuse.

Copy link
Contributor Author

Copilot AI commented Sep 1, 2025

@copilot Thanks for the PR—great improvements. I have a suggestion to make the test pattern and docs clearer, especially regarding the two different chat clients involved.

Current sample pattern...

I've implemented all the suggested improvements to clarify the two-client pattern:

Key Changes:

  • Updated SemanticTestBase to use clearer variable names: assertionClient for evaluations and SystemUnderTestClient for testing
  • All test classes now use constructor-based initialization with ScenarioRunner property
  • Test methods simply call ScenarioRunner.RunAsync(scenarios, systemUnderTestClient)
  • Updated ChatScenarioRunner constructor parameter to assertionClient with comprehensive XML documentation
  • Updated README with clearer examples showing the two-client separation
  • Updated all demo projects to use the improved pattern

Documentation improvements show:

public GreetingTests()
{
    var assertionClient = /* assertion/evaluation model */;
    systemUnderTestClient = /* system under test model */;
    ScenarioRunner = new ChatScenarioRunner(assertionClient);
}

[Fact]
public async Task TestGreeting()
{
    await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
}

All tests pass and the pattern now clearly distinguishes between the assertion client (for semantic evaluations) and the system under test client (whose behavior is being validated). Commit: a7b84ea

Copilot AI changed the title Enhance API consistency: Introduce ChatScenarioRunner, ASSERT keyword, and assertion synonyms Enhance API consistency and clarify test pattern: ChatScenarioRunner with two-client separation Sep 1, 2025
Copilot finished work on behalf of mehrandvd September 1, 2025 20:54
@mehrandvd
Copy link
Owner

@copilot Avoid adding tests to verify if synonyms work, as they are only supported for backward compatibility. Remove any mention or use of synonyms in the documentation or tests. Synonyms should remain in the code solely to support legacy users of the library.

Copy link
Contributor Author

Copilot AI commented Sep 1, 2025

@copilot Avoid adding tests to verify if synonyms work, as they are only supported for backward compatibility. Remove any mention or use of synonyms in the documentation or tests. Synonyms should rema...

I've removed all synonym documentation and tests as requested. The changes include:

  • Removed the "Assertion Name Synonyms" section from check-statements-spec.md
  • Updated all documentation examples to use original assertion names (SemanticCondition, JsonCheck, FunctionCall, etc.)
  • Removed tests that specifically verified synonym functionality
  • Updated all test scenario .md files and demo projects to use original assertion names
  • Kept synonym parsing code in KernelAssertionParser.cs for backward compatibility

Synonyms now remain in the code solely to support legacy users but are no longer documented or tested. All tests pass. Commit: 9d6ff18

Copilot finished work on behalf of mehrandvd September 1, 2025 21:21
@mehrandvd mehrandvd marked this pull request as ready for review September 1, 2025 22:06
@mehrandvd mehrandvd merged commit 8957c44 into main Sep 1, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants