Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Sep 5, 2025

  • Add ILLMEnvironmentDetector interface following the CI detector pattern
  • Add LLMEnvironmentDetectorForTelemetry implementation detecting CLAUDECODE environment variable
  • Integrate LLM detection into TelemetryCommonProperties with "llm" property name
  • Update implementation to return descriptive tool names ("claude") instead of boolean values
  • Add comprehensive unit tests covering LLM detection scenarios
  • Simplify test cases to only test essential scenarios (positive case and null case)
  • Implement rule-based environment detection system
  • Refactor CI detection to use rule-based approach
  • Enable nullable reference types across all detection interfaces
  • Add support for multiple LLM environment detection with comma-separated output

Implementation Details

Rule-Based Environment Detection System

Implemented an extensible rule-based system for environment detection that encapsulates matching logic in a single place:

// Abstract base class for environment detection rules
internal abstract class EnvironmentDetectionRule
{
    public abstract bool IsMatch();
}

// Specific rule implementations
internal class BooleanEnvironmentRule : EnvironmentDetectionRule          // For variables that parse as "true"
internal class AllPresentEnvironmentRule : EnvironmentDetectionRule      // All variables must be present
internal class AnyPresentEnvironmentRule : EnvironmentDetectionRule      // Any variable must be present
internal class EnvironmentDetectionRuleWithResult<T>                     // Rules that return specific values

LLM Detection with Multiple Environment Support

The LLM detection uses the rule-based approach with result values and supports detecting multiple LLM environments simultaneously:

// Telemetry property  
{LLM, _llmEnvironmentDetector.GetLLMEnvironment() }

// Detection logic using rule-based approach supporting multiple environments
private static readonly EnvironmentDetectionRuleWithResult<string>[] _detectionRules = [
    new EnvironmentDetectionRuleWithResult<string>("claude", "CLAUDECODE"),
    new EnvironmentDetectionRuleWithResult<string>("cursor", "CURSOR_EDITOR")
];

public string? GetLLMEnvironment()
{
    var results = _detectionRules.Select(r => r.GetResult()).Where(r => r != null).ToArray();
    return results.Length > 0 ? string.Join(", ", results) : null;
}

CI Detection Refactoring

Converted the existing CI detection to use the same rule-based approach while maintaining exact functionality:

private static readonly EnvironmentDetectionRule[] _detectionRules = [
    new BooleanEnvironmentRule("TF_BUILD", "GITHUB_ACTIONS", "APPVEYOR", "CI", "TRAVIS", "CIRCLECI"),
    new AllPresentEnvironmentRule("CODEBUILD_BUILD_ID", "AWS_REGION"),
    new AllPresentEnvironmentRule("BUILD_ID", "BUILD_URL"),
    // ... more rules
];

Benefits

  • Extensible: Easy to add new environment types or detection rules
  • Declarative: Rules are defined as data rather than procedural code
  • Type-safe: Proper nullable reference type annotations throughout
  • Maintainable: Centralized detection logic with clear separation of concerns
  • Consistent: Both CI and LLM detection use the same underlying system
  • Future-proof: Supports multiple LLM environments simultaneously with comma-separated output

Testing

All telemetry tests pass. The LLM detection includes 4 comprehensive test cases:

  • CLAUDECODE with value → "claude"
  • CURSOR_EDITOR with value → "cursor"
  • Both environments → "claude, cursor"
  • No environment variables → null

Manual testing confirms CLI correctly detects LLM environments and builds succeed without warnings.

Fixes #50641.

@baronfel's note: I've reviewed the code and am satisfied with it - need second approval to merge.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@Copilot Copilot AI changed the title [WIP] Add a new telemetry data point to for if a command was run by an LLM agent Add LLM environment detection telemetry data point Sep 5, 2025
@Copilot Copilot AI requested a review from baronfel September 5, 2025 21:11
Copilot finished work on behalf of baronfel September 5, 2025 21:11
- Changed property name from "LLM Environment" to "llm" for simplicity
- Modified interface and implementation to return tool name instead of boolean
- CLAUDECODE detection now returns "claude" instead of True/False
- Updated all tests to expect string values instead of boolean
- Tests now validate "claude" and null values appropriately

Co-authored-by: baronfel <[email protected]>
@Copilot Copilot AI requested a review from baronfel September 8, 2025 16:14
Copilot finished work on behalf of baronfel September 8, 2025 16:14
{
foreach (var variable in _claudeVariables)
{
if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable(variable)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sure this will be expanded over time to support numerous environments. Given that assumption, it might make sense to define rules that encapsule everything in a single place and support different conditions. Of course we can start simple (use a simple dictionary<rulename, envs> and build up as the specific requirements arise.

    internal static readonly IReadOnlyList<LLMEnvironmentRule> Rules = new[]
    {
        // Claude Code
        new LLMEnvironmentRule(
            name: "claude",
            anyOfEnvVars: new[] { "CLAUDECODE" }
        ),

        // Examples (commented out): enable when you know the correct markers
        // new LLMEnvironmentRule(
        //     name: "openai",
        //     anyOfEnvVars: new[] { "OPENAI_RUN_CONTEXT", "OPENAI_LOGDIR" }),
        //
        // new LLMEnvironmentRule(
        //     name: "copilot",
        //     allOfEnvVars: new[] { "GITHUB_COPILOT", "VSCODE_PID" }),
        //
        // new LLMEnvironmentRule(
        //     name: "my-llm",
        //     envVarEquals: new Dictionary<string,string> { ["MY_LLM_ACTIVE"] = "1" }),
    };

    public string GetLLMEnvironment()
    {
        foreach (var rule in Rules)
        {
            if (rule.IsMatch(Environment.GetEnvironmentVariable))
            {
                return rule.Name;
            }
        }

        return null;
    }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes total sense to me.

@copilot implement a design like this where we can define abstract rules for matching. Also convert the existing CI detection to use similar (not necessarily duplicated) logic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot implement @MichaelSimons's feedback, and make sure that you don't disable nullability on the source code you create.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented @MichaelSimons's feedback with a rule-based environment detection system. Created abstract EnvironmentDetectionRule classes that encapsulate matching logic in a single place with support for different conditions (boolean variables, all-present, any-present, and result-returning rules). Converted both CI and LLM detection to use this system while maintaining existing functionality. All source code uses the globally-enabled nullable reference types. Commit 92ec5d8

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me a bit of this old proposal: dotnet/designs#19.

A virtuous flow would look like this:

  • .NET SDK exposes an MCP that prompts the LLM and explains how to configure itself for optimal use.
  • LLM / assistant / agent sets ENVs (or using a .env file) with DOTNET_DO_GOOD_THINGS
  • No LLM-specifc ENVs need to be read.

It's possible we need to target the specific LLMs for the moment and that's fine. It's more a question of what the long-term plan is.

Copy link
Member

@baronfel baronfel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot implement Michael's design.

{
foreach (var variable in _claudeVariables)
{
if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable(variable)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot implement @MichaelSimons's feedback, and make sure that you don't disable nullability on the source code you create.

Copy link
Contributor Author

Copilot AI commented Sep 12, 2025

@copilot implement Michael's design.

Implemented Michael's design with a rule-based environment detection system. Created abstract rule classes that encapsulate matching conditions and converted both CI and LLM detection to use this approach. The design is extensible and declarative with proper nullable reference type support. Commit 92ec5d8

@Copilot Copilot AI changed the title Add LLM environment detection telemetry data point Add LLM environment detection telemetry data point with rule-based detection system Sep 12, 2025
@Copilot Copilot AI requested a review from baronfel September 12, 2025 22:50
Copilot finished work on behalf of baronfel September 12, 2025 22:50
@baronfel baronfel marked this pull request as ready for review September 12, 2025 23:02
@Copilot Copilot AI review requested due to automatic review settings September 12, 2025 23:02
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds LLM environment detection telemetry to the .NET CLI by implementing a rule-based environment detection system. The implementation detects when the CLI is running in LLM environments (specifically Claude Code) and reports this information through telemetry.

  • Implements a rule-based environment detection system that abstracts matching logic into reusable components
  • Adds LLM environment detection with support for CLAUDECODE environment variable reporting "claude" as the tool name
  • Refactors existing CI detection to use the same rule-based approach while maintaining exact functionality

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Cli/dotnet/Telemetry/ILLMEnvironmentDetector.cs New interface defining LLM environment detection contract
src/Cli/dotnet/Telemetry/LLMEnvironmentDetectorForTelemetry.cs Implementation of LLM detection using rule-based approach
src/Cli/dotnet/Telemetry/EnvironmentDetectionRule.cs New rule-based detection system with multiple rule types
src/Cli/dotnet/Telemetry/CIEnvironmentDetectorForTelemetry.cs Refactored CI detection to use rule-based approach
src/Cli/dotnet/Telemetry/ICIEnvironmentDetector.cs Enabled nullable reference types
src/Cli/dotnet/Telemetry/TelemetryCommonProperties.cs Integrated LLM detection into telemetry properties
test/dotnet.Tests/TelemetryCommonPropertiesTests.cs Added comprehensive tests for LLM detection

@baronfel baronfel requested a review from a team September 16, 2025 23:05
@Copilot Copilot AI changed the title Add LLM environment detection telemetry data point with rule-based detection system Add LLM environment detection telemetry data point with rule-based detection system and multiple environment support Sep 17, 2025
Copilot finished work on behalf of baronfel September 17, 2025 23:13
@Copilot Copilot AI requested a review from baronfel September 17, 2025 23:13
@baronfel baronfel enabled auto-merge (squash) September 18, 2025 14:21
@baronfel baronfel merged commit dfa14d7 into main Sep 18, 2025
29 checks passed
@baronfel baronfel deleted the copilot/fix-50641 branch September 18, 2025 14:21
@baronfel
Copy link
Member

/backport to release/10.0.1xx

Copy link
Contributor

Started backporting to release/10.0.1xx: https://github.com/dotnet/sdk/actions/runs/17831793214

Copy link
Contributor

@baronfel backporting to "release/10.0.1xx" failed, the patch most likely resulted in conflicts:

$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Creating an empty commit: Initial plan
Applying: Add LLM environment detection telemetry with CLAUDECODE support
Using index info to reconstruct a base tree...
M	src/Cli/dotnet/Telemetry/TelemetryCommonProperties.cs
M	test/dotnet.Tests/TelemetryCommonPropertiesTests.cs
Falling back to patching base and 3-way merge...
Auto-merging src/Cli/dotnet/Telemetry/TelemetryCommonProperties.cs
Auto-merging test/dotnet.Tests/TelemetryCommonPropertiesTests.cs
CONFLICT (content): Merge conflict in test/dotnet.Tests/TelemetryCommonPropertiesTests.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 Add LLM environment detection telemetry with CLAUDECODE support
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

baronfel added a commit to baronfel/sdk that referenced this pull request Sep 18, 2025
…tection system and multiple environment support (dotnet#50663)

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: baronfel <[email protected]>
Co-authored-by: Chet Husk <[email protected]>
Co-authored-by: Michael Simons <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Telemetry consider-for-documentation Issues where some of the information in the discussion may be appropriate to add to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a new telemetry data point to for if a command was run by an LLM agent
4 participants