[API Proposal]: System.ClientModel Open Telemetry extension APIs #114084

m-redding · 2025-03-31T21:36:05Z

Background and motivation

The System.ClientModel library provides building blocks for .NET clients that call cloud services. For background, this package has been reviewed in the following previous issues: #94126 | #97711 | #104617 | #106197 | #111046

These additional APIs are adding extension methods for client library authors to add distributed tracing to their libraries built on System.ClientModel. Client libraries have the following spans:

HTTP spans - these are provided by HttpClient, we will not be duplicating these in System.ClientModel
Spans for each call to public APIs that send HTTP requests - this proposal addresses this area

The proposed API is simply adding extension methods to Activity and ActivitySource for library authors to use to create distributed tracing spans in each public method call. These methods provide common base functionality that is applicable to all client libraries. Library authors can then use the returned Activity directly to add any additional functionality to meet their own needs.

API Proposal

namespace System.ClientModel.Primitives
{
+    public static partial class ActivityExtensions
+    {
+        public static System.Diagnostics.Activity MarkFailed(this System.Diagnostics.Activity activity, System.Exception? exception) { throw null; }
+        public static System.Diagnostics.Activity? StartClientActivity(this System.Diagnostics.ActivitySource activitySource, System.ClientModel.Primitives.ClientPipelineOptions options, string name, System.Diagnostics.ActivityKind kind = System.Diagnostics.ActivityKind.Internal, System.Diagnostics.ActivityContext parentContext = default(System.Diagnostics.ActivityContext), System.Collections.Generic.IEnumerable<System.Collections.Generic.KeyValuePair<string, object?>>? tags = null) { throw null; }
+    }
...
    public partial class ClientPipelineOptions
    {
        public ClientPipelineOptions() { }
        public System.ClientModel.Primitives.ClientLoggingOptions? ClientLoggingOptions { get { throw null; } set { } }
+      public bool? EnableDistributedTracing { get { throw null; } set { } }
        public System.ClientModel.Primitives.PipelinePolicy? MessageLoggingPolicy { get { throw null; } set { } }
        public System.TimeSpan? NetworkTimeout { get { throw null; } set { } }
        public System.ClientModel.Primitives.PipelinePolicy? RetryPolicy { get { throw null; } set { } }
        public System.ClientModel.Primitives.PipelineTransport? Transport { get { throw null; } set { } }
        public void AddPolicy(System.ClientModel.Primitives.PipelinePolicy policy, System.ClientModel.Primitives.PipelinePosition position) { }
        protected void AssertNotFrozen() { }
        public virtual void Freeze() { }
    }
...
}
...

API Usage

  public class SampleClient
  {
      private readonly Uri _endpoint;
      private readonly ApiKeyCredential _credential;
      private readonly ClientPipeline _pipeline;
      private readonly SampleClientOptions _sampleClientOptions;

      // Each client should have a static ActivitySource named after the full name
      // of the client.
      // Tracing should start as experimental.
      internal static ActivitySource ActivitySource { get; } = new($"Experimental.{typeof(MapsClient).FullName!}");

      public SampleClient(Uri endpoint, ApiKeyCredential credential, SampleClientOptions? options = default)
      {
          options ??= new SampleClientOptions();
          _sampleClientOptions = options;

          _endpoint = endpoint;
          _credential = credential;
          ApiKeyAuthenticationPolicy authenticationPolicy = ApiKeyAuthenticationPolicy.CreateBearerAuthorizationPolicy(credential);

          _pipeline = ClientPipeline.Create(options,
              perCallPolicies: ReadOnlySpan<PipelinePolicy>.Empty,
              perTryPolicies: new PipelinePolicy[] { authenticationPolicy },
              beforeTransportPolicies: ReadOnlySpan<PipelinePolicy>.Empty);
      }

      public ClientResult<SampleResource> UpdateResource(SampleResource resource)
      {
          // Attempt to create and start an Activity for this operation.
          // StartClientActivity does nothing and returns null if distributed tracing was
          // disabled by the consuming application or if there are not any active listeners.
          using Activity? activity = ActivitySource.StartClientActivity(_sampleClientOptions, $"{nameof(SampleClient)}.{nameof(UpdateResource)}");

          try
          {
              using PipelineMessage message = _pipeline.CreateMessage();
              PipelineRequest request = message.Request;
              request.Method = "PATCH";
              request.Uri = new Uri($"https://www.example.com/update?id={resource.Id}");
              request.Headers.Add("Accept", "application/json");
              request.Content = BinaryContent.Create(resource);

              _pipeline.Send(message);

              PipelineResponse response = message.Response!;
              if (response.IsError)
              {
                  throw new ClientResultException(response);
              }
              SampleResource updated = ModelReaderWriter.Read<SampleResource>(response.Content)!;
              return ClientResult.FromValue(updated, response);
          }
          catch (Exception ex)
          {
              // Catch any exceptions and update the activity. Then re-throw the exception.
              activity?.MarkFailed(ex);
              throw;
          }
      }
  }

Alternative Designs

No response

Risks

No response

teo-tsirpanis · 2025-04-01T00:26:25Z

The MarkFailed method already exists in Activity.AddException.

m-redding · 2025-04-01T00:36:55Z

@teo-tsirpanis
AddException is different than MarkFailed, it adds the exception as a span event, which we aren't doing at all because in general, customers don't want all that exception info and it seems like span events are probably going to be deprecated - open-telemetry/opentelemetry-specification#4430

MarkFailed follows the OTel spec for recording errors https://opentelemetry.io/docs/specs/semconv/general/recording-errors/ by setting the span status and description, as well as setting error.type to a predictable value based on typical errors seen in clients built on System.ClientModel (with fallbacks if it's not typical). Makes it super easy for client library authors to instrument their client methods.

teo-tsirpanis · 2025-04-01T00:44:20Z

Thanks. In this case it might be better to define MarkFailed as an instance method on Activity itself, since it has broad applicability and does not really depend on System.ClientModel (IIUC).

m-redding · 2025-04-01T15:12:46Z

@teo-tsirpanis I think setting the status and description is universal, but the error.type attribute would need to be customizable since the value is potentially defined differently by every instrumentation.

This is the draft implementation:

public static Activity MarkFailed(this Activity activity, Exception? exception)
{
    activity.SetStatus(ActivityStatusCode.Error, exception?.Message);
        
    string? errorCode = null;
    if (exception is ClientResultException clientResultException) // ClientResultException is a System.ClientModel exception type
    {
        errorCode = clientResultException.Status.ToString();
    }
    errorCode ??= exception?.GetType().FullName;
    errorCode ??= "_OTHER";

    activity.SetTag("error.type", errorCode);

    return activity;
}

The activity.SetStatus is something that applies everywhere, but everything else is specific to System.ClientModel. I'm definitely not opposed to adding the method to System.Diagnostics instead, but the method wouldn't be able to do much more than just the SetStatus line, so it doesn't seem as helpful for users in the general case.

stephentoub · 2025-04-03T19:18:40Z

@tarekgh, @samsp-msft, @noahfalk. This is proposing additional Activity{Source} APIs (albeit as extension methods)

tarekgh · 2025-04-03T19:40:31Z

Regarding MarkFailed, Activity already has AddException API. I am not sure if this is worth extension API either as it is simple setting the error and adding the exception. But I wouldn't mind having it as an extension method there if it is important.

tarekgh · 2025-04-03T19:56:02Z

Regrarding StartClientActivity, Do you have any scenarios that need to start the Activity with links? I mean with IEnumerable<ActivityLink>? links = null.

m-redding · 2025-04-03T22:55:42Z

@tarekgh

Activity already has AddException API.

idk if you saw my reply about that one above - #114084 (comment)

I am not sure if this is worth extension API either as it is simple setting the error and adding the exception

Benefits of having it are:

Limiting extra code in clients and lighten/remove instrumentation design burden on libraries
- Even though this method is small it's easier to have the implementation in one place rather than having the same method copied into every client or client library internally
- No need for every client to define what error.type should be, makes it so library authors can instrument their client methods without needing to understand Open Telemetry or Activity/ActivitySource if they just want basic client method instrumentation
Abstraction for changes in the future
- Distributed tracing in clients built on SCM will be experimental bc lots of OTel conventions are in development (including the convention for recording errors). If something changes, library authors can just bump the dependency, rather than needing to rewrite or regenerate clients

Do you have any scenarios that need to start the Activity with links? I mean with IEnumerable? links = null.

No, we don't in SCM. There are a few Azure SDKs that start Activities with links, so I added the parameter for consistency, but I'm fine with simplifying this method if it's not very common.

tarekgh · 2025-04-03T23:32:56Z

I am fine with the proposal.

m-redding added the api-suggestion Early API idea and discussion, it is NOT ready for implementation label Mar 31, 2025

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Mar 31, 2025

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Mar 31, 2025

vcsjones added area-System.ClientModel and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 1, 2025

m-redding mentioned this issue Apr 2, 2025

3p distributed tracing extension methods Azure/azure-sdk-for-net#48929

Merged

m-redding closed this as completed Apr 30, 2025

dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Apr 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[API Proposal]: System.ClientModel Open Telemetry extension APIs #114084

[API Proposal]: System.ClientModel Open Telemetry extension APIs #114084

m-redding commented Mar 31, 2025

teo-tsirpanis commented Apr 1, 2025

m-redding commented Apr 1, 2025

teo-tsirpanis commented Apr 1, 2025

m-redding commented Apr 1, 2025

stephentoub commented Apr 3, 2025

tarekgh commented Apr 3, 2025 •

edited

Loading

tarekgh commented Apr 3, 2025

m-redding commented Apr 3, 2025

tarekgh commented Apr 3, 2025

[API Proposal]: System.ClientModel Open Telemetry extension APIs #114084

[API Proposal]: System.ClientModel Open Telemetry extension APIs #114084

Comments

m-redding commented Mar 31, 2025

Background and motivation

API Proposal

API Usage

Alternative Designs

Risks

teo-tsirpanis commented Apr 1, 2025

m-redding commented Apr 1, 2025

teo-tsirpanis commented Apr 1, 2025

m-redding commented Apr 1, 2025

stephentoub commented Apr 3, 2025

tarekgh commented Apr 3, 2025 • edited Loading

tarekgh commented Apr 3, 2025

m-redding commented Apr 3, 2025

tarekgh commented Apr 3, 2025

tarekgh commented Apr 3, 2025 •

edited

Loading