Investigate if OOB metrics from ASP.NET Core and HttpClient can be enabled from instrumentation libraries

[vishweshbankwar]

.NET8.0 added OOB support for metrics in ASP.NET Core and HttpClient that aligns with OpenTelemetry semantic conventions. Needs further investigation to check if the instrumentation libraries can be updated to enable these by adding AddMeter in .NET8.0 onwards. Things we need to check

1. What will be the experience when users upgrade their target framework to .NET8.0 from lower versions? Will there be significant difference in terms of metric dimensions, request duration value etc.?
2. There are multiple metrics that are emitted from a single meter, e.g. from Microsoft.AspNetCore.Hosting, there are two metrics that are emitted http.server.active_requests and http.server.request.duration. Only http.server.request.duration is part of stability plan. How do we handle non-stable metrics that are emitted? As it cannot be part of stable instrumentation library release.
3. Semantic conventions are in feature freeze, how would we handle the changes if .NET8.0 is released before stable release of semantic conventions?
4. How do we handle Enrich/Filter. At present, Asp.Net Core instrumentation offers this for metrics, and .NET also has some mechanism to enrich. Need to ensure a smooth experience for enrich/filter exists. Update 10/6: Follow Enrich and Filter support for metrics [ASP.NET Core] and [HttpClient] #1733

NOTE: This is the initial list of things, may add more as we go through the investigation.

Reference PRs
HttpClient
https://github.com/dotnet/runtime/pull/89809/files
dotnet/runtime#87319

ASP.NET Core
dotnet/aspnetcore#48375
dotnet/aspnetcore#46834

[vishweshbankwar]

Moving the discussion here open-telemetry/opentelemetry-dotnet#4437 (comment)

@eerhardt @noahfalk @JamesNK @cijothomas

[JamesNK]

• @davidfowl

Semantic conventions are in feature freeze, how would we handle the changes if .NET8.0 is released before stable release of semantic conventions?

My opinion: If breaking changes happen to OTel after .NET 8 ships, the .NET team is interested in staying compatible with OTel. We'll mitigate the breaking change by having a configuration setting to preserve old behavior for people who depend on it.

[cijothomas]

@vishweshbankwar Please add to the list how do we handle Enrich/Filter. The instrumentation library has that feature (only for Asp.Net Core instrumentation), and .NET also has some mechanism to enrich. Need to ensure a smooth experience for enrich/filter exists.

[vishweshbankwar]

Update

Background

The request from the .NET team is to incorporate specific metrics as default features in the instrumentation libraries provided by this repository. This change will facilitate a seamless integration of metrics for users on .NET 8.0 through the utilization of extension methods like AddAspNetCoreInstrumentation and AddHttpClientInstrumentation.

To simplify comprehension, I've categorized these metrics into three groups:

Category-1

Metrics conforming to semantic conventions specification and marked as feature-freeze:

• System.Net.Http

  • Specification Version: http.client.request.duration
  • Proposed .NET Version: http.client.request.duration

• Microsoft.AspNetCore.Hosting

  • Specification Version: http.server.request.duration
  • Proposed .NET Version: http.server.request.duration

Category-2

Metrics also adhering to the semantic conventions specification but categorized as Experimental:

• Microsoft.AspNetCore.Hosting

  • Specification Version: http.server.active_requests
  • Proposed .NET Version: http.server.active_requests

Category-3

Metrics not covered by semantic conventions specification, specifically designed for applications using HttpClient and ASP.NET Core:

• System.Net.Http

  • http.client.active_requests
  • http.client.open_connections
  • http.client.connection.duration
  • http.client.request.time_in_queue

• System.Net.NameResolution

  • dns.lookup.duration

• Microsoft.AspNetCore.Server.Kestrel

  • kestrel.active_connections
  • kestrel.connection.duration
  • kestrel.rejected_connections
  • kestrel.queued_connections
  • kestrel.queued_requests
  • kestrel.upgraded_connections
  • kestrel.tls_handshake.duration
  • kestrel.active_tls_handshakes

• Microsoft.AspNetCore.Http.Connections

  • signalr.server.connection.duration
  • signalr.server.active_connections

• Microsoft.AspNetCore.Routing

  • aspnetcore.routing.match_attempts

• Microsoft.AspNetCore.Diagnostics

  • aspnetcore.diagnostics.exceptions

• Microsoft.AspNetCore.RateLimiting

  • aspnetcore.rate_limiting.active_request_leases
  • aspnetcore.rate_limiting.request_lease.duration
  • aspnetcore.rate_limiting.queued_requests
  • aspnetcore.rate_limiting.request.time_in_queue
  • aspnetcore.rate_limiting.requests

Option-1

Include all the metrics from all three categories as default features for .NET 8.0 users in the stable release of instrumentation library.

Pros:

• Simplifies onboarding for .NET 8.0 users to the newly added metrics.
• Ensures a smoother transition for users migrating from .NET 6.0/.NET 7.0to.NET 8.0`.
• Provides users Category-2 that are not yet stable from specification.

Cons:

• Potential introduction of breaking changes in .NET 8.0 Category-2 as the corresponding metrics in the specification become stable. These changes would require a major version update or ability for users to switch to otel specification one.

Option-2

Include Category-1 and Category-3 as default features for .NET 8.0 users in the stable release. Category-2 will be offered through an experimental feature flag until the specification stabilizes (using Technical Workaround).

Pros:

• Provides easy onboarding for .NET 8.0 users to the newly added metrics.
• Prevents potential breaking changes by offering Category-2 as an experimental feature.

Technical Workaround

Since metrics emitted in Category-1 and Category-2 belong to the same Meter, a workaround is needed in the instrumentation library, such as using views to offer them as experimental.

NOTE : Users will experience some changes when migrating from .NET 6.0 to .NET 8.0 regarding dimensions in Category-1 due to differences in specification and DotNet versions. However, the impact should be minimal assuming .NET versions does not vary significantly from the spec version.

[vishweshbankwar]

I think we should go with option-1. It brings considerable value to .NET users by adopting OOTB metrics. The additional metrics are designed specifically for .NET applications to provide deeper insights and complements the metrics defined by OpenTelemetry specification. In addition to this, adding them as part of instrumentation library provides consistent configuration experience for users on different .NET versions (i.e. .NET6.0, .NET7.0 and .NET8.0).

It's important to note that any potential breaking changes would affect just one metric, namely http.server.active_requests. In the event of a breaking change from the .NET side, our approach should involve retaining the previous behavior while allowing users to configure the new behavior through a feature flag. To achieve this, a collaborative effort with the .NET team is required. ref.

[reyang]

In the event of a breaking change from the .NET side, our approach should involve retaining the previous behavior while allowing users to configure the new behavior through a feature flag. To achieve this, a collaborative effort with the .NET team is required. ref.

Sounds reasonable to me 👍

[cijothomas]

If going with option1: Please share perf numbers too. We should be seeing a lot of improvement due to avoiding reflection/diagnostic listeners etc., but its possible that the extra metrics might kill the gains and could make things worse. If we find that the extra metircs is affecting perf, document a view recipe so users can easily turn them off.

[cijothomas]

If going with option1: Please share perf numbers too. We should be seeing a lot of improvement due to avoiding reflection/diagnostic listeners etc., but its possible that the extra metrics might kill the gains and could make things worse. If we find that the extra metircs is affecting perf, document a view recipe so users can easily turn them off.

Well...irrespective of this, we still need to add the view recipe in the documentation!