Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dotnet-counters does not report values from multiple Metrics Meters with distinct tags #4564

Closed
PJB3005 opened this issue Mar 22, 2024 · 3 comments
Closed

dotnet-counters does not report values from multiple Metrics Meters with distinct tags #4564

PJB3005 opened this issue Mar 22, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@PJB3005
Copy link

PJB3005 commented Mar 22, 2024 •
edited
Loading

Description

I was adding built-in support for System.Diagnostics.Metrics to a library I maintain. In the library I want it to be possible to report separate metrics for each instance of a network socket, so I made each object instantiate a Meter and have an API for passing in a list of tags to said meter to distinguish them. Then I can just instantiate a new observable counter/gauge for each property I want to report and it looks pretty nice. For reference, here is the code.

However, dotnet counters only seems to report the metrics from one meter. In this example I have a separate meter with bind=:: and bind=0.0.0.0 created, but I only get results for bind=::.

Name                                                               Current Value
[Lidgren.Network.NetPeer]
    messages_dropped (Count)
        bind=::                                                            0
    messages_received (Count)
        bind=::                                                       97,375
    messages_resent (Count)
        bind=::,drop_reason=delay                                         20
        bind=::,drop_reason=hole                                         481
    messages_sent (Count)
        bind=::                                                      106,354
    packets_received (Count)
        bind=::                                                       82,033
    packets_sent (Count)
        bind=::                                                      102,192
    recycle_pool_bytes (bytes)
        bind=::                                                      922,600
    sent_bytes (bytes)
        bind=::                                                    8,875,636
    storage_allocated_bytes (bytes)
        bind=::                                                      932,688

Cross-referencing against prometheus-net1, these values are correctly reported however.

> curl localhost:44880/metrics --silent | rg -i lidgren
# HELP lidgren_network_netpeer_packets_sent Number of packets sent by this NetPeer. (ObservableCounter`1)
# TYPE lidgren_network_netpeer_packets_sent gauge
lidgren_network_netpeer_packets_sent{bind="0.0.0.0"} 93498
lidgren_network_netpeer_packets_sent{bind="::"} 94293
# HELP lidgren_network_netpeer_packets_received Number of packets received by this NetPeer. (ObservableCounter`1)
# TYPE lidgren_network_netpeer_packets_received gauge
lidgren_network_netpeer_packets_received{bind="0.0.0.0"} 74491
lidgren_network_netpeer_packets_received{bind="::"} 75749
# HELP lidgren_network_netpeer_messages_sent Number of messages sent by this NetPeer. (ObservableCounter`1)
# TYPE lidgren_network_netpeer_messages_sent gauge
lidgren_network_netpeer_messages_sent{bind="0.0.0.0"} 97988
lidgren_network_netpeer_messages_sent{bind="::"} 98752

I already noticed that the same thing happened if I created two separate instruments with identical name but different tags, but I figured that one was at least somewhat reasonable. Instead I ended up just using callback-form that returns multiple measurements for the counter that needed to report tags.

Is this intended behavior or a bug? If it's intended behavior, I suppose I should make a static Meter instance and use it for all peers, but that would be rather annoying and inflexible IMO.

Configuration

Tool: dotnet-counters, version 8.0.510501+8c08c89a0643d31db91e119b1adb463be3e0ffe5
.NET runtime I'm testing against: .NET 8.0.3
OS: Windows 11 Pro, 23H2, build 22631.3296
Architecture: x64
Environment: just running stuff directly from my IDE/terminal, no containers or whatever.

dotnet info 
> dotnet --info
.NET SDK:
 Version:           8.0.103
 Commit:            6a90b4b4bc
 Workload version:  8.0.100-manifests.4e94be9c

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.103\

.NET workloads installed:
 Workload version: 8.0.100-manifests.4e94be9c
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.1.24080.9
  Architecture: x64
  Commit:       1d1bf92fcf

.NET SDKs installed:
  8.0.100 [C:\Program Files\dotnet\sdk]
  8.0.103 [C:\Program Files\dotnet\sdk]
  9.0.100-preview.1.24101.2 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-preview.1.24081.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-preview.1.24080.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.0-preview.1.24081.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  E:\ss14\space-station-14\RobustToolbox\global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Regression?

Meter/instrument tags are a new feature in .NET 8, so I doubt it. I had trouble before because the tags wouldn't show up at all, but that was fixed by just making sure my dotnet-counters was at the latest version.

Footnotes

  1. I just made a PR to make prometheus-net respect meter/instrument tags. ↩
@PJB3005 PJB3005 added the bug Something isn't working label Mar 22, 2024
@PJB3005 PJB3005 changed the title dotnet-counters does not report values from multiple meters with distinct tags dotnet-counters does not report values from multiple Metrics Meters with distinct tags Mar 22, 2024
@noahfalk
Copy link
Member

I'm pretty sure are hitting a known issue tracked here and the current plan is to fix it during .NET 9 for the scenario involving dotnet-counters. I wouldn't take that as a promise, but I think its likely.

Just so you are aware OpenTelemetry.NET also doesn't support multiple Meters distinguished solely by tags and at the moment the OTel spec officially says that isn't supported. Others have also encountered this issue and asked them to please support it in the future. I believe they are looking into it but if you care about OpenTelmetry support you can always open an issue in OTel.NET repo. The changes needed to make dotnet-counters and OTel support this are fully independent of one another.

Hope that helps explain where things are!
-Noah

@PJB3005
Copy link
Author

PJB3005 commented Mar 26, 2024

at the moment the OTel spec officially says that isn't supported.

Damn, that's kind of annoying. Guess I have to change my library code then. Thanks though!

@KalleOlaviNiemitalo KalleOlaviNiemitalo mentioned this issue Aug 8, 2024
Closed
noahfalk added a commit to noahfalk/diagnostics that referenced this issue Sep 17, 2024
@noahfalk
Metric name disambiguation
4e0c07d 
In .NET 8.0 we added tags on Meters and Instruments but MetricsEventSource only included the names when emitting value publishing events. This made it ambiguous when there was more than one Meter or Instrument with the same name. In 9.0 MetricsEventSource started included InstrumentIDs in the BeginInstrumentReporting events and in the value publishing events that provides a stronger correlation identifier. This change consumes the IDs in those events within Microsoft.Diagnostics.Monitoring.EventPipe, allowing dotnet-counters (and presumably dotnet-monitor too) to track time series that differ only by the tags provided to the Meter/Instrument constructors.

I also re-enabled a disabled test that got broken .NET 9.0 added the System.Runtime Meter.

Fixes dotnet#4843, dotnet#4564
@noahfalk noahfalk mentioned this issue Sep 17, 2024
Merged
noahfalk added a commit that referenced this issue Sep 19, 2024
@noahfalk
Metric name disambiguation (#4938)
37b70e0 
In .NET 8.0 we added tags on Meters and Instruments but
MetricsEventSource only included the names when emitting value
publishing events. This made it ambiguous when there was more than one
Meter or Instrument with the same name. In 9.0 MetricsEventSource
started included InstrumentIDs in the BeginInstrumentReporting events
and in the value publishing events that provides a stronger correlation
identifier. This change consumes the IDs in those events within
Microsoft.Diagnostics.Monitoring.EventPipe, allowing dotnet-counters
(and presumably dotnet-monitor too) to track time series that differ
only by the tags provided to the Meter/Instrument constructors.

I also re-enabled a disabled test that got broken when .NET 9.0 added
the System.Runtime Meter.

Fixes #4843, #4564
@tommcdon
Copy link
Member

Closing via #4938

@github-actions github-actions bot locked and limited conversation to collaborators Oct 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@noahfalk @PJB3005 @tommcdon