Add EventSource/EventCounter tracing and metrics for Caching

[analogrelay]

Some events listed below. The goal is to add events that will help users solve problems so if these aren't going to do that job, feel free to suggest/discuss others!

• Microsoft.Extensions.Caching.Memory EventSource

  • cache-hits event & counter - Triggered when a cache lookup succeeds.
  • cache-misses event & counter - Triggered when a cache lookup fails.
  • cache-count counter - Counts the number of items in the cache
  • estimated-size counter - Counts the number of bytes based on Size values? (which are opt-in and estimates already)

• Microsoft.Extensions.Caching.Distributed EventSource

  • cache-hits event & counter - Triggered when a cache lookup succeeds.
  • cache-misses event & counter - Triggered when a cache lookup fails.
  • Size and count should be tracked on the server/service providing the cache.

See https://gist.github.com/anurse/af1859663ac91c6cf69c820cebe92303 for some guidance on adding EventSources and EventCounters to ASP.NET projects. This is out of date.

[ericstj]

@noahfalk / @shirhatti -- what's your thinking around this? Does it make sense to add EventCounter usage in Microsoft.Extensions.Caching, or do we need to take a look at the broader customer scenario of diagnostics in ASP.NETCore to identify more libraries to instrument? I'd want to make sure that any additions we make here fit well with the overall diagnostics story.

[shirhatti]

My vote is to wait till we have support for dimensions on metrics. With dimensions you can always disambiguate hit/miss counter by filtering on the cache name (or some other unique identifier) dimension that you care about.

[ericstj]

@noahfalk @shirhatti how should we be thinking about EventCounter support in general? Is it already supported everywhere customers want it and this is just one place it was missed, or does it need a more thorough look to ensure it's added everywhere?

[noahfalk]

Is it already supported everywhere customers want it and this is just one place it was missed, or does it need a more thorough look to ensure it's added everywhere?

So far the strategy has been a mixture, we proactively identified counters we thought would be most impactful and got those added during .NET 3 and 5. The remainder are things that didn't stand out/we weren't aware of and we have been handling those reactively. IMO it is easier to fault in cases like these on-demand because we have a specific customer who can help us validate the scenario + drive the priority. So this probably isn't "just one place we missed", but I also wasn't planning to initiate a broad survey where we add more counters that we don't yet have any .NET Core customers asking for. If requests to fill gaps become numerous and handling them as a batch looks more efficient I'm happy to switch course.

My vote is to wait till we have support for dimensions on metrics

Were the previous .NET Framework counters multi-dimensional? I still need to look more closely.