Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Instrumentation.AspNetCore, Instrumentation.Http] Cache activity names #1854

Closed
wants to merge 7 commits into from

Conversation

joegoldman2
Copy link
Contributor

Fixes #1759.

Changes

Introduce a static ConcurrentDictionary to cache the activity display name. I don't think it's worth adding an entry in the changelog for this change.

For significant contributions please make sure you have completed the following items:

  • Appropriate CHANGELOG.md updated for non-trivial changes
  • Design discussion issue #
  • Changes in public API reviewed

Copy link

codecov bot commented Jun 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.10%. Comparing base (71655ce) to head (77319f3).
Report is 376 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1854      +/-   ##
==========================================
- Coverage   73.91%   72.10%   -1.81%     
==========================================
  Files         267      302      +35     
  Lines        9615    11175    +1560     
==========================================
+ Hits         7107     8058     +951     
- Misses       2508     3117     +609     
Flag Coverage Δ
unittests-Exporter.Geneva 53.20% <ø> (?)
unittests-Exporter.InfluxDB 94.65% <ø> (?)
unittests-Exporter.Instana 71.24% <ø> (?)
unittests-Exporter.OneCollector 91.29% <ø> (?)
unittests-Exporter.Stackdriver 75.73% <ø> (?)
unittests-Extensions 79.33% <ø> (?)
unittests-Extensions.AWS 77.24% <ø> (?)
unittests-Extensions.Enrichment 100.00% <ø> (?)
unittests-Instrumentation.AWS 87.56% <ø> (?)
unittests-Instrumentation.AWSLambda 87.96% <ø> (?)
unittests-Instrumentation.AspNetCore 85.27% <ø> (?)
unittests-Instrumentation.ElasticsearchClient 79.87% <ø> (?)
unittests-Instrumentation.EntityFrameworkCore 55.49% <ø> (?)
unittests-Instrumentation.EventCounters 76.36% <ø> (?)
unittests-Instrumentation.GrpcNetClient 79.61% <ø> (?)
unittests-Instrumentation.Hangfire 93.58% <ø> (?)
unittests-Instrumentation.Http 81.08% <ø> (?)
unittests-Instrumentation.Owin 83.43% <ø> (?)
unittests-Instrumentation.Process 100.00% <ø> (?)
unittests-Instrumentation.Quartz 78.94% <ø> (?)
unittests-Instrumentation.Runtime 100.00% <ø> (?)
unittests-Instrumentation.SqlClient 91.89% <ø> (?)
unittests-Instrumentation.StackExchangeRedis 67.02% <ø> (?)
unittests-Instrumentation.Wcf 78.47% <ø> (?)
unittests-PersistentStorage 65.78% <ø> (?)
unittests-Resources.AWS 75.88% <ø> (?)
unittests-Resources.Azure 77.94% <ø> (?)
unittests-Resources.Container 72.41% <ø> (?)
unittests-Resources.Gcp 72.54% <ø> (?)
unittests-Resources.Host 47.67% <ø> (?)
unittests-Resources.Process 81.81% <ø> (?)
unittests-Resources.ProcessRuntime 82.35% <ø> (?)
unittests-Sampler.AWS 87.97% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 325 files with indirect coverage changes

@Kielek Kielek requested a review from CodeBlanch June 3, 2024 04:57
@@ -26,6 +27,7 @@ internal sealed class RequestDataHelper
private const string OtherHttpMethod = "_OTHER";

private static readonly char[] SplitChars = new[] { ',' };
private static readonly ConcurrentDictionary<string, string> DisplayNameCache = new ConcurrentDictionary<string, string>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be the size limit for this collection?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joegoldman2 - What happens if a service receives huge number of requests with invalid routes?

Copy link
Contributor Author

@joegoldman2 joegoldman2 Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, the dictionary will grow without limit. Let me try to think about a solution.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that there's no built-in "bounded" collection in .NET. While it's possible to implement a custom one, probably by wrapping ConcurrentDictionary with additional checks, I'm not entirely enthusiastic about it. I'm open to suggestions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joegoldman2 - One possible option is to check the Count of items within the dictionary before adding a new item. The Count could be some pre-defined number. If the Count exceeds that number, then we could simply skip adding items to the Dictionary.

@vishweshbankwar vishweshbankwar added comp:instrumentation.aspnetcore Things related to OpenTelemetry.Instrumentation.AspNetCore comp:instrumentation.http Things related to OpenTelemetry.Instrumentation.Http labels Jun 4, 2024
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jun 15, 2024
@github-actions github-actions bot removed comp:instrumentation.aspnetcore Things related to OpenTelemetry.Instrumentation.AspNetCore comp:instrumentation.http Things related to OpenTelemetry.Instrumentation.Http labels Jun 15, 2024

if (DisplayNameCache.Count < DisplayNameCacheSize)
{
return DisplayNameCache.GetOrAdd(httpRoute!, $"{namePrefix} {httpRoute}");
Copy link
Member

@vishweshbankwar vishweshbankwar Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checking - Can two http methods have the same route values?. if that happens, we will incorrectly set the display name here.

e.g.
Put /api/document/{id}
Get api/document/{id}

@@ -84,7 +87,27 @@ public void SetActivityDisplayName(Activity activity, string originalHttpMethod,
var normalizedHttpMethod = this.GetNormalizedHttpMethod(originalHttpMethod);
var namePrefix = normalizedHttpMethod == "_OTHER" ? "HTTP" : normalizedHttpMethod;

activity.DisplayName = string.IsNullOrEmpty(httpRoute) ? namePrefix : $"{namePrefix} {httpRoute}";
activity.DisplayName = GetDisplayName();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much performance would it gain? The new method seems to have enough if checks to counter the benefit. Plus the additional memory usage.

@joegoldman2
Copy link
Contributor Author

Re-checking - Can two http methods have the same route values?. if that happens, we will incorrectly set the display name here.

Right, so the cache key should take into account the http method.

How much performance would it gain? The new method seems to have enough if checks to counter the benefit. Plus the additional memory usage.

I did a benchmark and here is the result:

Method HttpRoute HttpMethod Mean Error StdDev Gen0 Allocated
Original _OTHER 0.9941 ns 0.1721 ns 0.0094 ns - -
New_CacheKeyString _OTHER 1.2971 ns 0.8686 ns 0.0476 ns - -
New_CacheKeyTuple _OTHER 1.3056 ns 0.8290 ns 0.0454 ns - -
Original GET 0.9614 ns 0.5957 ns 0.0327 ns - -
New_CacheKeyString GET 0.8245 ns 1.0305 ns 0.0565 ns - -
New_CacheKeyTuple GET 0.9863 ns 0.0562 ns 0.0031 ns - -
Original /resources/{id} _OTHER 8.8834 ns 4.1025 ns 0.2249 ns 0.0068 64 B
New_CacheKeyString /resources/{id} _OTHER 26.0081 ns 3.1322 ns 0.1717 ns 0.0076 72 B
New_CacheKeyTuple /resources/{id} _OTHER 19.5532 ns 7.1305 ns 0.3908 ns - -
Original /resources/{id} GET 9.0777 ns 4.2050 ns 0.2305 ns 0.0068 64 B
New_CacheKeyString /resources/{id} GET 25.6794 ns 3.3270 ns 0.1824 ns 0.0068 64 B
New_CacheKeyTuple /resources/{id} GET 17.8230 ns 4.8392 ns 0.2653 ns - -
Benchmark code
[MemoryDiagnoser]
[ShortRunJob]
public class Tests
{
    private static readonly ConcurrentDictionary<string, string> DisplayNameCache = new ConcurrentDictionary<string, string>();
    private static readonly ConcurrentDictionary<(string, string), string> DisplayNameCache2 = new ConcurrentDictionary<(string, string), string>();
    private const int DisplayNameCacheSize = 1000;

    [Params("", "/resources/{id}")]
    public string? HttpRoute { get; set; }

    [Params("GET", "_OTHER")]
    public string HttpMethod { get; set; }

    [Benchmark]
    public string Original()
    {
        var namePrefix = HttpMethod == "_OTHER" ? "HTTP" : HttpMethod;
        return string.IsNullOrEmpty(HttpRoute) ? namePrefix : $"{namePrefix} {HttpRoute}";
    }

    [Benchmark]
    public string New_CacheKeyString()
    {
        var namePrefix = HttpMethod == "_OTHER" ? "HTTP" : HttpMethod;
        return GetDisplayName();

        string GetDisplayName()
        {
            if (string.IsNullOrEmpty(HttpRoute))
            {
                return namePrefix;
            }

            if (DisplayNameCache.TryGetValue($"{HttpMethod} {HttpRoute!}", out var displayName))
            {
                return displayName;
            }

            if (DisplayNameCache.Count < DisplayNameCacheSize)
            {
                return DisplayNameCache.GetOrAdd($"{HttpMethod} {HttpRoute!}", $"{namePrefix} {HttpRoute}");
            }

            return $"{namePrefix} {HttpRoute}";
        }
    }

    [Benchmark]
    public string New_CacheKeyTuple()
    {
        var namePrefix = HttpMethod == "_OTHER" ? "HTTP" : HttpMethod;
        return GetDisplayName();

        string GetDisplayName()
        {
            if (string.IsNullOrEmpty(HttpRoute))
            {
                return namePrefix;
            }

            if (DisplayNameCache2.TryGetValue((HttpMethod, HttpRoute!), out var displayName))
            {
                return displayName;
            }

            if (DisplayNameCache2.Count < DisplayNameCacheSize)
            {
                return DisplayNameCache2.GetOrAdd((HttpMethod, HttpRoute!), $"{namePrefix} {HttpRoute}");
            }

            return $"{namePrefix} {HttpRoute}";
        }
    }
}

The original version is better in all cases. Either my approach isn't right, or this optimization isn't worth it.

@Kielek Kielek added comp:instrumentation.aspnetcore Things related to OpenTelemetry.Instrumentation.AspNetCore comp:instrumentation.http Things related to OpenTelemetry.Instrumentation.Http labels Jun 24, 2024
@github-actions github-actions bot removed comp:instrumentation.aspnetcore Things related to OpenTelemetry.Instrumentation.AspNetCore comp:instrumentation.http Things related to OpenTelemetry.Instrumentation.Http labels Jun 28, 2024
@joegoldman2
Copy link
Contributor Author

I'm not sure what to do with this PR considering the benchmark result. Any suggestion?

return displayName;
}

if (DisplayNameCache.Count < DisplayNameCacheSize)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Count gets a lock on all buckets, so this won't be efficient on the hot path

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's what the benchmark above shows.

@vishweshbankwar
Copy link
Member

@joegoldman2 - Sorry for the delay on this. I will take a look at benchmark and get back.

Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added Stale and removed Stale labels Jul 19, 2024
Copy link
Contributor

github-actions bot commented Aug 3, 2024

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Aug 3, 2024
Copy link
Contributor

Closed as inactive. Feel free to reopen if this PR is still being worked on.

@github-actions github-actions bot closed this Aug 10, 2024
@joegoldman2 joegoldman2 deleted the fix/1759 branch September 11, 2024 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Instrumentation.AspNetCore, Instrumentation.Http] Cache activity names
6 participants