-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wire up the DD_TRACE_BATCH_INTERVAL setting #4940
Conversation
Datadog ReportBranch report: ✅ |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (71ms) : 62, 80
. : milestone, 71,
master - mean (71ms) : 60, 83
. : milestone, 71,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (998ms) : 966, 1031
. : milestone, 998,
master - mean (998ms) : 972, 1023
. : milestone, 998,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (106ms) : 103, 109
. : milestone, 106,
master - mean (107ms) : 102, 112
. : milestone, 107,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (689ms) : 674, 703
. : milestone, 689,
master - mean (691ms) : 668, 714
. : milestone, 691,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (91ms) : 88, 93
. : milestone, 91,
master - mean (90ms) : 87, 92
. : milestone, 90,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (667ms) : 645, 689
. : milestone, 667,
master - mean (660ms) : 638, 682
. : milestone, 660,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (189ms) : 185, 192
. : milestone, 189,
master - mean (189ms) : 186, 192
. : milestone, 189,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (1,131ms) : 1106, 1157
. : milestone, 1131,
master - mean (1,137ms) : 1112, 1161
. : milestone, 1137,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (273ms) : 269, 277
. : milestone, 273,
master - mean (272ms) : 269, 276
. : milestone, 272,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (1,088ms) : 1067, 1110
. : milestone, 1088,
master - mean (1,088ms) : 1062, 1115
. : milestone, 1088,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (4940) - mean (261ms) : 258, 265
. : milestone, 261,
master - mean (262ms) : 259, 264
. : milestone, 262,
section CallTarget+Inlining+NGEN
This PR (4940) - mean (1,058ms) : 1033, 1084
. : milestone, 1058,
master - mean (1,062ms) : 1033, 1091
. : milestone, 1062,
|
Throughput/Crank Report:zap:Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (4940) (11.303M) : 0, 11303327
master (11.333M) : 0, 11332518
benchmarks/2.9.0 (11.258M) : 0, 11257868
section Automatic
This PR (4940) (7.782M) : 0, 7781864
master (7.657M) : 0, 7657438
benchmarks/2.9.0 (8.046M) : 0, 8046105
section Trace stats
This PR (4940) (8.085M) : 0, 8085406
master (8.054M) : 0, 8053668
section Manual
This PR (4940) (9.858M) : 0, 9858481
master (10.004M) : 0, 10003794
section Manual + Automatic
This PR (4940) (7.255M) : 0, 7255122
master (7.437M) : 0, 7436536
section Version Conflict
This PR (4940) (6.490M) : 0, 6489909
master (6.771M) : 0, 6770626
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (4940) (9.601M) : 0, 9600956
master (9.642M) : 0, 9641863
benchmarks/2.9.0 (9.571M) : 0, 9571091
section Automatic
This PR (4940) (6.620M) : 0, 6619773
master (6.503M) : 0, 6503245
section Trace stats
This PR (4940) (6.919M) : 0, 6919412
master (6.987M) : 0, 6987377
section Manual
This PR (4940) (7.934M) : crit ,0, 7934046
master (8.537M) : 0, 8536637
section Manual + Automatic
This PR (4940) (6.109M) : 0, 6108521
master (6.089M) : 0, 6089321
section Version Conflict
This PR (4940) (5.538M) : 0, 5538299
master (5.724M) : 0, 5723643
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (4940) (10.929M) : 0, 10928700
master (10.254M) : 0, 10253707
benchmarks/2.9.0 (8.981M) : 0, 8981100
section Automatic
This PR (4940) (7.718M) : 0, 7717536
master (6.421M) : 0, 6420836
benchmarks/2.9.0 (6.621M) : 0, 6621015
section Trace stats
This PR (4940) (8.048M) : 0, 8048034
master (6.589M) : 0, 6589319
section Manual
This PR (4940) (9.541M) : 0, 9541064
master (7.901M) : 0, 7900787
section Manual + Automatic
This PR (4940) (7.345M) : 0, 7345071
master (6.157M) : 0, 6157174
section Version Conflict
This PR (4940) (6.623M) : 0, 6623004
master (5.598M) : 0, 5598196
gantt
title Throughput Linux x64 (ASM) (Total requests)
dateFormat X
axisFormat %s
section Baseline
master (7.353M) : 0, 7352773
benchmarks/2.9.0 (7.805M) : 0, 7805126
section No attack
master (1.971M) : 0, 1971063
benchmarks/2.9.0 (3.177M) : 0, 3177224
section Attack
master (1.553M) : 0, 1553355
benchmarks/2.9.0 (2.505M) : 0, 2504790
section Blocking
master (3.237M) : 0, 3236693
section IAST default
master (6.655M) : 0, 6654796
section IAST full
master (5.949M) : 0, 5949094
section Base vuln
master (0.974M) : 0, 974472
section IAST vuln
master (0.921M) : 0, 920725
|
Benchmarks Report 🐌Benchmarks for #4940 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower
|
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net6.0 | 1.682 | 65,200.00 | 109,650.00 | multimodal |
Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net6.0 | 1.177 | 60,750.00 | 51,600.00 | multimodal |
Benchmark | Base Allocated | Diff Allocated | Change | Change % |
---|---|---|---|---|
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 | 59.66 KB | 58.52 KB | -1.14 KB | -1.92% |
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 | 65.54 KB | 57.34 KB | -8.19 KB | -12.50% |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
net6.0 | 60.4μs | 777ns | 7.77μs | 0 | 0 | 0 | 43.44 KB |
master | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
netcoreapp3.1 | 57.5μs | 738ns | 7.12μs | 0 | 0 | 0 | 42.64 KB |
master | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
net472 | 37.8μs | 96.9ns | 363ns | 0 | 0 | 0 | 59.66 KB |
master | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
net6.0 | 65.2μs | 319ns | 1.31μs | 0 | 0 | 0 | 43.29 KB |
master | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
netcoreapp3.1 | 97.9μs | 3.06μs | 30.6μs | 0 | 0 | 0 | 42.64 KB |
master | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
net472 | 57.3μs | 902ns | 8.84μs | 0 | 0 | 0 | 65.54 KB |
#4940 | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
net6.0 | 51.5μs | 194ns | 726ns | 0 | 0 | 0 | 43.44 KB |
#4940 | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
netcoreapp3.1 | 53.1μs | 244ns | 914ns | 0 | 0 | 0 | 42.64 KB |
#4940 | StringConcatBenchmark(parameters=Syste(...)ring] [48]) |
net472 | 37.9μs | 122ns | 455ns | 0 | 0 | 0 | 58.52 KB |
#4940 | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
net6.0 | 105μs | 2.98μs | 29.8μs | 0 | 0 | 0 | 43.29 KB |
#4940 | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
netcoreapp3.1 | 89.7μs | 3.19μs | 31.7μs | 0 | 0 | 0 | 42.64 KB |
#4940 | StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) |
net472 | 63.6μs | 324ns | 1.45μs | 0 | 0 | 0 | 57.34 KB |
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 1.52μs | 0.591ns | 2.29ns | 0.0221 | 0 | 0 | 1.57 KB |
master | EnrichedLog |
netcoreapp3.1 | 2.17μs | 0.943ns | 3.53ns | 0.0208 | 0 | 0 | 1.57 KB |
master | EnrichedLog |
net472 | 2.66μs | 1.07ns | 3.85ns | 0.238 | 0 | 0 | 1.5 KB |
#4940 | EnrichedLog |
net6.0 | 1.48μs | 0.722ns | 2.7ns | 0.022 | 0 | 0 | 1.57 KB |
#4940 | EnrichedLog |
netcoreapp3.1 | 2.21μs | 1.45ns | 5.43ns | 0.0209 | 0 | 0 | 1.57 KB |
#4940 | EnrichedLog |
net472 | 2.56μs | 2.45ns | 9.5ns | 0.238 | 0 | 0 | 1.5 KB |
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 113μs | 140ns | 542ns | 0.0563 | 0 | 0 | 4.21 KB |
master | EnrichedLog |
netcoreapp3.1 | 119μs | 296ns | 1.15μs | 0.0587 | 0 | 0 | 4.21 KB |
master | EnrichedLog |
net472 | 149μs | 277ns | 1.07μs | 0.667 | 0.222 | 0 | 4.39 KB |
#4940 | EnrichedLog |
net6.0 | 112μs | 128ns | 494ns | 0.0561 | 0 | 0 | 4.21 KB |
#4940 | EnrichedLog |
netcoreapp3.1 | 117μs | 259ns | 1μs | 0 | 0 | 0 | 4.21 KB |
#4940 | EnrichedLog |
net472 | 148μs | 147ns | 568ns | 0.664 | 0.221 | 0 | 4.39 KB |
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 3.03μs | 1.11ns | 4.28ns | 0.0288 | 0 | 0 | 2.13 KB |
master | EnrichedLog |
netcoreapp3.1 | 4.13μs | 1.76ns | 6.81ns | 0.0288 | 0 | 0 | 2.13 KB |
master | EnrichedLog |
net472 | 4.89μs | 3.71ns | 14.4ns | 0.308 | 0 | 0 | 1.95 KB |
#4940 | EnrichedLog |
net6.0 | 2.97μs | 1.24ns | 4.63ns | 0.0295 | 0 | 0 | 2.13 KB |
#4940 | EnrichedLog |
netcoreapp3.1 | 4.15μs | 1.87ns | 7.26ns | 0.0287 | 0 | 0 | 2.13 KB |
#4940 | EnrichedLog |
net472 | 4.91μs | 10.6ns | 41.2ns | 0.308 | 0 | 0 | 1.95 KB |
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | SendReceive |
net6.0 | 1.33μs | 0.84ns | 3.14ns | 0.0158 | 0 | 0 | 1.1 KB |
master | SendReceive |
netcoreapp3.1 | 1.78μs | 0.908ns | 3.52ns | 0.0152 | 0 | 0 | 1.1 KB |
master | SendReceive |
net472 | 2.01μs | 1.22ns | 4.72ns | 0.177 | 0 | 0 | 1.12 KB |
#4940 | SendReceive |
net6.0 | 1.32μs | 1.17ns | 4.52ns | 0.0151 | 0 | 0 | 1.1 KB |
#4940 | SendReceive |
netcoreapp3.1 | 1.75μs | 1.03ns | 3.99ns | 0.0148 | 0 | 0 | 1.1 KB |
#4940 | SendReceive |
net472 | 2.18μs | 2.27ns | 8.77ns | 0.177 | 0 | 0 | 1.12 KB |
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 2.73μs | 3.2ns | 12.4ns | 0.0204 | 0 | 0 | 1.53 KB |
master | EnrichedLog |
netcoreapp3.1 | 3.87μs | 1.41ns | 4.89ns | 0.0214 | 0 | 0 | 1.58 KB |
master | EnrichedLog |
net472 | 4.33μs | 1.26ns | 4.87ns | 0.312 | 0 | 0 | 1.97 KB |
#4940 | EnrichedLog |
net6.0 | 2.7μs | 0.725ns | 2.81ns | 0.0215 | 0 | 0 | 1.53 KB |
#4940 | EnrichedLog |
netcoreapp3.1 | 4.06μs | 2.28ns | 8.83ns | 0.0204 | 0 | 0 | 1.58 KB |
#4940 | EnrichedLog |
net472 | 4.28μs | 0.848ns | 3.28ns | 0.31 | 0 | 0 | 1.97 KB |
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️
Slower ⚠️ in #4940
Benchmark
diff/base
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0
1.339
461.31
617.61
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472
1.130
728.43
823.26
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 | 1.339 | 461.31 | 617.61 | |
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 | 1.130 | 728.43 | 823.26 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | StartFinishSpan |
net6.0 | 461ns | 0.401ns | 1.55ns | 0.00759 | 0 | 0 | 536 B |
master | StartFinishSpan |
netcoreapp3.1 | 694ns | 0.455ns | 1.64ns | 0.00704 | 0 | 0 | 536 B |
master | StartFinishSpan |
net472 | 728ns | 0.915ns | 3.42ns | 0.0854 | 0 | 0 | 538 B |
master | StartFinishScope |
net6.0 | 572ns | 0.705ns | 2.73ns | 0.00908 | 0 | 0 | 656 B |
master | StartFinishScope |
netcoreapp3.1 | 857ns | 0.47ns | 1.82ns | 0.00864 | 0 | 0 | 656 B |
master | StartFinishScope |
net472 | 1.05μs | 1.32ns | 5.09ns | 0.0977 | 0 | 0 | 618 B |
#4940 | StartFinishSpan |
net6.0 | 618ns | 0.389ns | 1.51ns | 0.00744 | 0 | 0 | 536 B |
#4940 | StartFinishSpan |
netcoreapp3.1 | 725ns | 0.174ns | 0.651ns | 0.00723 | 0 | 0 | 536 B |
#4940 | StartFinishSpan |
net472 | 825ns | 1.13ns | 4.39ns | 0.0853 | 0 | 0 | 538 B |
#4940 | StartFinishScope |
net6.0 | 528ns | 0.136ns | 0.527ns | 0.00923 | 0 | 0 | 656 B |
#4940 | StartFinishScope |
netcoreapp3.1 | 888ns | 0.396ns | 1.48ns | 0.00873 | 0 | 0 | 656 B |
#4940 | StartFinishScope |
net472 | 985ns | 0.704ns | 2.72ns | 0.0978 | 0 | 0 | 618 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️
Faster 🎉 in #4940
Benchmark
base/diff
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0
1.345
834.89
620.96
Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 | 1.345 | 834.89 | 620.96 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | RunOnMethodBegin |
net6.0 | 836ns | 1.11ns | 4.3ns | 0.00929 | 0 | 0 | 656 B |
master | RunOnMethodBegin |
netcoreapp3.1 | 988ns | 0.798ns | 3.09ns | 0.00908 | 0 | 0 | 656 B |
master | RunOnMethodBegin |
net472 | 1.16μs | 1.24ns | 4.81ns | 0.098 | 0 | 0 | 618 B |
#4940 | RunOnMethodBegin |
net6.0 | 621ns | 0.182ns | 0.705ns | 0.00908 | 0 | 0 | 656 B |
#4940 | RunOnMethodBegin |
netcoreapp3.1 | 929ns | 0.387ns | 1.45ns | 0.0085 | 0 | 0 | 656 B |
#4940 | RunOnMethodBegin |
net472 | 1.06μs | 0.223ns | 0.862ns | 0.0978 | 0 | 0 | 618 B |
Summary of changes
Use the value of
DD_TRACE_BATCH_INTERVAL
in the AgentWriter to change the batch interval in the serialization thread.Also, allow to completely disable the delay.
Reason for change
The AWS Lambda waits on the traces to be flushed before finishing the invocation, so the delay is getting in the way.
Implementation details
Fun fact, the setting existed from the very beginning but apparently I forgot to pass the value to the AgentWriter :awkward-monkey:
Test coverage
One could argue that the setting would have been implemented properly if I added tests to begin with. But I can't really think of a good way to test this.