Revert "[ASM] New marshalling system for Waf.Run calls to improve spe… #4891

robertpi · 2023-11-18T13:33:15Z

Summary of changes

Roll back to old non-pooled encoder.

datadog-ddstaging · 2023-11-18T13:57:42Z

Datadog Report

Branch report: robert/asm/revert-new-encoder
Commit report: de21fc9

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 301850 Passed, 977 Skipped, 5 Regressions, 26m 17.71s Wall Time

⌛ Performance Regressions vs Default Branch (5)

AllCycleSimpleBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 49.99µs (+3.1µs, +7%) - Details
AllCycleMoreComplexBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 238.41µs (+26.43µs, +12%) - Details
ObjectExtractorMoreComplexBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 3.06µs (+303ns, +11%) - Details
AllCycleMoreComplexBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 225.03µs (+16.5µs, +8%) - Details
Profiler_liveheap - scenarios 3.77s (+392.66ms, +12%) - Details

andrewlock · 2023-11-18T14:00:16Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (69ms)  : 62, 76
     .   : milestone, 69,
    master - mean (70ms)  : 64, 77
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (996ms)  : 974, 1018
     .   : milestone, 996,
    master - mean (997ms)  : 975, 1018
     .   : milestone, 997,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (105ms)  : 102, 108
     .   : milestone, 105,
    master - mean (106ms)  : 103, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (686ms)  : 669, 704
     .   : milestone, 686,
    master - mean (691ms)  : 676, 706
     .   : milestone, 691,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (89ms)  : 87, 92
     .   : milestone, 89,
    master - mean (90ms)  : 87, 93
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (660ms)  : 634, 685
     .   : milestone, 660,
    master - mean (661ms)  : 636, 686
     .   : milestone, 661,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (188ms)  : 184, 191
     .   : milestone, 188,
    master - mean (189ms)  : 186, 191
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (1,133ms)  : 1107, 1159
     .   : milestone, 1133,
    master - mean (1,126ms)  : 1105, 1147
     .   : milestone, 1126,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (272ms)  : 266, 277
     .   : milestone, 272,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (1,097ms)  : 1073, 1122
     .   : milestone, 1097,
    master - mean (1,099ms)  : 1068, 1131
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4891) - mean (261ms)  : 256, 265
     .   : milestone, 261,
    master - mean (262ms)  : 258, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4891) - mean (1,057ms)  : 1036, 1078
     .   : milestone, 1057,
    master - mean (1,060ms)  : 1035, 1086
     .   : milestone, 1060,

andrewlock · 2023-11-18T15:08:42Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4891) (10.893M)   : 0, 10893307
    master (10.858M)   : 0, 10857649
    benchmarks/2.9.0 (11.283M)   : 0, 11283028

    section Automatic
    This PR (4891) (7.469M)   : 0, 7468791
    master (7.528M)   : 0, 7528231
    benchmarks/2.9.0 (8.174M)   : 0, 8173604

    section Trace stats
    This PR (4891) (7.823M)   : 0, 7823029
    master (7.804M)   : 0, 7803650

    section Manual
    This PR (4891) (9.357M)   : 0, 9356681
    master (9.609M)   : 0, 9609283

    section Manual + Automatic
    This PR (4891) (7.211M)   : 0, 7211196
    master (7.137M)   : 0, 7136766

    section Version Conflict
    This PR (4891) (6.391M)   : 0, 6391216
    master (6.389M)   : 0, 6389302

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4891) (9.617M)   : 0, 9616931
    master (9.616M)   : 0, 9615519
    benchmarks/2.9.0 (9.589M)   : 0, 9589125

    section Automatic
    This PR (4891) (6.588M)   : 0, 6587578
    master (6.631M)   : 0, 6631296

    section Trace stats
    This PR (4891) (6.972M)   : 0, 6972308
    master (6.961M)   : 0, 6960530

    section Manual
    This PR (4891) (8.286M)   : 0, 8285933
    master (8.247M)   : 0, 8246632

    section Manual + Automatic
    This PR (4891) (6.026M)   : 0, 6026259
    master (6.112M)   : 0, 6111650

    section Version Conflict
    This PR (4891) (5.641M)   : 0, 5640946
    master (5.704M)   : 0, 5704170

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4891) (10.614M)   : 0, 10614499
    master (10.684M)   : 0, 10683600
    benchmarks/2.9.0 (10.855M)   : 0, 10855303

    section Automatic
    This PR (4891) (7.664M)   : 0, 7664101
    master (7.649M)   : 0, 7649198
    benchmarks/2.9.0 (7.860M)   : 0, 7859587

    section Trace stats
    This PR (4891) (7.974M)   : 0, 7974303
    master (7.827M)   : 0, 7827159

    section Manual
    This PR (4891) (9.298M)   : 0, 9298104
    master (9.309M)   : 0, 9309400

    section Manual + Automatic
    This PR (4891) (7.370M)   : 0, 7370436
    master (7.098M)   : 0, 7097770

    section Version Conflict
    This PR (4891) (6.562M)   : 0, 6561758
    master (6.440M)   : 0, 6439867

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4891) (7.508M)   : 0, 7507572
    master (7.328M)   : 0, 7328497
    benchmarks/2.9.0 (7.720M)   : 0, 7719760

    section No attack
    This PR (4891) (1.972M)   : crit ,0, 1972410
    master (2.111M)   : 0, 2111005
    benchmarks/2.9.0 (3.147M)   : 0, 3147138

    section Attack
    This PR (4891) (1.567M)   : crit ,0, 1567296
    master (1.669M)   : 0, 1668613
    benchmarks/2.9.0 (2.525M)   : 0, 2524657

    section Blocking
    This PR (4891) (3.274M)   : crit ,0, 3273916
    master (3.477M)   : 0, 3476656

    section IAST default
    This PR (4891) (6.548M)   : 0, 6548455
    master (6.749M)   : 0, 6749076

    section IAST full
    This PR (4891) (5.796M)   : 0, 5795747
    master (5.901M)   : 0, 5901130

    section Base vuln
    This PR (4891) (0.940M)   : 0, 940188
    master (0.955M)   : 0, 954974

    section IAST vuln
    This PR (4891) (0.874M)   : 0, 873659
    master (0.837M)   : 0, 836978

andrewlock · 2023-11-18T16:24:44Z

Benchmarks Report 🐌

Benchmarks for #4891 compared to master:

1 benchmarks are faster, with geometric mean 1.125
23 benchmarks are slower, with geometric mean 2.118
3 benchmarks have fewer allocations
21 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.69μs	46.7ns	284ns	0.0289	0.0124	0	7.47 KB
master	`StartStopWithChild`	netcoreapp3.1	10.7μs	48.9ns	208ns	0.0267	0.0107	0	7.57 KB
master	`StartStopWithChild`	net472	17μs	46.4ns	180ns	1.31	0.313	0.0846	7.94 KB
#4891	`StartStopWithChild`	net6.0	8.54μs	48.2ns	341ns	0.0209	0.00835	0	7.48 KB
#4891	`StartStopWithChild`	netcoreapp3.1	10.5μs	50.9ns	222ns	0.0204	0.0102	0	7.58 KB
#4891	`StartStopWithChild`	net472	16.9μs	62.8ns	243ns	1.35	0.374	0.125	7.95 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	474μs	227ns	877ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	637μs	222ns	860ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	791μs	313ns	1.21μs	0.393	3.3 KB
#4891	`WriteAndFlushEnrichedTraces`	net6.0	456μs	497ns	1.92μs	0	2.7 KB
#4891	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	227ns	880ns	0	2.7 KB
#4891	`WriteAndFlushEnrichedTraces`	net472	793μs	515ns	2μs	0.396	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #4891

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472	1.121	212,924.73	238,745.57
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472	1.120	3,617.80	4,052.28

More allocations ⚠️ in #4891

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑netcoreapp3.1	8.26 KB	9.14 KB	880 B	10.66%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0	8.37 KB	9.25 KB	880 B	10.52%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472	8.43 KB	9.32 KB	883 B	10.47%

Fewer allocations 🎉 in #4891

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	2.07 KB	1.81 KB	-256 B	-12.37%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0	2.02 KB	1.77 KB	-256 B	-12.65%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑netcoreapp3.1	2 KB	1.74 KB	-256 B	-12.80%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	43.6μs	34.2ns	132ns	0.0217	0	2.02 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	45.3μs	182ns	705ns	0.0226	0	2 KB
master	`AllCycleSimpleBody`	net472	47.1μs	173ns	668ns	0.329	0	2.07 KB
master	`AllCycleMoreComplexBody`	net6.0	205μs	60.6ns	235ns	0.102	0	8.37 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	209μs	79ns	296ns	0.104	0	8.26 KB
master	`AllCycleMoreComplexBody`	net472	213μs	89.6ns	347ns	1.27	0	8.43 KB
master	`ObjectExtractorSimpleBody`	net6.0	129ns	0.05ns	0.18ns	0.00393	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	196ns	0.071ns	0.275ns	0.00375	0	272 B
master	`ObjectExtractorSimpleBody`	net472	152ns	0.104ns	0.404ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.75μs	1.29ns	4.83ns	0.0523	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.79μs	16.7ns	64.6ns	0.05	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.62μs	2.86ns	10.7ns	0.602	0.00554	3.8 KB
#4891	`AllCycleSimpleBody`	net6.0	44.5μs	133ns	705ns	0.0221	0	1.77 KB
#4891	`AllCycleSimpleBody`	netcoreapp3.1	46.7μs	28.2ns	109ns	0.0234	0	1.74 KB
#4891	`AllCycleSimpleBody`	net472	48.7μs	89.1ns	345ns	0.265	0	1.81 KB
#4891	`AllCycleMoreComplexBody`	net6.0	216μs	45.4ns	157ns	0.107	0	9.25 KB
#4891	`AllCycleMoreComplexBody`	netcoreapp3.1	224μs	285ns	1.1μs	0.113	0	9.14 KB
#4891	`AllCycleMoreComplexBody`	net472	239μs	78.3ns	303ns	1.44	0	9.32 KB
#4891	`ObjectExtractorSimpleBody`	net6.0	133ns	0.0536ns	0.201ns	0.00396	0	280 B
#4891	`ObjectExtractorSimpleBody`	netcoreapp3.1	188ns	0.0542ns	0.196ns	0.00378	0	272 B
#4891	`ObjectExtractorSimpleBody`	net472	155ns	0.121ns	0.469ns	0.0446	0	281 B
#4891	`ObjectExtractorMoreComplexBody`	net6.0	2.81μs	3.06ns	11.8ns	0.0532	0	3.78 KB
#4891	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.72μs	1.25ns	4.69ns	0.0502	0	3.69 KB
#4891	`ObjectExtractorMoreComplexBody`	net472	4.05μs	1.86ns	6.97ns	0.603	0.00609	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #4891

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net6.0	4.315	24,169.79	104,300.34
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑net6.0	3.907	23,904.71	93,389.79
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net6.0	3.561	13,731.60	48,895.28
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑netcoreapp3.1	3.516	20,062.96	70,537.21
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑netcoreapp3.1	3.410	39,680.12	135,327.94
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net472	3.264	30,451.18	99,400.67
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑net472	3.233	58,863.77	190,332.69
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net472	3.210	59,245.39	190,176.21	bimodal
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑netcoreapp3.1	3.185	39,519.77	125,878.18
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafWithAttack(args: NestedMap (100, attack))‑net472	2.051	132,490.16	271,727.60

More allocations ⚠️ in #4891

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑netcoreapp3.1	9.41 KB	16.06 KB	6.65 KB	70.66%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net6.0	9.42 KB	16.06 KB	6.65 KB	70.60%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net472	9.47 KB	16.14 KB	6.67 KB	70.42%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net6.0	19.66 KB	32.76 KB	13.1 KB	66.67%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑net6.0	19.39 KB	32.18 KB	12.79 KB	65.97%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑netcoreapp3.1	20.41 KB	33.33 KB	12.92 KB	63.31%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net472	20.62 KB	33.67 KB	13.05 KB	63.27%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑net472	20.03 KB	32.63 KB	12.61 KB	62.95%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (20))‑netcoreapp3.1	19.83 KB	32.3 KB	12.46 KB	62.85%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafWithAttack(args: NestedMap (100, attack))‑net6.0	26 KB	39.1 KB	13.1 KB	50.40%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	13.7μs	3.67ns	13.2ns	0.13	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	20.1μs	8.65ns	31.2ns	0.13	0	9.41 KB
master	`RunWaf(args=NestedMap (10))`	net472	30.4μs	15.4ns	59.6ns	1.49	0	9.47 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	78μs	33.6ns	130ns	0.192	0	15.76 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	87.1μs	187ns	724ns	0.174	0	15.71 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	100μs	55.6ns	192ns	2.51	0	16.03 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	24.2μs	9.45ns	36.6ns	0.277	0	19.66 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	39.5μs	10.8ns	40.3ns	0.277	0	20.41 KB
master	`RunWaf(args=NestedMap (100))`	net472	59.2μs	22.2ns	86.1ns	3.25	0.0591	20.62 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	92μs	28.9ns	108ns	0.373	0	26 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	111μs	235ns	909ns	0.334	0	26.71 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	133μs	138ns	536ns	4.28	0.0658	27.19 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	23.9μs	8.99ns	34.8ns	0.273	0	19.39 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	39.7μs	12.7ns	49ns	0.257	0	19.83 KB
master	`RunWaf(args=NestedMap (20))`	net472	58.8μs	18.6ns	67.2ns	3.17	0.0588	20.03 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	91μs	27.9ns	104ns	0.32	0	25.74 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	110μs	186ns	720ns	0.336	0	26.14 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	132μs	64ns	248ns	4.2	0.0657	26.59 KB
#4891	`RunWaf(args=NestedMap (10))`	net6.0	48.9μs	15.9ns	57.3ns	0.22	0	16.06 KB
#4891	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	69.9μs	364ns	1.86μs	0.206	0	16.06 KB
#4891	`RunWaf(args=NestedMap (10))`	net472	99.1μs	303ns	1.13μs	2.53	0.0938	16.14 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	114μs	547ns	2.12μs	0.3	0	22.41 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	134μs	125ns	450ns	0.267	0	22.36 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	168μs	83.5ns	323ns	3.58	0.166	22.7 KB
#4891	`RunWaf(args=NestedMap (100))`	net6.0	104μs	150ns	541ns	0.419	0	32.76 KB
#4891	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	126μs	129ns	482ns	0.409	0	33.33 KB
#4891	`RunWaf(args=NestedMap (100))`	net472	188μs	945ns	4.01μs	5.32	0.367	33.67 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	166μs	875ns	4.63μs	0.494	0	39.1 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	209μs	722ns	2.8μs	0.522	0	39.63 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	268μs	1.26μs	5.21μs	6.38	0.521	40.23 KB
#4891	`RunWaf(args=NestedMap (20))`	net6.0	93.4μs	24.6ns	95.1ns	0.423	0	32.18 KB
#4891	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	136μs	281ns	1.09μs	0.395	0	32.3 KB
#4891	`RunWaf(args=NestedMap (20))`	net472	190μs	85.8ns	332ns	5.19	0.37	32.63 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	176μs	139ns	538ns	0.527	0	38.53 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	213μs	423ns	1.64μs	0.521	0	38.6 KB
#4891	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	262μs	141ns	546ns	6.11	0.52	39.2 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	171μs	180ns	698ns	0.256	18.24 KB
master	`SendRequest`	netcoreapp3.1	191μs	275ns	1.03μs	0.193	20.4 KB
master	`SendRequest`	net472	0.00142ns	0.000374ns	0.00145ns	0	0 b
#4891	`SendRequest`	net6.0	173μs	99.4ns	372ns	0.26	18.24 KB
#4891	`SendRequest`	netcoreapp3.1	193μs	359ns	1.39μs	0.191	20.4 KB
#4891	`SendRequest`	net472	0.000208ns	0.000152ns	0.00057ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	532μs	515ns	1.99μs	0.539	0	0	41.58 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	652μs	601ns	2.33μs	0.327	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	3.92μs	15.2μs	8.39	2.52	0.419	53.24 KB
#4891	`WriteAndFlushEnrichedTraces`	net6.0	529μs	255ns	954ns	0.546	0	0	41.54 KB
#4891	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	671μs	1.08μs	4.19μs	0.326	0	0	41.69 KB
#4891	`WriteAndFlushEnrichedTraces`	net472	841μs	3.46μs	13.4μs	8.39	2.52	0.419	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.12μs	0.767ns	2.97ns	0.0106	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.49μs	1.32ns	5.1ns	0.0103	768 B
master	`ExecuteNonQuery`	net472	1.69μs	0.447ns	1.73ns	0.116	730 B
#4891	`ExecuteNonQuery`	net6.0	1.15μs	0.448ns	1.73ns	0.0109	768 B
#4891	`ExecuteNonQuery`	netcoreapp3.1	1.43μs	1.12ns	4.19ns	0.00999	768 B
#4891	`ExecuteNonQuery`	net472	1.71μs	0.675ns	2.34ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.13μs	0.51ns	1.91ns	0.013	0	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	1.03ns	3.99ns	0.0123	0	936 B
master	`CallElasticsearch`	net472	2.56μs	0.815ns	3.16ns	0.151	0.00128	955 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	0.25ns	0.937ns	0.0124	0	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.55μs	1.25ns	4.85ns	0.0134	0	984 B
master	`CallElasticsearchAsync`	net472	2.77μs	0.536ns	2.07ns	0.16	0	1.01 KB
#4891	`CallElasticsearch`	net6.0	1.22μs	0.584ns	2.18ns	0.0127	0	936 B
#4891	`CallElasticsearch`	netcoreapp3.1	1.46μs	2.54ns	9.82ns	0.0126	0	936 B
#4891	`CallElasticsearch`	net472	2.71μs	5.97ns	23.1ns	0.151	0	955 B
#4891	`CallElasticsearchAsync`	net6.0	1.25μs	0.955ns	3.57ns	0.0125	0	912 B
#4891	`CallElasticsearchAsync`	netcoreapp3.1	1.59μs	0.687ns	2.57ns	0.0134	0	984 B
#4891	`CallElasticsearchAsync`	net472	2.7μs	0.879ns	3.29ns	0.161	0	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.3μs	0.575ns	2.15ns	0.013	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.52μs	0.328ns	1.23ns	0.0123	912 B
master	`ExecuteAsync`	net472	1.85μs	0.872ns	3.38ns	0.138	875 B
#4891	`ExecuteAsync`	net6.0	1.28μs	0.972ns	3.77ns	0.0127	912 B
#4891	`ExecuteAsync`	netcoreapp3.1	1.55μs	0.722ns	2.6ns	0.0119	912 B
#4891	`ExecuteAsync`	net472	1.86μs	0.428ns	1.6ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.03μs	2.46ns	8.85ns	0.0282	2.1 KB
master	`SendAsync`	netcoreapp3.1	5.06μs	1.73ns	6.23ns	0.0355	2.63 KB
master	`SendAsync`	net472	7.8μs	1.91ns	7.16ns	0.524	3.31 KB
#4891	`SendAsync`	net6.0	4μs	1.27ns	4.92ns	0.0278	2.1 KB
#4891	`SendAsync`	netcoreapp3.1	5.02μs	2.42ns	9.38ns	0.0349	2.63 KB
#4891	`SendAsync`	net472	7.79μs	2.67ns	10.4ns	0.523	3.31 KB

Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4891

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0	1.125	1,593.36	1,416.25

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.59μs	0.692ns	2.59ns	0.0216	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	2.07μs	0.551ns	1.99ns	0.0209	1.57 KB
master	`EnrichedLog`	net472	2.51μs	1.53ns	5.91ns	0.238	1.5 KB
#4891	`EnrichedLog`	net6.0	1.41μs	0.783ns	3.03ns	0.022	1.57 KB
#4891	`EnrichedLog`	netcoreapp3.1	2.25μs	1.15ns	4.47ns	0.0213	1.57 KB
#4891	`EnrichedLog`	net472	2.53μs	2.22ns	8.6ns	0.237	1.5 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	71.1ns	276ns	0	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	180ns	675ns	0.0584	0	4.21 KB
master	`EnrichedLog`	net472	149μs	241ns	934ns	0.662	0.221	4.39 KB
#4891	`EnrichedLog`	net6.0	114μs	91.7ns	343ns	0.0575	0	4.21 KB
#4891	`EnrichedLog`	netcoreapp3.1	119μs	247ns	924ns	0	0	4.21 KB
#4891	`EnrichedLog`	net472	147μs	110ns	427ns	0.664	0.221	4.39 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.14μs	1.26ns	4.71ns	0.0297	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	4.13μs	1.36ns	5.28ns	0.0272	2.13 KB
master	`EnrichedLog`	net472	4.99μs	2.21ns	8.58ns	0.309	1.95 KB
#4891	`EnrichedLog`	net6.0	2.97μs	1.62ns	6.28ns	0.0298	2.13 KB
#4891	`EnrichedLog`	netcoreapp3.1	4.31μs	22.7ns	118ns	0.028	2.13 KB
#4891	`EnrichedLog`	net472	4.8μs	1.92ns	7.19ns	0.308	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.29μs	0.373ns	1.29ns	0.0155	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.77μs	0.901ns	3.49ns	0.015	1.1 KB
master	`SendReceive`	net472	2.19μs	2.1ns	8.13ns	0.176	1.12 KB
#4891	`SendReceive`	net6.0	1.36μs	0.557ns	2.01ns	0.0156	1.1 KB
#4891	`SendReceive`	netcoreapp3.1	1.71μs	0.624ns	2.33ns	0.0145	1.1 KB
#4891	`SendReceive`	net472	2.13μs	3.09ns	12ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.69μs	2.09ns	8.11ns	0.0214	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	3.98ns	15.4ns	0.0198	1.58 KB
master	`EnrichedLog`	net472	4.41μs	1.07ns	4.13ns	0.311	1.97 KB
#4891	`EnrichedLog`	net6.0	2.68μs	0.876ns	3.39ns	0.0214	1.53 KB
#4891	`EnrichedLog`	netcoreapp3.1	3.8μs	5.22ns	19.5ns	0.0209	1.58 KB
#4891	`EnrichedLog`	net472	4.5μs	1.5ns	5.63ns	0.311	1.97 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4891

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.271	528.78	672.31
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.149	456.75	524.74
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.116	755.33	842.67

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	457ns	0.125ns	0.483ns	0.00752	536 B
master	`StartFinishSpan`	netcoreapp3.1	687ns	0.425ns	1.65ns	0.00726	536 B
master	`StartFinishSpan`	net472	755ns	0.274ns	1.06ns	0.0854	538 B
master	`StartFinishScope`	net6.0	531ns	0.929ns	3.6ns	0.0091	656 B
master	`StartFinishScope`	netcoreapp3.1	852ns	1.16ns	4.5ns	0.00845	656 B
master	`StartFinishScope`	net472	993ns	1.04ns	4.02ns	0.0982	618 B
#4891	`StartFinishSpan`	net6.0	525ns	0.236ns	0.914ns	0.00766	536 B
#4891	`StartFinishSpan`	netcoreapp3.1	649ns	0.247ns	0.926ns	0.00728	536 B
#4891	`StartFinishSpan`	net472	842ns	0.305ns	1.18ns	0.0854	538 B
#4891	`StartFinishScope`	net6.0	672ns	0.339ns	1.31ns	0.0091	656 B
#4891	`StartFinishScope`	netcoreapp3.1	842ns	0.304ns	1.18ns	0.00893	656 B
#4891	`StartFinishScope`	net472	1.04μs	0.283ns	1.09ns	0.098	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	630ns	0.0964ns	0.347ns	0.00909	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	964ns	0.534ns	2ns	0.0088	656 B
master	`RunOnMethodBegin`	net472	1.18μs	1.21ns	4.69ns	0.0981	618 B
#4891	`RunOnMethodBegin`	net6.0	653ns	0.25ns	0.968ns	0.00933	656 B
#4891	`RunOnMethodBegin`	netcoreapp3.1	1.03μs	1.23ns	4.75ns	0.00889	656 B
#4891	`RunOnMethodBegin`	net472	1.2μs	0.594ns	2.3ns	0.098	618 B

…ed and reduce allocations (#4302)" This reverts commit 096f106.

robertpi requested review from a team as code owners November 18, 2023 13:33

daniel-romano-DD approved these changes Nov 20, 2023

View reviewed changes

anna-git approved these changes Nov 20, 2023

View reviewed changes

kevingosse approved these changes Nov 20, 2023

View reviewed changes

robertpi added 2 commits November 20, 2023 15:02

Revert "[ASM] New marshalling system for Waf.Run calls to improve spe…

d4820ad

…ed and reduce allocations (#4302)" This reverts commit 096f106.

fix schema test by syncing with master

de21fc9

robertpi force-pushed the robert/asm/revert-new-encoder branch from 067db23 to de21fc9 Compare November 20, 2023 14:03

robertpi merged commit 457af03 into master Nov 20, 2023
51 of 54 checks passed

robertpi deleted the robert/asm/revert-new-encoder branch November 20, 2023 15:44

github-actions bot added this to the vNext milestone Nov 20, 2023

andrewlock added type:bug area:asm labels Nov 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "[ASM] New marshalling system for Waf.Run calls to improve spe… #4891

Revert "[ASM] New marshalling system for Waf.Run calls to improve spe… #4891

robertpi commented Nov 18, 2023

datadog-ddstaging bot commented Nov 18, 2023 •

edited

Loading

andrewlock commented Nov 18, 2023 •

edited

Loading

andrewlock commented Nov 18, 2023 •

edited

Loading

andrewlock commented Nov 18, 2023

Raw results

Raw results

Slower ⚠️ in #4891

More allocations ⚠️ in #4891

Fewer allocations 🎉 in #4891

Raw results

Slower ⚠️ in #4891

More allocations ⚠️ in #4891

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4891

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4891

Raw results

Raw results

Revert "[ASM] New marshalling system for Waf.Run calls to improve spe… #4891

Revert "[ASM] New marshalling system for Waf.Run calls to improve spe… #4891

Conversation

robertpi commented Nov 18, 2023

Summary of changes

datadog-ddstaging bot commented Nov 18, 2023 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (5)

andrewlock commented Nov 18, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Nov 18, 2023 • edited Loading

Throughput/Crank Report:zap:

andrewlock commented Nov 18, 2023

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Slower ⚠️ in #4891

More allocations ⚠️ in #4891

Fewer allocations 🎉 in #4891

Raw results

Slower ⚠️ in #4891

More allocations ⚠️ in #4891

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4891

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4891

Raw results

Raw results

datadog-ddstaging bot commented Nov 18, 2023 •

edited

Loading

andrewlock commented Nov 18, 2023 •

edited

Loading

andrewlock commented Nov 18, 2023 •

edited

Loading