Improve lib-injection tagging script #5598

andrewlock · 2024-05-22T13:56:58Z

Summary of changes

Use a more restrictive regex when searching for tags for lib-injection

Reason for change

Suggested here - not necessary for us as we only have vX.Y.Z-pre tags, but preferable for other libraries, and we want to keep the scripts in sync if possible

Implementation details

Replace grep -v '-' (i.e. reject any tags with a -) with grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' (explicit Regex)

Test coverage

Re-ran all the checks from #5544 and looks good

Other details

Original implementation in

Update lib-injection docker image tags #5544

This reverts commit d5dc44d.

datadog-ddstaging · 2024-05-22T14:13:13Z

Datadog Report

Branch report: andrew/improve-script
Commit report: 0807a53
Test service: dd-trace-dotnet

✅ 0 Failed, 294487 Passed, 1429 Skipped, 13h 25m 10.39s Total Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

Baseline_cpu - scenarios 2.39s (+50.7ms, +2%) - Details

andrewlock · 2024-05-22T14:21:48Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (75ms)  : 65, 85
     .   : milestone, 75,
    master - mean (70ms)  : 66, 74
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (1,000ms)  : 970, 1030
     .   : milestone, 1000,
    master - mean (1,002ms)  : 976, 1028
     .   : milestone, 1002,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (110ms)  : 107, 114
     .   : milestone, 110,
    master - mean (108ms)  : 104, 111
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (729ms)  : 698, 759
     .   : milestone, 729,
    master - mean (720ms)  : 683, 758
     .   : milestone, 720,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (94ms)  : 91, 98
     .   : milestone, 94,
    master - mean (92ms)  : 89, 95
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (675ms)  : 649, 701
     .   : milestone, 675,
    master - mean (672ms)  : 641, 703
     .   : milestone, 672,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (193ms)  : 190, 195
     .   : milestone, 193,
    master - mean (192ms)  : 188, 196
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (1,111ms)  : 1089, 1132
     .   : milestone, 1111,
    master - mean (1,105ms)  : 1088, 1122
     .   : milestone, 1105,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (280ms)  : 274, 286
     .   : milestone, 280,
    master - mean (276ms)  : 272, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (908ms)  : 886, 929
     .   : milestone, 908,
    master - mean (903ms)  : 877, 929
     .   : milestone, 903,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5598) - mean (268ms)  : 264, 271
     .   : milestone, 268,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5598) - mean (891ms)  : 865, 916
     .   : milestone, 891,
    master - mean (881ms)  : 851, 910
     .   : milestone, 881,

andrewlock · 2024-05-22T14:59:32Z

Benchmarks Report for tracer 🐌

Benchmarks for #5598 compared to master:

1 benchmarks are faster, with geometric mean 1.119
1 benchmarks are slower, with geometric mean 1.155
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.7μs	47ns	266ns	0.0226	0.00904	0	7.56 KB
master	`StartStopWithChild`	netcoreapp3.1	10.7μs	56.9ns	301ns	0.0261	0.0104	0	7.66 KB
master	`StartStopWithChild`	net472	17.2μs	63.5ns	246ns	1.36	0.39	0.116	8.06 KB
#5598	`StartStopWithChild`	net6.0	8.79μs	44.7ns	214ns	0.0208	0.00833	0	7.56 KB
#5598	`StartStopWithChild`	netcoreapp3.1	10.8μs	58ns	307ns	0.0211	0.0105	0	7.65 KB
#5598	`StartStopWithChild`	net472	17μs	34.7ns	134ns	1.35	0.346	0.11	8.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	471μs	240ns	930ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	625μs	157ns	565ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	825μs	264ns	1.02μs	0.411	3.3 KB
#5598	`WriteAndFlushEnrichedTraces`	net6.0	468μs	315ns	1.18μs	0	2.7 KB
#5598	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	639μs	219ns	790ns	0	2.7 KB
#5598	`WriteAndFlushEnrichedTraces`	net472	828μs	188ns	729ns	0.411	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	171μs	289ns	1.12μs	0.258	18.44 KB
master	`SendRequest`	netcoreapp3.1	191μs	556ns	2.15μs	0.19	20.6 KB
master	`SendRequest`	net472	0.157ns	0.00127ns	0.00492ns	0	0 b
#5598	`SendRequest`	net6.0	172μs	304ns	1.18μs	0.254	18.44 KB
#5598	`SendRequest`	netcoreapp3.1	191μs	353ns	1.37μs	0.189	20.6 KB
#5598	`SendRequest`	net472	0.000472ns	0.000223ns	0.000865ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	558μs	370ns	1.33μs	0.563	0	0	41.57 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	656μs	1.9μs	7.37μs	0.322	0	0	41.69 KB
master	`WriteAndFlushEnrichedTraces`	net472	846μs	3.58μs	13.9μs	8.28	2.48	0.414	53.27 KB
#5598	`WriteAndFlushEnrichedTraces`	net6.0	549μs	832ns	3.11μs	0.534	0	0	41.76 KB
#5598	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	1.85μs	7.16μs	0.331	0	0	41.76 KB
#5598	`WriteAndFlushEnrichedTraces`	net472	830μs	3.65μs	14.1μs	8.12	2.44	0.406	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.12μs	0.851ns	3.3ns	0.0112	808 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.51μs	0.983ns	3.81ns	0.0105	808 B
master	`ExecuteNonQuery`	net472	1.71μs	1.16ns	4.51ns	0.122	770 B
#5598	`ExecuteNonQuery`	net6.0	1.04μs	0.536ns	2.08ns	0.0114	808 B
#5598	`ExecuteNonQuery`	netcoreapp3.1	1.51μs	0.84ns	3.25ns	0.0107	808 B
#5598	`ExecuteNonQuery`	net472	1.71μs	1.07ns	4.16ns	0.122	770 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.23μs	1.13ns	4.37ns	0.0141	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.49μs	0.765ns	2.86ns	0.0134	976 B
master	`CallElasticsearch`	net472	2.51μs	1.23ns	4.76ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.25μs	0.694ns	2.6ns	0.0132	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	1.38ns	5.35ns	0.0137	1.02 KB
master	`CallElasticsearchAsync`	net472	2.52μs	1.87ns	7.26ns	0.166	1.05 KB
#5598	`CallElasticsearch`	net6.0	1.14μs	0.519ns	1.94ns	0.0137	976 B
#5598	`CallElasticsearch`	netcoreapp3.1	1.56μs	3.6ns	13.9ns	0.0127	976 B
#5598	`CallElasticsearch`	net472	2.56μs	1.3ns	5.03ns	0.158	995 B
#5598	`CallElasticsearchAsync`	net6.0	1.37μs	0.963ns	3.47ns	0.0137	952 B
#5598	`CallElasticsearchAsync`	netcoreapp3.1	1.71μs	0.648ns	2.51ns	0.0138	1.02 KB
#5598	`CallElasticsearchAsync`	net472	2.49μs	2.48ns	9.6ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.805ns	3.01ns	0.0135	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.58μs	1.12ns	4.34ns	0.0126	952 B
master	`ExecuteAsync`	net472	1.79μs	0.911ns	3.53ns	0.145	915 B
#5598	`ExecuteAsync`	net6.0	1.36μs	0.653ns	2.44ns	0.013	952 B
#5598	`ExecuteAsync`	netcoreapp3.1	1.62μs	2.24ns	8.67ns	0.0129	952 B
#5598	`ExecuteAsync`	net472	1.72μs	0.747ns	2.89ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.19μs	1.86ns	7.21ns	0.0315	2.22 KB
master	`SendAsync`	netcoreapp3.1	4.97μs	2.84ns	11ns	0.0373	2.76 KB
master	`SendAsync`	net472	7.75μs	19.9ns	77ns	0.494	3.12 KB
#5598	`SendAsync`	net6.0	4.26μs	1.1ns	3.98ns	0.0318	2.22 KB
#5598	`SendAsync`	netcoreapp3.1	5.07μs	1.34ns	5.19ns	0.0356	2.76 KB
#5598	`SendAsync`	net472	7.67μs	3.15ns	12.2ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.48μs	0.504ns	1.89ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.28μs	1.66ns	6.44ns	0.0227	1.64 KB
master	`EnrichedLog`	net472	2.72μs	0.682ns	2.46ns	0.249	1.57 KB
#5598	`EnrichedLog`	net6.0	1.42μs	0.698ns	2.61ns	0.0234	1.64 KB
#5598	`EnrichedLog`	netcoreapp3.1	2.09μs	1.19ns	4.45ns	0.0229	1.64 KB
#5598	`EnrichedLog`	net472	2.61μs	1.66ns	6.43ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	71.5ns	277ns	0.0572	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	99.8ns	386ns	0.0589	0	4.28 KB
master	`EnrichedLog`	net472	148μs	54.7ns	212ns	0.666	0.222	4.46 KB
#5598	`EnrichedLog`	net6.0	115μs	109ns	423ns	0.0568	0	4.28 KB
#5598	`EnrichedLog`	netcoreapp3.1	119μs	146ns	545ns	0.0596	0	4.28 KB
#5598	`EnrichedLog`	net472	151μs	175ns	679ns	0.678	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.93μs	1.34ns	5.2ns	0.031	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4μs	1.79ns	6.69ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.85μs	1.51ns	5.83ns	0.319	2.02 KB
#5598	`EnrichedLog`	net6.0	3.1μs	0.968ns	3.75ns	0.031	2.2 KB
#5598	`EnrichedLog`	netcoreapp3.1	4.32μs	12.2ns	47.2ns	0.0297	2.2 KB
#5598	`EnrichedLog`	net472	4.78μs	1.91ns	7.4ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.28μs	0.412ns	1.59ns	0.016	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.68μs	0.825ns	3.2ns	0.0158	1.14 KB
master	`SendReceive`	net472	2.09μs	1.02ns	3.96ns	0.183	1.16 KB
#5598	`SendReceive`	net6.0	1.3μs	0.548ns	2.12ns	0.0161	1.14 KB
#5598	`SendReceive`	netcoreapp3.1	1.7μs	0.745ns	2.88ns	0.0153	1.14 KB
#5598	`SendReceive`	net472	2.08μs	1.99ns	7.72ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.62μs	0.933ns	3.62ns	0.0226	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.07μs	0.891ns	3.45ns	0.0223	1.65 KB
master	`EnrichedLog`	net472	4.26μs	3.67ns	14.2ns	0.323	2.04 KB
#5598	`EnrichedLog`	net6.0	2.8μs	1.19ns	4.44ns	0.0226	1.6 KB
#5598	`EnrichedLog`	netcoreapp3.1	3.82μs	10ns	38.7ns	0.0227	1.65 KB
#5598	`EnrichedLog`	net472	4.37μs	3.71ns	13.9ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5598

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.155	531.41	613.98

Faster 🎉 in #5598

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.119	763.09	682.00

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	460ns	0.388ns	1.5ns	0.00809	576 B
master	`StartFinishSpan`	netcoreapp3.1	581ns	0.3ns	1.12ns	0.00786	576 B
master	`StartFinishSpan`	net472	656ns	0.366ns	1.42ns	0.0916	578 B
master	`StartFinishScope`	net6.0	532ns	0.202ns	0.781ns	0.00989	696 B
master	`StartFinishScope`	netcoreapp3.1	763ns	0.84ns	3.25ns	0.00916	696 B
master	`StartFinishScope`	net472	949ns	0.819ns	3.07ns	0.105	658 B
#5598	`StartFinishSpan`	net6.0	453ns	0.112ns	0.433ns	0.00811	576 B
#5598	`StartFinishSpan`	netcoreapp3.1	566ns	0.423ns	1.58ns	0.0078	576 B
#5598	`StartFinishSpan`	net472	612ns	0.194ns	0.753ns	0.0917	578 B
#5598	`StartFinishScope`	net6.0	614ns	0.25ns	0.968ns	0.00988	696 B
#5598	`StartFinishScope`	netcoreapp3.1	682ns	0.25ns	0.937ns	0.00927	696 B
#5598	`StartFinishScope`	net472	930ns	0.358ns	1.39ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	654ns	0.526ns	2.04ns	0.00971	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	933ns	0.972ns	3.77ns	0.00892	696 B
master	`RunOnMethodBegin`	net472	1.09μs	0.523ns	2.03ns	0.104	658 B
#5598	`RunOnMethodBegin`	net6.0	653ns	0.322ns	1.2ns	0.00984	696 B
#5598	`RunOnMethodBegin`	netcoreapp3.1	952ns	0.322ns	1.25ns	0.00945	696 B
#5598	`RunOnMethodBegin`	net472	1.15μs	0.57ns	2.21ns	0.104	658 B

andrewlock added 3 commits May 22, 2024 14:35

Be more restrictive with the version tags

8e65879

TESTING

d5dc44d

Revert "TESTING"

0807a53

This reverts commit d5dc44d.

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label May 22, 2024

andrewlock requested a review from a team as a code owner May 22, 2024 13:56

kevingosse approved these changes May 22, 2024

View reviewed changes

andrewlock merged commit e81cd24 into master May 22, 2024
54 of 57 checks passed

andrewlock deleted the andrew/improve-script branch May 22, 2024 16:11

github-actions bot added this to the vNext-v2 milestone May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve lib-injection tagging script #5598

Improve lib-injection tagging script #5598

andrewlock commented May 22, 2024

datadog-ddstaging bot commented May 22, 2024 •

edited

Loading

andrewlock commented May 22, 2024

andrewlock commented May 22, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5598

Faster 🎉 in #5598

Raw results

Raw results

Improve lib-injection tagging script #5598

Improve lib-injection tagging script #5598

Conversation

andrewlock commented May 22, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented May 22, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented May 22, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented May 22, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5598

Faster 🎉 in #5598

Raw results

Raw results

datadog-ddstaging bot commented May 22, 2024 •

edited

Loading