Fix dependabot and improve `GeneratePackageVersions` #5579

andrewlock · 2024-05-17T12:34:12Z

Summary of changes

Fixes our broken Dependabot integration

Reason for change

Our dependabot integration broke at some point, so we haven't been getting notifications for new packages. The reason is that GitHub tries to run a dotnet restore on the "honeypot" dependabot project file, and unfortunately it was getting a bunch of errors (and warnings) and bailing out:

error NU1202: Package Aerospike.Client 7.2.0 is not compatible with netcoreapp3.1 
error NU1202: Package Yarp.ReverseProxy 2.1.0 is not compatible with netcoreapp3.1
error NU1202: Package Grpc.AspNetCore 2.62.0 is not compatible with netcoreapp3.1

So this PR fixes the issue (by bumping the TFM), solves some other warnings (about duplicate entries) and sets up alerts and fallback approaches

Implementation details

Bump the TFM to .NET 8 to fix the restore errors. This should cover us for a little while at least
"Fix" the file generator to group by packagename so we don't duplicate packages when they're referenced in multiple integrations.
Added a verification stage to the end of the Nuke target that makes sure the generated file can be restored, to hopefully avoid this issue happening again
Added a "fallback" schedule so we run the tool every week, regardless of dependabot updates
Send a slack notification if the run fails

Test coverage

Tested locally a bunch, but we need to merge it before I can check if dependabot is fixed

Other details

Supersedes #5567

…) the project file

Dependabot does an explicit restore of the project, and if it fails, we don't get dependabot updates Added an explicit notification so we find out when this happens

…re nothing's broken)

andrewlock · 2024-05-17T13:00:18Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (74ms)  : 64, 84
     .   : milestone, 74,
    master - mean (71ms)  : 65, 77
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (1,013ms)  : 989, 1036
     .   : milestone, 1013,
    master - mean (1,003ms)  : 984, 1023
     .   : milestone, 1003,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (109ms)  : 107, 112
     .   : milestone, 109,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (731ms)  : 709, 753
     .   : milestone, 731,
    master - mean (725ms)  : 700, 751
     .   : milestone, 725,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (94ms)  : 90, 98
     .   : milestone, 94,
    master - mean (93ms)  : 90, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (683ms)  : 654, 712
     .   : milestone, 683,
    master - mean (679ms)  : 651, 707
     .   : milestone, 679,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (193ms)  : 188, 199
     .   : milestone, 193,
    master - mean (191ms)  : 188, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (1,111ms)  : 1089, 1133
     .   : milestone, 1111,
    master - mean (1,107ms)  : 1088, 1127
     .   : milestone, 1107,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (275ms)  : 271, 280
     .   : milestone, 275,
    master - mean (274ms)  : 269, 279
     .   : milestone, 274,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (914ms)  : 894, 934
     .   : milestone, 914,
    master - mean (911ms)  : 892, 930
     .   : milestone, 911,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5579) - mean (266ms)  : 261, 271
     .   : milestone, 266,
    master - mean (266ms)  : 261, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (5579) - mean (899ms)  : 876, 922
     .   : milestone, 899,
    master - mean (892ms)  : 871, 913
     .   : milestone, 892,

datadog-ddstaging · 2024-05-17T13:04:24Z

Datadog Report

Branch report: andrew/ci/try-fix-dependabot
Commit report: 9fb5905
Test service: dd-trace-dotnet

✅ 0 Failed, 337224 Passed, 1597 Skipped, 13h 49m 46.64s Total Time

andrewlock · 2024-05-17T13:51:52Z

Benchmarks Report for tracer 🐌

Benchmarks for #5579 compared to master:

2 benchmarks are faster, with geometric mean 1.157
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.85μs	47.3ns	284ns	0.023	0.00921	0	7.56 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	55.5ns	293ns	0.0323	0.0161	0	7.65 KB
master	`StartStopWithChild`	net472	17.2μs	35.7ns	134ns	1.36	0.38	0.11	8.06 KB
#5579	`StartStopWithChild`	net6.0	8.84μs	47.7ns	274ns	0.0264	0.0132	0	7.56 KB
#5579	`StartStopWithChild`	netcoreapp3.1	10.6μs	56ns	274ns	0.0266	0.0107	0	7.66 KB
#5579	`StartStopWithChild`	net472	17.1μs	51.3ns	199ns	1.35	0.364	0.0931	8.05 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	460μs	362ns	1.4μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	209ns	781ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	815μs	355ns	1.37μs	0.406	3.3 KB
#5579	`WriteAndFlushEnrichedTraces`	net6.0	466μs	149ns	578ns	0	2.7 KB
#5579	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	156ns	585ns	0	2.7 KB
#5579	`WriteAndFlushEnrichedTraces`	net472	823μs	167ns	603ns	0.411	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	170μs	159ns	595ns	0.253	18.44 KB
master	`SendRequest`	netcoreapp3.1	190μs	236ns	912ns	0.192	20.6 KB
master	`SendRequest`	net472	0.000251ns	0.00018ns	0.000696ns	0	0 b
#5579	`SendRequest`	net6.0	169μs	212ns	821ns	0.256	18.44 KB
#5579	`SendRequest`	netcoreapp3.1	191μs	245ns	950ns	0.19	20.6 KB
#5579	`SendRequest`	net472	0.0006ns	0.000315ns	0.00118ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	553μs	507ns	1.96μs	0.553	0	0	41.63 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	1.23μs	4.28μs	0.322	0	0	41.78 KB
master	`WriteAndFlushEnrichedTraces`	net472	842μs	3.1μs	12μs	8.47	2.42	0.403	53.29 KB
#5579	`WriteAndFlushEnrichedTraces`	net6.0	567μs	1.01μs	3.9μs	0.561	0	0	41.73 KB
#5579	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	650μs	1.01μs	3.91μs	0.322	0	0	41.69 KB
#5579	`WriteAndFlushEnrichedTraces`	net472	866μs	3.25μs	12.6μs	8.19	2.59	0.431	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5579

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0	1.157	1,190.05	1,028.95

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.19μs	0.41ns	1.54ns	0.0112	808 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.42μs	0.53ns	2.05ns	0.0107	808 B
master	`ExecuteNonQuery`	net472	1.77μs	0.94ns	3.52ns	0.122	770 B
#5579	`ExecuteNonQuery`	net6.0	1.03μs	0.478ns	1.85ns	0.0113	808 B
#5579	`ExecuteNonQuery`	netcoreapp3.1	1.46μs	0.692ns	2.68ns	0.011	808 B
#5579	`ExecuteNonQuery`	net472	1.72μs	0.887ns	3.44ns	0.122	770 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.26μs	0.636ns	2.38ns	0.0134	0	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.57μs	1.47ns	5.51ns	0.0133	0	976 B
master	`CallElasticsearch`	net472	2.55μs	1.05ns	3.94ns	0.158	0	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.911ns	3.53ns	0.0135	0	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	0.766ns	2.87ns	0.0139	0	1.02 KB
master	`CallElasticsearchAsync`	net472	2.54μs	2.09ns	8.09ns	0.167	0.00126	1.05 KB
#5579	`CallElasticsearch`	net6.0	1.23μs	0.748ns	2.8ns	0.0136	0	976 B
#5579	`CallElasticsearch`	netcoreapp3.1	1.47μs	0.632ns	2.45ns	0.0132	0	976 B
#5579	`CallElasticsearch`	net472	2.54μs	0.818ns	3.06ns	0.158	0	995 B
#5579	`CallElasticsearchAsync`	net6.0	1.32μs	1.31ns	5.07ns	0.0133	0	952 B
#5579	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	0.888ns	3.44ns	0.0137	0	1.02 KB
#5579	`CallElasticsearchAsync`	net472	2.66μs	1.52ns	5.69ns	0.167	0	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	1.03ns	3.84ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.6μs	1.12ns	4.2ns	0.0127	952 B
master	`ExecuteAsync`	net472	1.83μs	0.386ns	1.44ns	0.145	915 B
#5579	`ExecuteAsync`	net6.0	1.19μs	0.433ns	1.62ns	0.0131	952 B
#5579	`ExecuteAsync`	netcoreapp3.1	1.6μs	1.3ns	5.05ns	0.0126	952 B
#5579	`ExecuteAsync`	net472	1.75μs	0.836ns	3.24ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.15μs	1.53ns	5.73ns	0.0313	2.22 KB
master	`SendAsync`	netcoreapp3.1	5μs	1.08ns	3.89ns	0.0374	2.76 KB
master	`SendAsync`	net472	7.37μs	3.22ns	12.5ns	0.495	3.12 KB
#5579	`SendAsync`	net6.0	4.18μs	0.997ns	3.86ns	0.0307	2.22 KB
#5579	`SendAsync`	netcoreapp3.1	5.04μs	2.67ns	9.98ns	0.0376	2.76 KB
#5579	`SendAsync`	net472	7.57μs	3.39ns	13.1ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.42μs	0.689ns	2.49ns	0.0228	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.1μs	0.543ns	2.03ns	0.022	1.64 KB
master	`EnrichedLog`	net472	2.47μs	1.22ns	4.72ns	0.25	1.57 KB
#5579	`EnrichedLog`	net6.0	1.41μs	0.824ns	3.08ns	0.0229	1.64 KB
#5579	`EnrichedLog`	netcoreapp3.1	2.2μs	2.2ns	8.23ns	0.022	1.64 KB
#5579	`EnrichedLog`	net472	2.64μs	1.51ns	5.45ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	178ns	690ns	0.0569	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	84.3ns	315ns	0.0586	0	4.28 KB
master	`EnrichedLog`	net472	147μs	108ns	403ns	0.662	0.221	4.46 KB
#5579	`EnrichedLog`	net6.0	115μs	89.4ns	335ns	0	0	4.28 KB
#5579	`EnrichedLog`	netcoreapp3.1	119μs	112ns	433ns	0.0598	0	4.28 KB
#5579	`EnrichedLog`	net472	149μs	184ns	711ns	0.669	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.94μs	1.16ns	4.48ns	0.0311	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.29μs	1.61ns	6.25ns	0.0279	2.2 KB
master	`EnrichedLog`	net472	4.87μs	0.804ns	3.01ns	0.32	2.02 KB
#5579	`EnrichedLog`	net6.0	2.85μs	1.26ns	4.89ns	0.0301	2.2 KB
#5579	`EnrichedLog`	netcoreapp3.1	4.06μs	1.4ns	5.25ns	0.0284	2.2 KB
#5579	`EnrichedLog`	net472	4.82μs	1.13ns	4.08ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.34μs	1.68ns	6.51ns	0.016	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	0.449ns	1.74ns	0.0154	0	1.14 KB
master	`SendReceive`	net472	2.15μs	1.13ns	4.22ns	0.183	0	1.16 KB
#5579	`SendReceive`	net6.0	1.33μs	1.33ns	5.16ns	0.0159	0	1.14 KB
#5579	`SendReceive`	netcoreapp3.1	1.68μs	1.33ns	5.15ns	0.0153	0	1.14 KB
#5579	`SendReceive`	net472	2.08μs	1.47ns	5.49ns	0.183	0.00105	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.7μs	1.13ns	4.37ns	0.0229	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.89μs	1.43ns	5.55ns	0.0214	1.65 KB
master	`EnrichedLog`	net472	4.48μs	2.54ns	9.5ns	0.324	2.04 KB
#5579	`EnrichedLog`	net6.0	2.72μs	0.59ns	2.13ns	0.0218	1.6 KB
#5579	`EnrichedLog`	netcoreapp3.1	3.81μs	1.4ns	5.43ns	0.0227	1.65 KB
#5579	`EnrichedLog`	net472	4.45μs	2.75ns	10.3ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5579

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.157	759.74	656.77

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	404ns	1.47ns	5.71ns	0.00797	576 B
master	`StartFinishSpan`	netcoreapp3.1	555ns	0.332ns	1.24ns	0.00778	576 B
master	`StartFinishSpan`	net472	669ns	0.478ns	1.85ns	0.0916	578 B
master	`StartFinishScope`	net6.0	519ns	0.178ns	0.689ns	0.00962	696 B
master	`StartFinishScope`	netcoreapp3.1	760ns	0.174ns	0.649ns	0.00941	696 B
master	`StartFinishScope`	net472	866ns	0.373ns	1.45ns	0.105	658 B
#5579	`StartFinishSpan`	net6.0	400ns	0.191ns	0.741ns	0.00805	576 B
#5579	`StartFinishSpan`	netcoreapp3.1	551ns	0.268ns	1ns	0.00782	576 B
#5579	`StartFinishSpan`	net472	644ns	0.396ns	1.53ns	0.0915	578 B
#5579	`StartFinishScope`	net6.0	554ns	0.132ns	0.51ns	0.00971	696 B
#5579	`StartFinishScope`	netcoreapp3.1	657ns	0.174ns	0.603ns	0.00929	696 B
#5579	`StartFinishScope`	net472	862ns	0.151ns	0.566ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	584ns	0.169ns	0.653ns	0.00966	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	878ns	0.337ns	1.26ns	0.00934	696 B
master	`RunOnMethodBegin`	net472	1.18μs	0.55ns	2.13ns	0.104	658 B
#5579	`RunOnMethodBegin`	net6.0	650ns	0.497ns	1.93ns	0.00975	696 B
#5579	`RunOnMethodBegin`	netcoreapp3.1	932ns	1.73ns	6.69ns	0.0093	696 B
#5579	`RunOnMethodBegin`	net472	1.2μs	0.327ns	1.27ns	0.104	658 B

andrewlock added 6 commits May 17, 2024 13:31

Bump dependabot template TFM (because we have to be _able_ to restore…

0a6d332

…) the project file

Fix dependabot file generator to avoid duplication and fix sorting

91d946f

Add verification step to GeneratePackageVersions

78c27e2

Dependabot does an explicit restore of the project, and if it fails, we don't get dependabot updates Added an explicit notification so we find out when this happens

Run the package version bump on a schedule (as a catchall, to make su…

9d2e382

…re nothing's broken)

Send slack notification if dependabot fails to build

c84940c

Update generated dependabot file

9fb5905

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label May 17, 2024

andrewlock requested review from a team as code owners May 17, 2024 12:34

kevingosse approved these changes May 17, 2024

View reviewed changes

andrewlock merged commit 9199672 into master May 17, 2024
55 of 57 checks passed

andrewlock deleted the andrew/ci/try-fix-dependabot branch May 17, 2024 15:11

github-actions bot added this to the vNext-v2 milestone May 17, 2024

andrewlock mentioned this pull request May 17, 2024

Run the package version bump on a schedule #5567

Closed

bouwkast mentioned this pull request Aug 15, 2024

Improve logging output of automated package bumps #5909

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix dependabot and improve `GeneratePackageVersions` #5579

Fix dependabot and improve `GeneratePackageVersions` #5579

andrewlock commented May 17, 2024

andrewlock commented May 17, 2024

datadog-ddstaging bot commented May 17, 2024 •

edited

Loading

andrewlock commented May 17, 2024

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5579

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5579

Raw results

Raw results

Fix dependabot and improve GeneratePackageVersions #5579

Fix dependabot and improve GeneratePackageVersions #5579

Conversation

andrewlock commented May 17, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented May 17, 2024

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented May 17, 2024 • edited Loading

Datadog Report

andrewlock commented May 17, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5579

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5579

Raw results

Raw results

Fix dependabot and improve `GeneratePackageVersions` #5579

Fix dependabot and improve `GeneratePackageVersions` #5579

datadog-ddstaging bot commented May 17, 2024 •

edited

Loading