[build] change agent image source #5874

robertpi · 2024-08-12T08:04:31Z

Summary of changes

Move from docker's image repository to erc.aws.

Reason for change

The quotas for unauthenticated users are more generous:

erc.aws

Rate of unauthenticated image pulls:	Each supported Region: 1 per second

https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html

docker

Anonymous users:	100 pulls per 6 hours per IP address

https://forums.docker.com/t/is-authenticated-user-also-rate-limited-by-ip-address-when-pulling/140031

Test coverage

it's part of the tests!

andrewlock · 2024-08-12T08:33:21Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (75ms)  : 65, 85
     .   : milestone, 75,
    master - mean (71ms)  : 64, 79
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (1,072ms)  : 1048, 1097
     .   : milestone, 1072,
    master - mean (1,066ms)  : 1043, 1089
     .   : milestone, 1066,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (111ms)  : 108, 115
     .   : milestone, 111,
    master - mean (108ms)  : 105, 111
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (759ms)  : 723, 796
     .   : milestone, 759,
    master - mean (756ms)  : 710, 801
     .   : milestone, 756,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (96ms)  : 91, 100
     .   : milestone, 96,
    master - mean (92ms)  : 89, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (711ms)  : 696, 726
     .   : milestone, 711,
    master - mean (698ms)  : 677, 718
     .   : milestone, 698,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (192ms)  : 189, 196
     .   : milestone, 192,
    master - mean (192ms)  : 188, 195
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (1,162ms)  : 1139, 1184
     .   : milestone, 1162,
    master - mean (1,159ms)  : 1135, 1182
     .   : milestone, 1159,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (275ms)  : 271, 280
     .   : milestone, 275,
    master - mean (277ms)  : 271, 283
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (919ms)  : 890, 948
     .   : milestone, 919,
    master - mean (917ms)  : 897, 937
     .   : milestone, 917,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5874) - mean (266ms)  : 261, 270
     .   : milestone, 266,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5874) - mean (903ms)  : 885, 921
     .   : milestone, 903,
    master - mean (902ms)  : 885, 920
     .   : milestone, 902,

datadog-ddstaging · 2024-08-12T08:34:34Z

Datadog Report

Branch report: robert/use-ecr-aws-for-agent-image
Commit report: f895e26
Test service: dd-trace-dotnet

✅ 0 Failed, 352344 Passed, 1789 Skipped, 28h 5m 52.03s Total Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

Profiler_cpu_walltime - scenarios 3.52s (+195.94ms, +6%) - Details

andrewlock · 2024-08-12T09:18:55Z

Benchmarks Report for tracer 🐌

Benchmarks for #5874 compared to master:

2 benchmarks are slower, with geometric mean 1.144
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.8μs	41ns	201ns	0.0111	0.00371	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.98μs	55.1ns	326ns	0.0145	0.00482	0	5.62 KB
master	`StartStopWithChild`	net472	16.1μs	50.8ns	197ns	1.02	0.306	0.0968	6.07 KB
#5874	`StartStopWithChild`	net6.0	7.69μs	44.2ns	336ns	0.0154	0.00768	0	5.42 KB
#5874	`StartStopWithChild`	netcoreapp3.1	9.73μs	52.4ns	301ns	0.0244	0.00977	0	5.62 KB
#5874	`StartStopWithChild`	net472	16μs	42.1ns	163ns	1.01	0.287	0.0958	6.08 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	192ns	745ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	298ns	1.15μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	826μs	1.09μs	4.08μs	0.408	3.3 KB
#5874	`WriteAndFlushEnrichedTraces`	net6.0	476μs	466ns	1.81μs	0	2.7 KB
#5874	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	631μs	322ns	1.2μs	0	2.7 KB
#5874	`WriteAndFlushEnrichedTraces`	net472	825μs	476ns	1.84μs	0.408	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	189μs	1.03μs	6.62μs	0.184	18.45 KB
master	`SendRequest`	netcoreapp3.1	220μs	1.29μs	12.2μs	0.208	20.61 KB
master	`SendRequest`	net472	0.000191ns	0.000182ns	0.000658ns	0	0 b
#5874	`SendRequest`	net6.0	191μs	1.03μs	5.54μs	0.185	18.45 KB
#5874	`SendRequest`	netcoreapp3.1	210μs	1.17μs	7.91μs	0.21	20.61 KB
#5874	`SendRequest`	net472	0ns	0ns	0ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	546μs	2.67μs	13.9μs	0.579	0	0	41.58 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	680μs	3.22μs	13.3μs	0.353	0	0	41.72 KB
master	`WriteAndFlushEnrichedTraces`	net472	866μs	4.08μs	17.3μs	8.13	2.57	0.428	53.27 KB
#5874	`WriteAndFlushEnrichedTraces`	net6.0	578μs	3.15μs	18.4μs	0.581	0	0	41.57 KB
#5874	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	692μs	3.49μs	15.2μs	0.345	0	0	41.81 KB
#5874	`WriteAndFlushEnrichedTraces`	net472	876μs	3.53μs	13.2μs	8.39	2.52	0.419	53.31 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.28μs	1ns	3.89ns	0.0141	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	0.637ns	2.3ns	0.0135	1.02 KB
master	`ExecuteNonQuery`	net472	1.99μs	1.18ns	4.56ns	0.156	987 B
#5874	`ExecuteNonQuery`	net6.0	1.27μs	1.27ns	4.91ns	0.0146	1.02 KB
#5874	`ExecuteNonQuery`	netcoreapp3.1	1.72μs	1.33ns	5.14ns	0.0137	1.02 KB
#5874	`ExecuteNonQuery`	net472	1.97μs	1.41ns	5.47ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.399ns	1.55ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.51μs	1.39ns	5.37ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.51μs	1.4ns	5.24ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.29μs	1.97ns	7.63ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.56μs	1.33ns	5.16ns	0.0135	1.02 KB
master	`CallElasticsearchAsync`	net472	2.63μs	2.77ns	10.7ns	0.166	1.05 KB
#5874	`CallElasticsearch`	net6.0	1.28μs	0.863ns	3.34ns	0.0135	976 B
#5874	`CallElasticsearch`	netcoreapp3.1	1.47μs	0.699ns	2.71ns	0.0126	976 B
#5874	`CallElasticsearch`	net472	2.45μs	2.34ns	9.05ns	0.157	995 B
#5874	`CallElasticsearchAsync`	net6.0	1.34μs	0.853ns	3.3ns	0.0134	952 B
#5874	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	1.37ns	5.13ns	0.0138	1.02 KB
#5874	`CallElasticsearchAsync`	net472	2.61μs	1.65ns	6.4ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.22μs	1.12ns	4.33ns	0.0135	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.64μs	0.632ns	2.28ns	0.0123	952 B
master	`ExecuteAsync`	net472	1.67μs	0.961ns	3.72ns	0.145	915 B
#5874	`ExecuteAsync`	net6.0	1.1μs	0.435ns	1.63ns	0.0132	952 B
#5874	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.314ns	1.13ns	0.0129	952 B
#5874	`ExecuteAsync`	net472	1.8μs	0.893ns	3.46ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.21μs	1.66ns	6.44ns	0.0296	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.12μs	1.64ns	5.9ns	0.0363	2.76 KB
master	`SendAsync`	net472	7.7μs	2.98ns	11.6ns	0.499	3.15 KB
#5874	`SendAsync`	net6.0	4.26μs	0.931ns	3.48ns	0.0298	2.22 KB
#5874	`SendAsync`	netcoreapp3.1	5.16μs	5.72ns	21.4ns	0.0357	2.76 KB
#5874	`SendAsync`	net472	7.78μs	1.47ns	5.71ns	0.5	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.47μs	0.858ns	3.21ns	0.0233	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.1μs	1.74ns	6.74ns	0.0221	1.64 KB
master	`EnrichedLog`	net472	2.69μs	0.911ns	3.41ns	0.25	1.57 KB
#5874	`EnrichedLog`	net6.0	1.54μs	0.883ns	3.42ns	0.0232	1.64 KB
#5874	`EnrichedLog`	netcoreapp3.1	2.09μs	1.18ns	4.41ns	0.022	1.64 KB
#5874	`EnrichedLog`	net472	2.68μs	1.16ns	4.48ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	91.3ns	353ns	0.0583	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	106ns	411ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	77.9ns	291ns	0.671	0.224	4.46 KB
#5874	`EnrichedLog`	net6.0	117μs	198ns	768ns	0	0	4.28 KB
#5874	`EnrichedLog`	netcoreapp3.1	120μs	159ns	615ns	0	0	4.28 KB
#5874	`EnrichedLog`	net472	147μs	119ns	429ns	0.67	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.01μs	0.78ns	2.92ns	0.0304	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.08μs	2.3ns	8.9ns	0.0287	2.2 KB
master	`EnrichedLog`	net472	4.91μs	1.15ns	4.3ns	0.318	2.02 KB
#5874	`EnrichedLog`	net6.0	3.04μs	0.833ns	3.11ns	0.0304	2.2 KB
#5874	`EnrichedLog`	netcoreapp3.1	4.05μs	3.76ns	14.6ns	0.0284	2.2 KB
#5874	`EnrichedLog`	net472	4.84μs	1.05ns	4.07ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.27μs	0.756ns	2.93ns	0.0158	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.504ns	1.82ns	0.015	0	1.14 KB
master	`SendReceive`	net472	2.14μs	0.687ns	2.66ns	0.183	0.00107	1.16 KB
#5874	`SendReceive`	net6.0	1.34μs	1.12ns	4.17ns	0.016	0	1.14 KB
#5874	`SendReceive`	netcoreapp3.1	1.79μs	0.7ns	2.71ns	0.0153	0	1.14 KB
#5874	`SendReceive`	net472	2.15μs	0.46ns	1.72ns	0.183	0.00108	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.605ns	2.1ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.97μs	6.61ns	25.6ns	0.0221	1.65 KB
master	`EnrichedLog`	net472	4.36μs	3.12ns	11.7ns	0.324	2.04 KB
#5874	`EnrichedLog`	net6.0	2.74μs	0.881ns	3.41ns	0.0216	1.6 KB
#5874	`EnrichedLog`	netcoreapp3.1	3.82μs	1.9ns	7.1ns	0.0209	1.65 KB
#5874	`EnrichedLog`	net472	4.43μs	1.9ns	7.36ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5874

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.151	663.11	763.04
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.137	474.59	539.78

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	405ns	0.257ns	0.961ns	0.00814	576 B
master	`StartFinishSpan`	netcoreapp3.1	594ns	0.863ns	3.34ns	0.00767	576 B
master	`StartFinishSpan`	net472	642ns	0.609ns	2.28ns	0.0917	578 B
master	`StartFinishScope`	net6.0	475ns	0.143ns	0.553ns	0.00969	696 B
master	`StartFinishScope`	netcoreapp3.1	663ns	0.645ns	2.5ns	0.00964	696 B
master	`StartFinishScope`	net472	894ns	0.393ns	1.52ns	0.104	658 B
#5874	`StartFinishSpan`	net6.0	398ns	0.104ns	0.39ns	0.00802	576 B
#5874	`StartFinishSpan`	netcoreapp3.1	634ns	0.227ns	0.88ns	0.00761	576 B
#5874	`StartFinishSpan`	net472	645ns	0.483ns	1.87ns	0.0916	578 B
#5874	`StartFinishScope`	net6.0	539ns	0.795ns	3.08ns	0.00969	696 B
#5874	`StartFinishScope`	netcoreapp3.1	763ns	0.551ns	2.06ns	0.00919	696 B
#5874	`StartFinishScope`	net472	894ns	0.414ns	1.6ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	617ns	0.161ns	0.623ns	0.00981	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	889ns	0.454ns	1.76ns	0.00931	696 B
master	`RunOnMethodBegin`	net472	1.16μs	0.259ns	0.935ns	0.104	658 B
#5874	`RunOnMethodBegin`	net6.0	625ns	2.81ns	10.9ns	0.00962	696 B
#5874	`RunOnMethodBegin`	netcoreapp3.1	906ns	0.853ns	3.3ns	0.00908	696 B
#5874	`RunOnMethodBegin`	net472	1.12μs	0.312ns	1.21ns	0.104	658 B

andrewlock

Thanks!

robertpi · 2024-08-12T11:53:08Z

Merging with one approval as it's a small change that will improve CI stability.

change agent image source

f895e26

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Aug 12, 2024

robertpi marked this pull request as ready for review August 12, 2024 11:11

robertpi requested a review from a team as a code owner August 12, 2024 11:11

robertpi changed the title ~~change agent image source~~ [build] change agent image source Aug 12, 2024

andrewlock approved these changes Aug 12, 2024

View reviewed changes

robertpi merged commit 6149a19 into master Aug 12, 2024
63 of 68 checks passed

robertpi deleted the robert/use-ecr-aws-for-agent-image branch August 12, 2024 11:53

github-actions bot added this to the vNext-v3 milestone Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] change agent image source #5874

[build] change agent image source #5874

robertpi commented Aug 12, 2024 •

edited

Loading

andrewlock commented Aug 12, 2024

datadog-ddstaging bot commented Aug 12, 2024 •

edited

Loading

andrewlock commented Aug 12, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5874

Raw results

Raw results

andrewlock left a comment

robertpi commented Aug 12, 2024

[build] change agent image source #5874

[build] change agent image source #5874

Conversation

robertpi commented Aug 12, 2024 • edited Loading

Summary of changes

Reason for change

erc.aws

docker

Test coverage

andrewlock commented Aug 12, 2024

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Aug 12, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Aug 12, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5874

Raw results

Raw results

andrewlock left a comment

Choose a reason for hiding this comment

robertpi commented Aug 12, 2024

robertpi commented Aug 12, 2024 •

edited

Loading

datadog-ddstaging bot commented Aug 12, 2024 •

edited

Loading