[ci] Add oci package build #5340

paullegranddc · 2024-03-22T14:30:54Z

Summary of changes

Add a ci step that packages the dotnet library to the new OCI package format.

Reason for change

This is required to be able to install dotnet packages with the auto-updater

Implementation details

Test coverage

I've branched out and run the pacakges-oci + release jobs manually, and checked with @BaptisteFoy that it created the expected package in OCI repo + rc catalog

Other details

datadog-ddstaging · 2024-03-22T15:00:15Z

Datadog Report

Branch report: paullgdc/auto_updater/add_oci_package_build
Commit report: fc52bdd
Test service: dd-trace-dotnet

✅ 0 Failed, 326753 Passed, 1583 Skipped, 43m 36.03s Wall Time

andrewlock · 2024-03-22T15:29:15Z

.gitlab/build-oci.sh

+echo -n $DOTNET_PACKAGE_VERSION > $TMP_DIR/opt/datadog/version/auto_inject-dotnet.version
+
+# Build packages
+datadog-package create \


I'd love to know exactly why this is necessary, and whether we can update our existing packages to make the re-packing unneccessary... lib-injection is already doing repackaging and I wonder if we could settle on a standard format 🤔

The main reason we need a new package format is that we are working on the auto update of APM libraries for host setups. This is going to be used in conjunction with lib-injection. The goal is for the OCI package to be the standard format in the future.

This auto update mechanism works by having the following fs structure
/opt/datadog-packages//
/stable <- symlink to the current version
and a process called the updater, connected to Remote Config that can download and unpack new versions.
To make this work nicely, we need a platform independent package format, that can be unpacked in a single relative path, which existing debs and rpm don't offer.

andrewlock · 2024-03-22T15:30:48Z

.gitlab-ci.yml

+        - arm64
+        - amd64


We need to support musl too - not sure how you need to handle that here though 😄

Do we? i don't think musl is supported by the auto injecter anyway

I don't think the agent (resp. updater) can run on a host with musl. How is musl used today for debs & rpms?

We can package it anyway though, there are ways to add "OS Features" to the platform selection. Happy to implement support for that in the packaging tool if needed 😄

Do we? i don't think musl is supported by the auto injecter anyway

Oh, really 🤔 I guess for host injection that may be true (I don't know), we do support it for k8s lib-injection.

How is musl used today for debs & rpms?

Currently we only produce musl tar.gz files 🙂

We can package it anyway though, there are ways to add "OS Features" to the platform selection

Interesting, I suspect we should add that to our existing deb/rpms then!

BaptisteFoy · 2024-03-22T17:42:26Z

.gitlab/build-oci.sh

+curl --location --fail \
+  --output $TMP_DIR/datadog-dotnet-apm.old \
+  "https://github.com/DataDog/dd-trace-dotnet/releases/download/v$DOTNET_PACKAGE_VERSION/datadog-dotnet-apm_${DOTNET_PACKAGE_VERSION}_$ARCH.deb"
+
+fpm --input-type deb \
+  --output-type dir \
+  --name datadog-dotnet-apm \
+  --package $TMP_DIR \
+  $TMP_DIR/datadog-dotnet-apm.old


❓ Question: ‏Can't we re-use the results from the build stage directly?

No, from what I've been told debs are not built in gitlab but in the azure devops CI

No, from what I've been told debs are not built in gitlab but in the azure devops CI

Out of interest, why does that mean you can't use them directly? (You're correct FWIW)

I'm not sure, but looking at .gitlab/build-deb-rpm.sh, which builds deb and rpm packages for the injector, I think the azure CI debs are not installed in the expected location. It puts the dotnet package in /opt/datadog whereas the injector expects packages to be in /opt/datadog/apm/libraries.

Plus I believe we'll do some signing in the future which is required to run on the the internal gitlab CI.

andrewlock · 2024-03-22T17:50:59Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (75ms)  : 65, 85
     .   : milestone, 75,
    master - mean (73ms)  : 67, 79
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (1,009ms)  : 988, 1030
     .   : milestone, 1009,
    master - mean (1,005ms)  : 987, 1024
     .   : milestone, 1005,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (112ms)  : 109, 115
     .   : milestone, 112,
    master - mean (110ms)  : 107, 113
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (733ms)  : 706, 759
     .   : milestone, 733,
    master - mean (723ms)  : 702, 744
     .   : milestone, 723,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (95ms)  : 93, 98
     .   : milestone, 95,
    master - mean (93ms)  : 90, 97
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (684ms)  : 661, 707
     .   : milestone, 684,
    master - mean (677ms)  : 651, 704
     .   : milestone, 677,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (188ms)  : 183, 192
     .   : milestone, 188,
    master - mean (189ms)  : 186, 192
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (1,074ms)  : 1052, 1097
     .   : milestone, 1074,
    master - mean (1,087ms)  : 1058, 1115
     .   : milestone, 1087,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (273ms)  : 268, 277
     .   : milestone, 273,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (868ms)  : 842, 894
     .   : milestone, 868,
    master - mean (877ms)  : 855, 900
     .   : milestone, 877,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5340) - mean (259ms)  : 255, 264
     .   : milestone, 259,
    master - mean (262ms)  : 258, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (5340) - mean (853ms)  : 830, 876
     .   : milestone, 853,
    master - mean (859ms)  : 833, 884
     .   : milestone, 859,

andrewlock · 2024-03-22T18:44:29Z

Benchmarks Report for tracer 🐌

Benchmarks for #5340 compared to master:

2 benchmarks are slower, with geometric mean 1.200
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.86μs	39.4ns	201ns	0.017	0.00849	0	7.51 KB
master	`StartStopWithChild`	netcoreapp3.1	10.8μs	58.4ns	320ns	0.0216	0.0108	0	7.6 KB
master	`StartStopWithChild`	net472	17μs	32.9ns	123ns	1.36	0.337	0.126	8.05 KB
#5340	`StartStopWithChild`	net6.0	8.69μs	46ns	226ns	0.0295	0.0126	0	7.51 KB
#5340	`StartStopWithChild`	netcoreapp3.1	10.6μs	56.4ns	276ns	0.0263	0.0105	0	7.61 KB
#5340	`StartStopWithChild`	net472	17.1μs	77.8ns	301ns	1.35	0.337	0.0928	8.04 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	467μs	119ns	462ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	620μs	280ns	1.01μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	810μs	627ns	2.43μs	0.406	3.3 KB
#5340	`WriteAndFlushEnrichedTraces`	net6.0	461μs	117ns	440ns	0	2.7 KB
#5340	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	600μs	219ns	819ns	0	2.7 KB
#5340	`WriteAndFlushEnrichedTraces`	net472	806μs	249ns	897ns	0.401	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	171μs	187ns	723ns	0.17	18.33 KB
master	`SendRequest`	netcoreapp3.1	190μs	206ns	797ns	0.19	20.49 KB
master	`SendRequest`	net472	0.000717ns	0.000164ns	0.000612ns	0	0 b
#5340	`SendRequest`	net6.0	173μs	219ns	849ns	0.256	18.33 KB
#5340	`SendRequest`	netcoreapp3.1	193μs	412ns	1.59μs	0.194	20.49 KB
#5340	`SendRequest`	net472	0.000937ns	0.000322ns	0.00116ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5340

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.62 KB	41.88 KB	267 B	0.64%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	558μs	584ns	2.19μs	0.536	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	642μs	1.75μs	6.78μs	0.316	0	0	41.62 KB
master	`WriteAndFlushEnrichedTraces`	net472	851μs	3.7μs	14.3μs	8.28	2.48	0.414	53.22 KB
#5340	`WriteAndFlushEnrichedTraces`	net6.0	546μs	1.49μs	5.75μs	0.541	0	0	41.81 KB
#5340	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	1.15μs	4.45μs	0.327	0	0	41.88 KB
#5340	`WriteAndFlushEnrichedTraces`	net472	891μs	4.15μs	17.6μs	8.25	2.6	0.434	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.28μs	0.735ns	2.85ns	0.0109	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.52μs	0.921ns	3.57ns	0.0107	784 B
master	`ExecuteNonQuery`	net472	1.82μs	0.946ns	3.54ns	0.118	746 B
#5340	`ExecuteNonQuery`	net6.0	1.17μs	0.334ns	1.29ns	0.0111	784 B
#5340	`ExecuteNonQuery`	netcoreapp3.1	1.51μs	0.701ns	2.72ns	0.0105	784 B
#5340	`ExecuteNonQuery`	net472	1.84μs	0.857ns	2.97ns	0.118	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.29μs	0.66ns	2.47ns	0.0142	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.6μs	0.853ns	3.19ns	0.0137	1.01 KB
master	`CallElasticsearch`	net472	2.51μs	1.33ns	5.13ns	0.161	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.31μs	0.468ns	1.81ns	0.0137	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	1.23ns	4.76ns	0.0144	1.06 KB
master	`CallElasticsearchAsync`	net472	2.55μs	1.87ns	7.25ns	0.171	1.08 KB
#5340	`CallElasticsearch`	net6.0	1.29μs	0.6ns	2.32ns	0.0142	1.01 KB
#5340	`CallElasticsearch`	netcoreapp3.1	1.64μs	0.825ns	3.09ns	0.0141	1.01 KB
#5340	`CallElasticsearch`	net472	2.65μs	1.06ns	3.96ns	0.162	1.02 KB
#5340	`CallElasticsearchAsync`	net6.0	1.41μs	0.39ns	1.35ns	0.0141	984 B
#5340	`CallElasticsearchAsync`	netcoreapp3.1	1.79μs	1.6ns	6ns	0.0142	1.06 KB
#5340	`CallElasticsearchAsync`	net472	2.53μs	1.64ns	6.15ns	0.17	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.43μs	0.82ns	3.18ns	0.0129	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.993ns	3.72ns	0.0119	928 B
master	`ExecuteAsync`	net472	1.91μs	2.14ns	8.3ns	0.141	891 B
#5340	`ExecuteAsync`	net6.0	1.37μs	0.771ns	2.89ns	0.0131	928 B
#5340	`ExecuteAsync`	netcoreapp3.1	1.61μs	5.7ns	22.1ns	0.0126	928 B
#5340	`ExecuteAsync`	net472	1.88μs	0.526ns	1.82ns	0.141	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.13μs	2.53ns	9.81ns	0.0288	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.15μs	2.71ns	9.78ns	0.0361	2.7 KB
master	`SendAsync`	net472	7.79μs	6.85ns	25.6ns	0.484	3.05 KB
#5340	`SendAsync`	net6.0	4.23μs	2.43ns	8.78ns	0.0293	2.16 KB
#5340	`SendAsync`	netcoreapp3.1	5.15μs	6.21ns	24.1ns	0.0358	2.7 KB
#5340	`SendAsync`	net472	7.66μs	4.34ns	16.8ns	0.481	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.62μs	1.04ns	4.01ns	0.0228	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.06μs	1.32ns	4.96ns	0.0218	1.65 KB
master	`EnrichedLog`	net472	2.68μs	1.54ns	5.54ns	0.25	1.57 KB
#5340	`EnrichedLog`	net6.0	1.55μs	1.46ns	5.64ns	0.0232	1.65 KB
#5340	`EnrichedLog`	netcoreapp3.1	2.22μs	1.58ns	6.1ns	0.0223	1.65 KB
#5340	`EnrichedLog`	net472	2.59μs	4.1ns	15.9ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	115μs	495ns	1.92μs	0.0583	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	300ns	1.16μs	0	0	4.23 KB
master	`EnrichedLog`	net472	147μs	120ns	449ns	0.664	0.221	4.41 KB
#5340	`EnrichedLog`	net6.0	114μs	221ns	854ns	0.0571	0	4.23 KB
#5340	`EnrichedLog`	netcoreapp3.1	119μs	154ns	576ns	0	0	4.23 KB
#5340	`EnrichedLog`	net472	148μs	183ns	709ns	0.661	0.22	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.06μs	2.83ns	10.6ns	0.0305	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.27μs	3.1ns	12ns	0.0302	2.21 KB
master	`EnrichedLog`	net472	5.06μs	3.77ns	14.1ns	0.319	2.02 KB
#5340	`EnrichedLog`	net6.0	2.99μs	1.7ns	6.35ns	0.0306	2.21 KB
#5340	`EnrichedLog`	netcoreapp3.1	4.3μs	3.33ns	12.9ns	0.0302	2.21 KB
#5340	`EnrichedLog`	net472	4.98μs	4.26ns	15.9ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.42μs	0.592ns	2.05ns	0.0163	0	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.87μs	1.52ns	5.49ns	0.0158	0	1.18 KB
master	`SendReceive`	net472	2.23μs	3.14ns	12.2ns	0.187	0.00112	1.18 KB
#5340	`SendReceive`	net6.0	1.39μs	4.26ns	16.5ns	0.0162	0	1.18 KB
#5340	`SendReceive`	netcoreapp3.1	1.86μs	1.36ns	5.08ns	0.016	0	1.18 KB
#5340	`SendReceive`	net472	2.12μs	2.66ns	10.3ns	0.187	0	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.67μs	0.695ns	2.6ns	0.0214	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	3.93μs	2.6ns	9.72ns	0.0215	1.6 KB
master	`EnrichedLog`	net472	4.49μs	3.26ns	12.2ns	0.315	1.99 KB
#5340	`EnrichedLog`	net6.0	2.74μs	0.622ns	2.41ns	0.022	1.55 KB
#5340	`EnrichedLog`	netcoreapp3.1	3.86μs	1.77ns	6.63ns	0.0212	1.6 KB
#5340	`EnrichedLog`	net472	4.39μs	2.64ns	9.89ns	0.315	1.99 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5340

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.146	897.67	1,028.91

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	514ns	0.248ns	0.961ns	0.00764	552 B
master	`StartFinishSpan`	netcoreapp3.1	783ns	3.37ns	13.1ns	0.00727	552 B
master	`StartFinishSpan`	net472	744ns	0.644ns	2.41ns	0.0879	554 B
master	`StartFinishScope`	net6.0	602ns	0.313ns	1.17ns	0.00934	672 B
master	`StartFinishScope`	netcoreapp3.1	922ns	0.187ns	0.725ns	0.00895	672 B
master	`StartFinishScope`	net472	899ns	1.21ns	4.69ns	0.101	634 B
#5340	`StartFinishSpan`	net6.0	496ns	0.2ns	0.747ns	0.00778	552 B
#5340	`StartFinishSpan`	netcoreapp3.1	789ns	0.598ns	2.24ns	0.00747	552 B
#5340	`StartFinishSpan`	net472	743ns	0.664ns	2.57ns	0.0878	554 B
#5340	`StartFinishScope`	net6.0	583ns	0.244ns	0.914ns	0.0094	672 B
#5340	`StartFinishScope`	netcoreapp3.1	870ns	0.984ns	3.81ns	0.00897	672 B
#5340	`StartFinishScope`	net472	1.02μs	5.58ns	31.6ns	0.101	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5340

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.257	662.80	833.11

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	663ns	0.625ns	2.34ns	0.00962	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.03μs	1.01ns	3.9ns	0.00888	672 B
master	`RunOnMethodBegin`	net472	1.09μs	1.5ns	5.83ns	0.101	634 B
#5340	`RunOnMethodBegin`	net6.0	833ns	0.626ns	2.34ns	0.00958	672 B
#5340	`RunOnMethodBegin`	netcoreapp3.1	970ns	4.47ns	21.4ns	0.00887	672 B
#5340	`RunOnMethodBegin`	net472	1.04μs	1.21ns	4.7ns	0.101	634 B

andrewlock · 2024-03-23T01:04:01Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5340) (11.092M)   : 0, 11091603
    master (10.699M)   : 0, 10699271
    benchmarks/2.9.0 (10.973M)   : 0, 10972668

    section Automatic
    This PR (5340) (7.578M)   : 0, 7577582
    master (7.452M)   : 0, 7452401
    benchmarks/2.9.0 (7.971M)   : 0, 7971440

    section Trace stats
    This PR (5340) (7.946M)   : 0, 7946417
    master (7.779M)   : 0, 7778567

    section Manual
    This PR (5340) (9.561M)   : 0, 9560783
    master (9.257M)   : 0, 9257432

    section Manual + Automatic
    This PR (5340) (7.188M)   : 0, 7187669
    master (7.107M)   : 0, 7106665

    section Version Conflict
    This PR (5340) (6.550M)   : 0, 6549714
    master (6.474M)   : 0, 6473551

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5340) (9.397M)   : 0, 9396569
    master (9.619M)   : 0, 9619455
    benchmarks/2.9.0 (9.671M)   : 0, 9670561

    section Automatic
    This PR (5340) (6.501M)   : 0, 6501430
    master (6.638M)   : 0, 6638391

    section Trace stats
    This PR (5340) (6.814M)   : 0, 6814252
    master (6.953M)   : 0, 6953277

    section Manual
    This PR (5340) (8.225M)   : 0, 8225091
    master (8.186M)   : 0, 8186485

    section Manual + Automatic
    This PR (5340) (6.071M)   : 0, 6070830
    master (6.173M)   : 0, 6173255

    section Version Conflict
    This PR (5340) (5.684M)   : 0, 5684039
    master (5.708M)   : 0, 5707836

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5340) (9.002M)   : 0, 9002208
    master (9.742M)   : 0, 9742113
    benchmarks/2.9.0 (9.723M)   : 0, 9723215

    section Automatic
    This PR (5340) (6.952M)   : 0, 6951820
    master (6.856M)   : 0, 6855830
    benchmarks/2.9.0 (7.095M)   : 0, 7094697

    section Trace stats
    This PR (5340) (7.248M)   : 0, 7247518
    master (7.248M)   : 0, 7248493

    section Manual
    This PR (5340) (8.745M)   : 0, 8745381
    master (8.661M)   : 0, 8660963

    section Manual + Automatic
    This PR (5340) (6.686M)   : 0, 6686365
    master (6.578M)   : 0, 6577645

    section Version Conflict
    This PR (5340) (6.125M)   : 0, 6124819
    master (6.055M)   : 0, 6054820

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    benchmarks/2.9.0 (20.689M)   : 0, 20689363

    section No attack
    benchmarks/2.9.0 (20.448M)   : 0, 20448059

    section Attack
    benchmarks/2.9.0 (17.167M)   : 0, 17166673

andrewlock

LGTM just a couple of Qs

andrewlock · 2024-03-26T15:56:39Z

.gitlab/build-oci.sh

+    --version="$DOTNET_PACKAGE_VERSION" \
+    --package="datadog-apm-library-dotnet" \
+    --archive=true \
+    --archive-path="datadog-apm-library-dotnet-$DOTNET_PACKAGE_VERSION-$ARCH.tar" \


Do you have any plans to support Windows containers with this? If so, should the package name contain a platform too?

A single OCI package contains all targets (arch/os) as different layers, which can be downloaded indivually.
The name of the tar archive in the CI doesn't really matter, because we merge everything into a single OCI package during the release-package step. The archive just needs to contain the arch and OS so we don't overwrite artifacts across jobs.

andrewlock · 2024-03-26T15:57:34Z

.gitlab-ci.yml

+    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+(-prerelease)?$/'
+      when: manual
+      allow_failure: false


Is this safe to start pushing them immediately (on next release)? Our current release process will run this automatically

Yes, it's safe. The versions can only be used by the updater after we add them to the remote-config catalog, so releasing doesn't have an immediate effect

Add oci package build

ac5eed0

paullegranddc requested a review from a team as a code owner March 22, 2024 14:30

Add stage

efc7597

paullegranddc added 2 commits March 22, 2024 16:04

fix datadog-package invocation

4d6d5ce

make script executable

7da4099

andrewlock reviewed Mar 22, 2024

View reviewed changes

paullegranddc added 5 commits March 22, 2024 17:28

Fix unpack directory

5febdab

fix cmd

67b9eec

fix deb path in fpm

30988da

write version correctly

187ea13

remove trailing space

fc52bdd

BaptisteFoy reviewed Mar 22, 2024

View reviewed changes

andrewlock approved these changes Mar 26, 2024

View reviewed changes

paullegranddc merged commit c5ece6f into master Mar 26, 2024
56 checks passed

paullegranddc deleted the paullgdc/auto_updater/add_oci_package_build branch March 26, 2024 18:31

github-actions bot added this to the vNext milestone Mar 26, 2024

andrewlock modified the milestones: vNext, vNext-v2 Apr 17, 2024

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 17, 2024

[ci] Add oci package build #5340

[ci] Add oci package build #5340

Conversation

paullegranddc commented Mar 22, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Mar 22, 2024 • edited Loading

Datadog Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewlock commented Mar 22, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented Mar 22, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #5340

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5340

Raw results

Slower ⚠️ in #5340

Raw results

andrewlock commented Mar 23, 2024

Throughput/Crank Report:zap:

andrewlock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paullegranddc commented Mar 22, 2024 •

edited

Loading

datadog-ddstaging bot commented Mar 22, 2024 •

edited

Loading