[Build] Extend Azure Service Bus testing from versions 7.4.x - 7.17.x #5343

zacharycmontoya · 2024-03-22T19:50:13Z

Summary of changes

Adds multi-version package testing for Azure Service Bus and tests versions 7.4.x - 7.17.x.

⚠️ These tests still do not run automatically in CI because the tests work on a Service Bus in Azure. We would like to migrate to a local container, perhaps https://github.com/piotr-rojek/devopsifyme-sbemulator, but until then we should not run this in CI

Reason for change

OpenTelemetry support for Azure Service Bus starts with Azure.Messaging.ServiceBus v7.4.0 according to the Introducing experimental OpenTelemetry support in the Azure SDK for .NET, so we want to test all the versions that implement the OpenTelemetry support.

Implementation details

Adds first-class "message_bus.destination" field to AzureServiceBusTags since it is the previous form of the "messaging.destination.name" tag in the Azure Service Bus instrumentation
Extends the IsPresent and IsOptional tag assertions to take fallback values, in the case where tags change over package versions. These assert that only one of the specified tag names is present and will error if multiple are present.
Adds unique snapshot files where the span data changes: v7.4.x, v7.10.x, v7.13.x, and v7.17.x

Test coverage

Expanded snapshot testing. The Azure Service Bus integration tests do not run in CI, so I tested this on my dev machine by doing the following:

Unskip Datadog.Trace.ClrProfiler.IntegrationTests.Azure.AzureServiceBusTests.SubmitsTraces`
Set environment variable ASB_CONNECTION_STRING to the connection string of a Service Bus in Azure that is session-enabled
.\tracer\build.ps1 BuildAndRunWindowsIntegrationTests --skip PrintSnapshotsDiff -TestAllPackageVersions --framework net462 --filter AzureServiceBusTests --SampleName Samples.AzureServiceBus

Detailed snapshot changes

Incremental changes by version

v7.04.0: Baseline snapshot
v7.10.0: Adds new Message spans (17) that becomes part of active trace
v7.13.0:
- First ServiceBusProcessor.ProcessMessage becomes part of the first trace (hence ID changes everywhere)
- Tag values updated
  - span.kind: client => span.kind: consumer for the following spans:
    - ServiceBusProcessor.ProcessMessage
    - ServiceBusSessionProcessor.ProcessSessionMessage
    - ServiceBusReceiver.Peek
    - ServiceBusReceiver.Receive
    - ServiceBusReceiver.ReceiveDeferred
  - Change OperationName from client.request to servicebus.process|servicebus.publish|servicebus.receive|servicebus.settle for message operations (not management operations)
- Tags removed
  - kind
- Tags added
  - messaging.operation
  - az.schema_url: https://opentelemetry.io/schemas/1.17.0
- Tags renamed
  - component: servicebus => messaging.system: servicebus
  - message_bus.destination => messaging.source.name|messaging.destination.name
  - peer.address => net.peer.name
v7.17.0:
- Tag values updated
  - az.schema_url: https://opentelemetry.io/schemas/1.17.0 => az.schema_url: https://opentelemetry.io/schemas/1.23.0
  - otel.library.name: Azure.Messaging.ServiceBus => otel.library.name: Azure.Messaging.ServiceBus.Message for Message spans
- Tags renamed
  - net.peer.name: localhost => server.address: localhost
  - messaging.source.name: samples-azureservicebus-queue => messaging.destination.name: samples-azureservicebus-queue for the following spans:
    - ServiceBusReceiver.Receive
    - ServiceBusReceiver.ReceiveDeferred
    - ServiceBusProcessor.ProcessMessage
    - ServiceBusSessionProcessor.ProcessSessionMessage
    - ServiceBusReceiver.Peek
    - ServiceBusReceiver.Receive

_dd.peer.service.source tag

<7.13.0: message_bus.destination
7.13.0+: messaging.destination.name

Other details

N/A

…ld have been done long ago but we do not run these tests in CI

…e fallback tag names, where it throws an error when there are 0 matches or 2+ matches. This will be needed to easily support Azure Service Bus where the tag name changes between versions of the library and we still only expect one tag value.

Test changes include: - Creating separate snapshots for v7.4, v7.10, v7.13, and v7.17 - SpanMetadata updates to accommodate older versions where some tags differ. For cases where a tag name changed, introduce usage of IsPresent or IsOptional where we test that exclusively one or the other tag is set but not both Note: Also adds first-class "message_bus.destination" field to AzureServiceBusTags since it is the previous form of the "messaging.destination.name" tag

…ntly only done locally)

datadog-ddstaging · 2024-03-22T20:04:48Z

Datadog Report

Branch report: zach/service-bus-testing
Commit report: 43c007e
Test service: dd-trace-dotnet

✅ 0 Failed, 328189 Passed, 1588 Skipped, 54m 16.67s Wall Time

andrewlock · 2024-03-22T20:13:36Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (73ms)  : 64, 83
     .   : milestone, 73,
    master - mean (75ms)  : 62, 87
     .   : milestone, 75,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (1,004ms)  : 979, 1029
     .   : milestone, 1004,
    master - mean (1,003ms)  : 981, 1025
     .   : milestone, 1003,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (111ms)  : 107, 114
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (721ms)  : 697, 744
     .   : milestone, 721,
    master - mean (722ms)  : 703, 740
     .   : milestone, 722,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (94ms)  : 91, 98
     .   : milestone, 94,
    master - mean (94ms)  : 91, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (685ms)  : 654, 716
     .   : milestone, 685,
    master - mean (671ms)  : 648, 694
     .   : milestone, 671,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (188ms)  : 184, 191
     .   : milestone, 188,
    master - mean (188ms)  : 184, 192
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (1,076ms)  : 1054, 1098
     .   : milestone, 1076,
    master - mean (1,067ms)  : 1039, 1095
     .   : milestone, 1067,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (271ms)  : 266, 275
     .   : milestone, 271,
    master - mean (270ms)  : 263, 277
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (871ms)  : 845, 897
     .   : milestone, 871,
    master - mean (867ms)  : 846, 888
     .   : milestone, 867,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5343) - mean (261ms)  : 257, 264
     .   : milestone, 261,
    master - mean (260ms)  : 256, 265
     .   : milestone, 260,

    section CallTarget+Inlining+NGEN
    This PR (5343) - mean (858ms)  : 832, 885
     .   : milestone, 858,
    master - mean (850ms)  : 821, 879
     .   : milestone, 850,

andrewlock · 2024-03-22T20:34:58Z

Benchmarks Report for tracer 🐌

Benchmarks for #5343 compared to master:

1 benchmarks are faster, with geometric mean 1.146
1 benchmarks are slower, with geometric mean 1.148
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.56μs	46.9ns	293ns	0.0254	0.00847	0	7.5 KB
master	`StartStopWithChild`	netcoreapp3.1	10.8μs	58.5ns	325ns	0.0313	0.0157	0	7.6 KB
master	`StartStopWithChild`	net472	16.9μs	50.2ns	195ns	1.36	0.342	0.128	8.05 KB
#5343	`StartStopWithChild`	net6.0	8.97μs	49.5ns	284ns	0.027	0.0135	0	7.5 KB
#5343	`StartStopWithChild`	netcoreapp3.1	10.7μs	59.1ns	340ns	0.0217	0.0109	0	7.6 KB
#5343	`StartStopWithChild`	net472	17μs	53.7ns	208ns	1.36	0.375	0.117	8.04 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	450μs	515ns	1.99μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	614μs	1.07μs	4.13μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	813μs	922ns	3.19μs	0.403	3.3 KB
#5343	`WriteAndFlushEnrichedTraces`	net6.0	464μs	796ns	3.08μs	0	2.7 KB
#5343	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	598μs	188ns	702ns	0	2.7 KB
#5343	`WriteAndFlushEnrichedTraces`	net472	814μs	367ns	1.37μs	0.408	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	174μs	186ns	719ns	0.259	18.33 KB
master	`SendRequest`	netcoreapp3.1	192μs	352ns	1.32μs	0.192	20.49 KB
master	`SendRequest`	net472	0.000406ns	0.000198ns	0.000714ns	0	0 b
#5343	`SendRequest`	net6.0	174μs	209ns	808ns	0.173	18.33 KB
#5343	`SendRequest`	netcoreapp3.1	193μs	249ns	963ns	0.192	20.49 KB
#5343	`SendRequest`	net472	0.00146ns	0.000437ns	0.00169ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	538μs	1.32μs	5.1μs	0.546	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	648μs	1.36μs	5.09μs	0.326	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	net472	851μs	4.15μs	18.6μs	8.22	2.47	0.411	53.23 KB
#5343	`WriteAndFlushEnrichedTraces`	net6.0	548μs	1.41μs	5.48μs	0.566	0	0	41.72 KB
#5343	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	652μs	870ns	3.37μs	0.329	0	0	41.75 KB
#5343	`WriteAndFlushEnrichedTraces`	net472	875μs	4.26μs	17.6μs	8.3	2.62	0.437	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.15μs	0.534ns	2.07ns	0.011	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.5μs	4.73ns	18.3ns	0.0107	784 B
master	`ExecuteNonQuery`	net472	1.82μs	3.78ns	14.6ns	0.118	746 B
#5343	`ExecuteNonQuery`	net6.0	1.15μs	0.447ns	1.73ns	0.011	784 B
#5343	`ExecuteNonQuery`	netcoreapp3.1	1.54μs	1.25ns	4.83ns	0.0106	784 B
#5343	`ExecuteNonQuery`	net472	1.84μs	1.25ns	4.86ns	0.118	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5343

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0	1.148	1,179.47	1,354.20

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.18μs	1.17ns	4.54ns	0.014	0	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.59μs	0.396ns	1.48ns	0.0133	0	1.01 KB
master	`CallElasticsearch`	net472	2.52μs	0.743ns	2.58ns	0.161	0.00127	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.36μs	1.32ns	4.93ns	0.0135	0	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.78μs	0.759ns	2.94ns	0.0144	0	1.06 KB
master	`CallElasticsearchAsync`	net472	2.52μs	1.76ns	6.8ns	0.17	0	1.08 KB
#5343	`CallElasticsearch`	net6.0	1.35μs	1.35ns	5.05ns	0.0143	0	1.01 KB
#5343	`CallElasticsearch`	netcoreapp3.1	1.72μs	0.745ns	2.89ns	0.0137	0	1.01 KB
#5343	`CallElasticsearch`	net472	2.62μs	0.908ns	3.52ns	0.162	0	1.02 KB
#5343	`CallElasticsearchAsync`	net6.0	1.46μs	0.722ns	2.7ns	0.0139	0	984 B
#5343	`CallElasticsearchAsync`	netcoreapp3.1	1.72μs	0.925ns	3.46ns	0.0148	0	1.06 KB
#5343	`CallElasticsearchAsync`	net472	2.65μs	1.82ns	6.81ns	0.17	0	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.898ns	3.36ns	0.0128	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.81μs	1.27ns	4.9ns	0.0127	928 B
master	`ExecuteAsync`	net472	1.95μs	3.04ns	11.8ns	0.141	891 B
#5343	`ExecuteAsync`	net6.0	1.43μs	1.21ns	4.67ns	0.0128	928 B
#5343	`ExecuteAsync`	netcoreapp3.1	1.68μs	0.997ns	3.73ns	0.0125	928 B
#5343	`ExecuteAsync`	net472	1.88μs	0.663ns	2.48ns	0.141	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.04μs	1.84ns	6.89ns	0.0304	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.1μs	2.97ns	11.1ns	0.0355	2.7 KB
master	`SendAsync`	net472	7.66μs	4.95ns	19.2ns	0.483	3.05 KB
#5343	`SendAsync`	net6.0	4.11μs	1.64ns	6.14ns	0.0304	2.16 KB
#5343	`SendAsync`	netcoreapp3.1	5.01μs	2.22ns	8.58ns	0.035	2.7 KB
#5343	`SendAsync`	net472	7.72μs	4.21ns	15.7ns	0.482	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.51μs	0.775ns	2.9ns	0.0227	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.29μs	2.04ns	7.62ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	2.56μs	1.57ns	5.87ns	0.249	1.57 KB
#5343	`EnrichedLog`	net6.0	1.5μs	0.618ns	2.39ns	0.0234	1.65 KB
#5343	`EnrichedLog`	netcoreapp3.1	2.25μs	5.83ns	22.6ns	0.0226	1.65 KB
#5343	`EnrichedLog`	net472	2.66μs	2.25ns	8.42ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	115μs	153ns	593ns	0.057	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	170ns	637ns	0.0612	0	4.23 KB
master	`EnrichedLog`	net472	148μs	145ns	543ns	0.665	0.222	4.41 KB
#5343	`EnrichedLog`	net6.0	112μs	150ns	580ns	0.0562	0	4.23 KB
#5343	`EnrichedLog`	netcoreapp3.1	118μs	150ns	583ns	0.0586	0	4.23 KB
#5343	`EnrichedLog`	net472	148μs	145ns	561ns	0.661	0.22	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.1μs	0.901ns	3.49ns	0.03	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	2.82ns	10.6ns	0.0295	2.21 KB
master	`EnrichedLog`	net472	4.88μs	3.14ns	12.1ns	0.321	2.02 KB
#5343	`EnrichedLog`	net6.0	3.2μs	1.31ns	5.06ns	0.0305	2.21 KB
#5343	`EnrichedLog`	netcoreapp3.1	4.2μs	5.69ns	21.3ns	0.0292	2.21 KB
#5343	`EnrichedLog`	net472	5.03μs	2.3ns	8.6ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.726ns	2.72ns	0.0163	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.88μs	1.76ns	6.11ns	0.016	1.18 KB
master	`SendReceive`	net472	2.18μs	1.09ns	4.22ns	0.187	1.18 KB
#5343	`SendReceive`	net6.0	1.41μs	1.94ns	7.26ns	0.0163	1.18 KB
#5343	`SendReceive`	netcoreapp3.1	1.86μs	0.79ns	3.06ns	0.0158	1.18 KB
#5343	`SendReceive`	net472	2.3μs	2.71ns	10.5ns	0.187	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.9μs	1.41ns	5.47ns	0.0211	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	4.02μs	1.78ns	6.4ns	0.0203	1.6 KB
master	`EnrichedLog`	net472	4.46μs	2.12ns	7.92ns	0.316	1.99 KB
#5343	`EnrichedLog`	net6.0	2.75μs	0.575ns	2.23ns	0.0221	1.55 KB
#5343	`EnrichedLog`	netcoreapp3.1	3.97μs	4.74ns	18.4ns	0.0202	1.6 KB
#5343	`EnrichedLog`	net472	4.33μs	1.68ns	6.29ns	0.316	1.99 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5343

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.146	695.70	607.19

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	483ns	0.591ns	2.21ns	0.00776	552 B
master	`StartFinishSpan`	netcoreapp3.1	757ns	1.42ns	5.5ns	0.0073	552 B
master	`StartFinishSpan`	net472	708ns	0.949ns	3.29ns	0.0879	554 B
master	`StartFinishScope`	net6.0	697ns	1.25ns	4.86ns	0.00956	672 B
master	`StartFinishScope`	netcoreapp3.1	878ns	1.42ns	5.49ns	0.00895	672 B
master	`StartFinishScope`	net472	901ns	1.66ns	6.43ns	0.101	634 B
#5343	`StartFinishSpan`	net6.0	524ns	0.664ns	2.57ns	0.00768	552 B
#5343	`StartFinishSpan`	netcoreapp3.1	705ns	0.686ns	2.66ns	0.00766	552 B
#5343	`StartFinishSpan`	net472	772ns	0.313ns	1.13ns	0.0879	554 B
#5343	`StartFinishScope`	net6.0	607ns	0.994ns	3.85ns	0.00928	672 B
#5343	`StartFinishScope`	netcoreapp3.1	865ns	1.05ns	4.08ns	0.00886	672 B
#5343	`StartFinishScope`	net472	940ns	1.22ns	4.74ns	0.1	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	638ns	1.08ns	4.18ns	0.00937	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.05μs	1.53ns	5.92ns	0.00914	672 B
master	`RunOnMethodBegin`	net472	1.22μs	1.84ns	7.13ns	0.1	634 B
#5343	`RunOnMethodBegin`	net6.0	649ns	0.99ns	3.84ns	0.00932	672 B
#5343	`RunOnMethodBegin`	netcoreapp3.1	1.03μs	1.32ns	4.93ns	0.00931	672 B
#5343	`RunOnMethodBegin`	net472	1.13μs	1ns	3.75ns	0.1	634 B

andrewlock · 2024-03-23T01:33:16Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5343) (11.249M)   : 0, 11248750
    master (10.699M)   : 0, 10699271
    benchmarks/2.9.0 (10.973M)   : 0, 10972668

    section Automatic
    This PR (5343) (7.740M)   : 0, 7739634
    master (7.452M)   : 0, 7452401
    benchmarks/2.9.0 (7.971M)   : 0, 7971440

    section Trace stats
    This PR (5343) (7.980M)   : 0, 7980474
    master (7.779M)   : 0, 7778567

    section Manual
    This PR (5343) (9.642M)   : 0, 9642498
    master (9.257M)   : 0, 9257432

    section Manual + Automatic
    This PR (5343) (7.331M)   : 0, 7331281
    master (7.107M)   : 0, 7106665

    section Version Conflict
    This PR (5343) (6.586M)   : 0, 6586259
    master (6.474M)   : 0, 6473551

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5343) (9.654M)   : 0, 9654140
    master (9.619M)   : 0, 9619455
    benchmarks/2.9.0 (9.671M)   : 0, 9670561

    section Automatic
    This PR (5343) (6.705M)   : 0, 6705403
    master (6.638M)   : 0, 6638391

    section Trace stats
    This PR (5343) (6.941M)   : 0, 6940634
    master (6.953M)   : 0, 6953277

    section Manual
    This PR (5343) (8.302M)   : 0, 8301666
    master (8.186M)   : 0, 8186485

    section Manual + Automatic
    This PR (5343) (6.212M)   : 0, 6211947
    master (6.173M)   : 0, 6173255

    section Version Conflict
    This PR (5343) (5.701M)   : 0, 5701063
    master (5.708M)   : 0, 5707836

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5343) (9.700M)   : 0, 9699742
    master (9.742M)   : 0, 9742113
    benchmarks/2.9.0 (9.723M)   : 0, 9723215

    section Automatic
    This PR (5343) (6.893M)   : 0, 6892951
    master (6.856M)   : 0, 6855830
    benchmarks/2.9.0 (7.095M)   : 0, 7094697

    section Trace stats
    This PR (5343) (7.303M)   : 0, 7303454
    master (7.248M)   : 0, 7248493

    section Manual
    This PR (5343) (8.713M)   : 0, 8713270
    master (8.661M)   : 0, 8660963

    section Manual + Automatic
    This PR (5343) (6.639M)   : 0, 6638901
    master (6.578M)   : 0, 6577645

    section Version Conflict
    This PR (5343) (6.000M)   : 0, 6000019
    master (6.055M)   : 0, 6054820

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    benchmarks/2.9.0 (20.689M)   : 0, 20689363

    section No attack
    benchmarks/2.9.0 (20.448M)   : 0, 20448059

    section Attack
    benchmarks/2.9.0 (17.167M)   : 0, 17166673

andrewlock

LGTM, though I'm taking your word for it on the actual snapshots!

tracer/build/PackageVersionsGeneratorDefinitions.json

tracer/src/Datadog.Trace/Tagging/AzureServiceBusTags.cs

andrewlock · 2024-03-25T12:02:55Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/Azure/AzureServiceBusTests.cs

+                var snapshotSuffix = version switch
+                {
+                    { Major: 7, Minor: >= 17 } => "7_17",
+                    { Major: 7, Minor: >= 13 } => "7_13",
+                    { Major: 7, Minor: >= 10 } => "7_10",
+                    _ => "7_04"
+                };


...r/test/test-applications/integrations/Samples.AzureServiceBus/Samples.AzureServiceBus.csproj

andrewlock · 2024-03-25T12:10:10Z

tracer/test/snapshots/AzureServiceBusTests_7_17.SchemaV1.verified.txt

View of the file is here: https://andrewlock.github.io/DatadogSpanSnapshotViewer/?q=https%3A%2F%2Fgithub.com%2FDataDog%2Fdd-trace-dotnet%2Fblob%2Fee80db0183fe0a877a4dacc4eaef02170682899f%2Ftracer%2Ftest%2Fsnapshots%2FAzureServiceBusTests_7_17.SchemaV1.verified.txt

This shows that we get a lot of isolated spans for (e.g. complete, defer, peek) etc. I assume that's all as expected, just flagging 🙂

Thanks! I added a new Detailed snapshot changes in the PR description that is an exhaustive description of how the snapshot changes between library versions 😄

- Update PackageVersionsGeneratorDefinitions.json for the Azure.Messaging.ServiceBus NuGet packages we test - Added a small code comment that explains the difference in Message spans

zacharycmontoya added 5 commits March 22, 2024 12:07

Update AzureServiceBusTests snapshots to remove git tags -- this shou…

79a4d77

…ld have been done long ago but we do not run these tests in CI

Fix the -TestAllPackageVersions flag for CompileSamplesWindows (curre…

e5e1d1a

…ntly only done locally)

Refactor SpanTagAssertion logic

ee80db0

zacharycmontoya self-assigned this Mar 22, 2024

zacharycmontoya requested review from a team as code owners March 22, 2024 19:50

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Mar 22, 2024

andrewlock approved these changes Mar 25, 2024

View reviewed changes

lucaspimentel approved these changes Mar 25, 2024

View reviewed changes

PR Feedback:

43c007e

- Update PackageVersionsGeneratorDefinitions.json for the Azure.Messaging.ServiceBus NuGet packages we test - Added a small code comment that explains the difference in Message spans

zacharycmontoya merged commit 4144021 into master Mar 25, 2024
54 of 56 checks passed

zacharycmontoya deleted the zach/service-bus-testing branch March 25, 2024 19:35

github-actions bot added this to the vNext milestone Mar 25, 2024

andrewlock modified the milestones: vNext, vNext-v2 Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] Extend Azure Service Bus testing from versions 7.4.x - 7.17.x #5343

[Build] Extend Azure Service Bus testing from versions 7.4.x - 7.17.x #5343

zacharycmontoya commented Mar 22, 2024 •

edited

Loading

datadog-ddstaging bot commented Mar 22, 2024 •

edited

Loading

andrewlock commented Mar 22, 2024 •

edited

Loading

andrewlock commented Mar 22, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5343

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5343

Raw results

Raw results

andrewlock commented Mar 23, 2024

andrewlock left a comment

andrewlock Mar 25, 2024

andrewlock Mar 25, 2024

zacharycmontoya Mar 25, 2024

[Build] Extend Azure Service Bus testing from versions 7.4.x - 7.17.x #5343

[Build] Extend Azure Service Bus testing from versions 7.4.x - 7.17.x #5343

Conversation

zacharycmontoya commented Mar 22, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Detailed snapshot changes

Incremental changes by version

_dd.peer.service.source tag

Other details

datadog-ddstaging bot commented Mar 22, 2024 • edited Loading

Datadog Report

andrewlock commented Mar 22, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Mar 22, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5343

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5343

Raw results

Raw results

andrewlock commented Mar 23, 2024

Throughput/Crank Report:zap:

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock Mar 25, 2024

Choose a reason for hiding this comment

andrewlock Mar 25, 2024

Choose a reason for hiding this comment

zacharycmontoya Mar 25, 2024

Choose a reason for hiding this comment

zacharycmontoya commented Mar 22, 2024 •

edited

Loading

datadog-ddstaging bot commented Mar 22, 2024 •

edited

Loading

andrewlock commented Mar 22, 2024 •

edited

Loading

andrewlock commented Mar 22, 2024 •

edited

Loading