Reinstate version mismatch tests #4879

andrewlock · 2023-11-16T16:41:44Z

Summary of changes

Reinstate the version-mismatch tests that were disabled in the .NET 8 upgrade

Reason for change

We want to run these tests, we just couldn't easily before

Implementation details

Pushed up a new "before" and "after" image that works with .NET 8 to the Azure Devops feed. We explicitly needed new packages because <2.41.0 is broken in .NET 8

Test coverage

Reinstated all the places we had previously skipped (I believe), so should be back to previous coverage.

Other details

The one exception currently are the trace annotation tests - we disabled the telemetry check before because we removed support for v1 from the repo, so it was causing issues. I don't think this is something we should worry about tbh.

github-actions · 2023-11-16T17:56:03Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

60 occurrences of :

+      _dd.git.commit.sha: aaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbb,
+      _dd.git.repository_url: https://github.com/DataDog/dd-trace-dotnet

datadog-ddstaging · 2023-11-16T18:21:15Z

Datadog Report

Branch report: andrew/ci/reinstate-version-mismatch
Commit report: 50b739e

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 298421 Passed, 937 Skipped, 37m 40.47s Wall Time

New Flaky Tests (1)

HttpClient_SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.HttpMessageHandlerTests - Last Failure

Expand for error

 Expected varintegration=latestIntegrations {["WinHttpHandler"] = IntegrationTelemetryData { Name = WinHttpHandler, Enabled = True, AutoEnabled = True, Error =  }, ["HttpMessageHandler"] = IntegrationTelemetryData { Name = HttpMessageHandler, Enabled = True, AutoEnabled = True, Error =  }} to contain key "HttpSocketsHandler".
 Expected varintegration=latestIntegrations {["WinHttpHandler"] = IntegrationTelemetryData { Name = WinHttpHandler, Enabled = True, AutoEnabled = True, Error =  }, ["HttpMessageHandler"] = IntegrationTelemetryData { Name = HttpMessageHandler, Enabled = True, AutoEnabled = True, Error =  }} to contain key "CurlHandler".

andrewlock · 2023-11-16T18:58:44Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (72ms)  : 63, 82
     .   : milestone, 72,
    master - mean (73ms)  : 61, 85
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (1,009ms)  : 985, 1032
     .   : milestone, 1009,
    master - mean (1,000ms)  : 975, 1024
     .   : milestone, 1000,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (106ms)  : 102, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (689ms)  : 673, 705
     .   : milestone, 689,
    master - mean (690ms)  : 672, 708
     .   : milestone, 690,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (91ms)  : 88, 94
     .   : milestone, 91,
    master - mean (91ms)  : 88, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (666ms)  : 648, 685
     .   : milestone, 666,
    master - mean (666ms)  : 629, 703
     .   : milestone, 666,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (188ms)  : 185, 191
     .   : milestone, 188,
    master - mean (189ms)  : 186, 191
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (1,132ms)  : 1107, 1157
     .   : milestone, 1132,
    master - mean (1,131ms)  : 1108, 1155
     .   : milestone, 1131,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (272ms)  : 266, 277
     .   : milestone, 272,
    master - mean (272ms)  : 268, 276
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (1,093ms)  : 1061, 1124
     .   : milestone, 1093,
    master - mean (1,086ms)  : 1060, 1111
     .   : milestone, 1086,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4879) - mean (261ms)  : 258, 265
     .   : milestone, 261,
    master - mean (260ms)  : 256, 264
     .   : milestone, 260,

    section CallTarget+Inlining+NGEN
    This PR (4879) - mean (1,064ms)  : 1043, 1085
     .   : milestone, 1064,
    master - mean (1,052ms)  : 1031, 1072
     .   : milestone, 1052,

…kage" This reverts commit 964bc39c6132b4701f48d4f873e58d441cf9511b.

andrewlock · 2023-11-17T14:42:24Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/TraceAnnotationsTests.cs

@@ -50,7 +40,7 @@ public class TraceAnnotationsVersionMismatchBeforeFeatureTests : TraceAnnotation
        public TraceAnnotationsVersionMismatchBeforeFeatureTests(ITestOutputHelper output)
            : base("TraceAnnotations.VersionMismatch.BeforeFeature", twoAssembliesLoaded: true, enableTelemetry: false, output)
        {
-#if NET8_0
+#if NET8_0_OR_GREATER


We won't ever be able to test this on .NET 8+

andrewlock · 2023-11-17T14:42:42Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/TraceAnnotationsTests.cs

@@ -66,18 +56,9 @@ public TraceAnnotationsVersionMismatchBeforeFeatureTests(ITestOutputHelper outpu
    public class TraceAnnotationsVersionMismatchNewerNuGetTests : TraceAnnotationsTests
    {
        public TraceAnnotationsVersionMismatchNewerNuGetTests(ITestOutputHelper output)
-#if NETFRAMEWORK


This weirdness seems to have resolved in the GA SDK

andrewlock · 2023-11-17T14:43:12Z

tracer/test/snapshots/TraceAnnotationsVersionMismatchNewerNuGetTests._.verified.txt

+      _dd.git.commit.sha: aaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbb,
+      _dd.git.repository_url: https://github.com/DataDog/dd-trace-dotnet


Missed in the "normal" update because the test was skipped

andrewlock · 2023-11-17T14:43:37Z

...ns.VersionMismatch.AfterFeature/Samples.TraceAnnotations.VersionMismatch.AfterFeature.csproj

@@ -6,7 +6,7 @@
  </ItemGroup>

  <ItemGroup>
-    <PackageReference Include="Datadog.Trace" Version="2.5.0" />
+    <PackageReference Include="Datadog.Trace" Version="2.41.0" />


This is the only "older" version which supports .NET 8

andrewlock · 2023-11-17T19:24:43Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4879) (11.150M)   : 0, 11150205
    master (10.978M)   : 0, 10978371
    benchmarks/2.9.0 (11.337M)   : 0, 11336710

    section Automatic
    This PR (4879) (7.685M)   : 0, 7685270
    master (7.556M)   : 0, 7556124
    benchmarks/2.9.0 (8.117M)   : 0, 8116938

    section Trace stats
    This PR (4879) (7.992M)   : 0, 7992124
    master (7.863M)   : 0, 7863349

    section Manual
    This PR (4879) (9.785M)   : 0, 9785382
    master (9.551M)   : 0, 9551488

    section Manual + Automatic
    This PR (4879) (7.394M)   : 0, 7393660
    master (7.243M)   : 0, 7242812

    section Version Conflict
    This PR (4879) (6.537M)   : 0, 6536602
    master (6.534M)   : 0, 6534405

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4879) (9.484M)   : 0, 9484180
    master (9.377M)   : 0, 9376531
    benchmarks/2.9.0 (9.584M)   : 0, 9583569

    section Automatic
    This PR (4879) (6.498M)   : 0, 6497966
    master (6.667M)   : 0, 6667445

    section Trace stats
    This PR (4879) (6.925M)   : 0, 6924769
    master (6.916M)   : 0, 6915894

    section Manual
    This PR (4879) (8.397M)   : 0, 8396934
    master (8.159M)   : 0, 8159374

    section Manual + Automatic
    This PR (4879) (6.276M)   : 0, 6275625
    master (6.161M)   : 0, 6160792

    section Version Conflict
    This PR (4879) (5.768M)   : 0, 5768228
    master (5.693M)   : 0, 5692502

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4879) (10.720M)   : 0, 10720077
    benchmarks/2.9.0 (10.428M)   : 0, 10428197

    section Automatic
    This PR (4879) (7.552M)   : 0, 7552268
    benchmarks/2.9.0 (7.850M)   : 0, 7850029

    section Trace stats
    This PR (4879) (7.787M)   : 0, 7787042

    section Manual
    This PR (4879) (9.184M)   : 0, 9183849

    section Manual + Automatic
    This PR (4879) (7.198M)   : 0, 7198086

    section Version Conflict
    This PR (4879) (6.545M)   : 0, 6544545

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.406M)   : 0, 7405989
    benchmarks/2.9.0 (7.902M)   : 0, 7901545

    section No attack
    master (2.073M)   : 0, 2072570
    benchmarks/2.9.0 (3.255M)   : 0, 3255073

    section Attack
    master (1.655M)   : 0, 1654787
    benchmarks/2.9.0 (2.551M)   : 0, 2551268

    section Blocking
    master (3.449M)   : 0, 3448566

    section IAST default
    master (6.437M)   : 0, 6437355

    section IAST full
    master (5.649M)   : 0, 5648815

    section Base vuln
    master (0.947M)   : 0, 946861

    section IAST vuln
    master (0.884M)   : 0, 883725

andrewlock force-pushed the andrew/ci/reinstate-version-mismatch branch from 280991d to d6b9d05 Compare November 16, 2023 17:55

github-actions bot added the area:tests unit tests, integration tests label Nov 16, 2023

andrewlock force-pushed the andrew/ci/reinstate-version-mismatch branch from d6b9d05 to 7b3d2af Compare November 17, 2023 10:28

andrewlock added 5 commits November 17, 2023 14:37

Reinstate tracer version mismatch tests where possible

029232a

Update package versions to support .NET 8 in version mismatch scenarios

a46fcad

Temporarily update the version to build the Datadog.Trace package

f634c36

Revert "Temporarily update the version to build the Datadog.Trace pac…

25413ec

…kage" This reverts commit 964bc39c6132b4701f48d4f873e58d441cf9511b.

Update snapshots

50b739e

andrewlock force-pushed the andrew/ci/reinstate-version-mismatch branch from 7b3d2af to 50b739e Compare November 17, 2023 14:37

andrewlock commented Nov 17, 2023

View reviewed changes

andrewlock marked this pull request as ready for review November 17, 2023 15:37

andrewlock requested a review from a team as a code owner November 17, 2023 15:37

zacharycmontoya approved these changes Nov 17, 2023

View reviewed changes

andrewlock merged commit 9e0d614 into master Nov 22, 2023
52 of 54 checks passed

andrewlock deleted the andrew/ci/reinstate-version-mismatch branch November 22, 2023 08:58

github-actions bot added this to the vNext milestone Nov 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reinstate version mismatch tests #4879

Reinstate version mismatch tests #4879

andrewlock commented Nov 16, 2023 •

edited

Loading

github-actions bot commented Nov 16, 2023

datadog-ddstaging bot commented Nov 16, 2023 •

edited

Loading

andrewlock commented Nov 16, 2023 •

edited

Loading

andrewlock Nov 17, 2023

andrewlock Nov 17, 2023

andrewlock Nov 17, 2023

andrewlock Nov 17, 2023

andrewlock commented Nov 17, 2023

		_dd.git.commit.sha: aaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbb,
		_dd.git.repository_url: https://github.com/DataDog/dd-trace-dotnet

Reinstate version mismatch tests #4879

Reinstate version mismatch tests #4879

Conversation

andrewlock commented Nov 16, 2023 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

github-actions bot commented Nov 16, 2023

Snapshots difference summary

datadog-ddstaging bot commented Nov 16, 2023 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Nov 16, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock Nov 17, 2023

Choose a reason for hiding this comment

andrewlock Nov 17, 2023

Choose a reason for hiding this comment

andrewlock Nov 17, 2023

Choose a reason for hiding this comment

andrewlock Nov 17, 2023

Choose a reason for hiding this comment

andrewlock commented Nov 17, 2023

Throughput/Crank Report:zap:

andrewlock commented Nov 16, 2023 •

edited

Loading

datadog-ddstaging bot commented Nov 16, 2023 •

edited

Loading

andrewlock commented Nov 16, 2023 •

edited

Loading