Don't print snapshots diff unless running in CI #5379

andrewlock · 2024-04-02T12:11:41Z

Summary of changes

Only prints the snapshot diff at the end of the run if running in CI

Reason for change

It adds quite a lot of noise and confusion when running locally, and consensus seems to be to disable it

Implementation details

Check IsServerBuild variable (I'm assuming that works as expected 😅)

Test coverage

I'll check this build and make sure it runs. Already tested locally that it doesn't run.

Other details

Alternative approach was here:

Add a step to clean snapshots before running tests #5045

datadog-ddstaging · 2024-04-02T12:32:59Z

Datadog Report

Branch report: andrew/ci/dont-print-snapshot-diff-by-default
Commit report: 30440cf
Test service: dd-trace-dotnet

✅ 0 Failed, 337377 Passed, 1603 Skipped, 41m 23.32s Wall Time
❄️ 1 New Flaky

New Flaky Tests (1)

SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.NpgsqlCommandTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 139.
```

andrewlock · 2024-04-02T12:34:38Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (74ms)  : 65, 83
     .   : milestone, 74,
    master - mean (73ms)  : 65, 80
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (1,001ms)  : 978, 1024
     .   : milestone, 1001,
    master - mean (1,003ms)  : 983, 1022
     .   : milestone, 1003,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (110ms)  : 106, 114
     .   : milestone, 110,
    master - mean (111ms)  : 106, 115
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (721ms)  : 695, 747
     .   : milestone, 721,
    master - mean (717ms)  : 692, 742
     .   : milestone, 717,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (94ms)  : 92, 97
     .   : milestone, 94,
    master - mean (94ms)  : 90, 98
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (679ms)  : 662, 696
     .   : milestone, 679,
    master - mean (676ms)  : 651, 700
     .   : milestone, 676,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (189ms)  : 186, 192
     .   : milestone, 189,
    master - mean (187ms)  : 182, 193
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (1,081ms)  : 1050, 1113
     .   : milestone, 1081,
    master - mean (1,077ms)  : 1056, 1097
     .   : milestone, 1077,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (270ms)  : 264, 277
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (874ms)  : 843, 905
     .   : milestone, 874,
    master - mean (868ms)  : 847, 889
     .   : milestone, 868,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5379) - mean (261ms)  : 256, 266
     .   : milestone, 261,
    master - mean (260ms)  : 253, 266
     .   : milestone, 260,

    section CallTarget+Inlining+NGEN
    This PR (5379) - mean (853ms)  : 836, 869
     .   : milestone, 853,
    master - mean (855ms)  : 824, 887
     .   : milestone, 855,

andrewlock · 2024-04-02T12:59:48Z

Benchmarks Report for tracer 🐌

Benchmarks for #5379 compared to master:

1 benchmarks are faster, with geometric mean 1.145
1 benchmarks are slower, with geometric mean 1.202
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.74μs	48.6ns	315ns	0.0211	0.00846	0	7.5 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	58.2ns	363ns	0.0257	0.0103	0	7.6 KB
master	`StartStopWithChild`	net472	16.9μs	40.9ns	158ns	1.35	0.351	0.117	8.05 KB
#5379	`StartStopWithChild`	net6.0	8.94μs	49.7ns	329ns	0.0218	0.00871	0	7.5 KB
#5379	`StartStopWithChild`	netcoreapp3.1	11.1μs	60.4ns	347ns	0.0274	0.00548	0	7.6 KB
#5379	`StartStopWithChild`	net472	16.9μs	76.9ns	298ns	1.37	0.361	0.0924	8.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	475μs	140ns	524ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	600μs	191ns	716ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	818μs	232ns	900ns	0.408	3.3 KB
#5379	`WriteAndFlushEnrichedTraces`	net6.0	459μs	188ns	703ns	0	2.7 KB
#5379	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	622μs	94.1ns	365ns	0	2.7 KB
#5379	`WriteAndFlushEnrichedTraces`	net472	813μs	196ns	761ns	0.406	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	172μs	150ns	580ns	0.257	18.33 KB
master	`SendRequest`	netcoreapp3.1	193μs	294ns	1.14μs	0.191	20.49 KB
master	`SendRequest`	net472	0.000103ns	7.47E‑05ns	0.00028ns	0	0 b
#5379	`SendRequest`	net6.0	174μs	413ns	1.6μs	0.175	18.33 KB
#5379	`SendRequest`	netcoreapp3.1	192μs	279ns	1.04μs	0.192	20.49 KB
#5379	`SendRequest`	net472	0.00112ns	0.000278ns	0.00108ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #5379

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.67 KB	41.45 KB	-217 B	-0.52%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	551μs	445ns	1.73μs	0.563	0	0	41.67 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	2.41μs	9.34μs	0.314	0	0	41.67 KB
master	`WriteAndFlushEnrichedTraces`	net472	868μs	3.75μs	14.5μs	8.08	2.55	0.425	53.25 KB
#5379	`WriteAndFlushEnrichedTraces`	net6.0	556μs	207ns	745ns	0.548	0	0	41.45 KB
#5379	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	681μs	802ns	3.11μs	0.332	0	0	41.77 KB
#5379	`WriteAndFlushEnrichedTraces`	net472	848μs	3.99μs	15.4μs	8.22	2.47	0.411	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.21μs	0.81ns	3.14ns	0.0109	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.53μs	2.36ns	9.16ns	0.0106	784 B
master	`ExecuteNonQuery`	net472	1.8μs	0.78ns	2.92ns	0.118	746 B
#5379	`ExecuteNonQuery`	net6.0	1.18μs	3.69ns	14.3ns	0.0111	784 B
#5379	`ExecuteNonQuery`	netcoreapp3.1	1.46μs	0.359ns	1.34ns	0.0103	784 B
#5379	`ExecuteNonQuery`	net472	1.88μs	0.467ns	1.81ns	0.118	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.33μs	0.504ns	1.75ns	0.014	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.6μs	1.05ns	3.92ns	0.0137	1.01 KB
master	`CallElasticsearch`	net472	2.51μs	1.76ns	6.81ns	0.162	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.34μs	3.64ns	14.1ns	0.014	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.73μs	0.781ns	2.82ns	0.0147	1.06 KB
master	`CallElasticsearchAsync`	net472	2.61μs	2.03ns	7.86ns	0.17	1.08 KB
#5379	`CallElasticsearch`	net6.0	1.35μs	1.29ns	5ns	0.0141	1.01 KB
#5379	`CallElasticsearch`	netcoreapp3.1	1.68μs	1.15ns	4.3ns	0.0134	1.01 KB
#5379	`CallElasticsearch`	net472	2.53μs	2.49ns	9.32ns	0.162	1.02 KB
#5379	`CallElasticsearchAsync`	net6.0	1.38μs	0.702ns	2.63ns	0.0133	984 B
#5379	`CallElasticsearchAsync`	netcoreapp3.1	1.73μs	0.672ns	2.51ns	0.014	1.06 KB
#5379	`CallElasticsearchAsync`	net472	2.71μs	1.79ns	6.92ns	0.171	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.4μs	0.62ns	2.4ns	0.0134	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.74μs	0.857ns	3.32ns	0.0121	928 B
master	`ExecuteAsync`	net472	1.95μs	1.08ns	4.02ns	0.141	891 B
#5379	`ExecuteAsync`	net6.0	1.34μs	0.687ns	2.57ns	0.0128	928 B
#5379	`ExecuteAsync`	netcoreapp3.1	1.69μs	1.13ns	4.22ns	0.0128	928 B
#5379	`ExecuteAsync`	net472	1.94μs	1.23ns	4.76ns	0.142	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.19μs	1.8ns	6.73ns	0.0293	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.09μs	1.39ns	5.21ns	0.0356	2.7 KB
master	`SendAsync`	net472	7.71μs	2.58ns	10ns	0.483	3.05 KB
#5379	`SendAsync`	net6.0	4.1μs	1.66ns	6.22ns	0.0287	2.16 KB
#5379	`SendAsync`	netcoreapp3.1	4.99μs	4.46ns	16.7ns	0.0349	2.7 KB
#5379	`SendAsync`	net472	7.62μs	3.84ns	14.4ns	0.484	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.6μs	0.497ns	1.86ns	0.0231	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.24μs	1.85ns	6.39ns	0.0225	1.65 KB
master	`EnrichedLog`	net472	2.72μs	1.2ns	4.48ns	0.249	1.57 KB
#5379	`EnrichedLog`	net6.0	1.51μs	0.891ns	3.21ns	0.0232	1.65 KB
#5379	`EnrichedLog`	netcoreapp3.1	2.21μs	1.36ns	5.28ns	0.0221	1.65 KB
#5379	`EnrichedLog`	net472	2.69μs	2.25ns	8.71ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	65.3ns	253ns	0.0559	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	110ns	413ns	0	0	4.23 KB
master	`EnrichedLog`	net472	146μs	58.7ns	220ns	0.657	0.219	4.41 KB
#5379	`EnrichedLog`	net6.0	111μs	68.7ns	257ns	0.0554	0	4.23 KB
#5379	`EnrichedLog`	netcoreapp3.1	116μs	87.1ns	326ns	0.0579	0	4.23 KB
#5379	`EnrichedLog`	net472	147μs	91.7ns	355ns	0.658	0.219	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.18μs	8.4ns	32.6ns	0.0299	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.22μs	2.59ns	10ns	0.0295	2.21 KB
master	`EnrichedLog`	net472	4.99μs	2.31ns	8.66ns	0.32	2.02 KB
#5379	`EnrichedLog`	net6.0	3.11μs	1.34ns	4.83ns	0.0311	2.21 KB
#5379	`EnrichedLog`	netcoreapp3.1	4.22μs	1.32ns	4.95ns	0.0298	2.21 KB
#5379	`EnrichedLog`	net472	4.79μs	1.88ns	7.03ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.4μs	1.35ns	5.06ns	0.0162	0	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	1.84ns	6.9ns	0.0154	0	1.18 KB
master	`SendReceive`	net472	2.27μs	2.9ns	11.2ns	0.187	0.00112	1.18 KB
#5379	`SendReceive`	net6.0	1.53μs	0.901ns	3.37ns	0.0167	0	1.18 KB
#5379	`SendReceive`	netcoreapp3.1	1.87μs	2.04ns	7.65ns	0.0159	0	1.18 KB
#5379	`SendReceive`	net472	2.33μs	1.68ns	6.04ns	0.187	0.00116	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.77μs	0.614ns	2.3ns	0.0221	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	1.41ns	5.46ns	0.0219	1.6 KB
master	`EnrichedLog`	net472	4.52μs	3.24ns	12.5ns	0.315	1.99 KB
#5379	`EnrichedLog`	net6.0	2.77μs	0.691ns	2.58ns	0.0208	1.55 KB
#5379	`EnrichedLog`	netcoreapp3.1	3.86μs	1.8ns	6.5ns	0.0213	1.6 KB
#5379	`EnrichedLog`	net472	4.33μs	2.15ns	8.31ns	0.315	1.99 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5379

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.202	477.87	574.21

Faster 🎉 in #5379

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.145	1,053.55	920.27

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	477ns	0.952ns	3.69ns	0.00779	552 B
master	`StartFinishSpan`	netcoreapp3.1	696ns	1.36ns	5.25ns	0.00753	552 B
master	`StartFinishSpan`	net472	766ns	1.12ns	4.33ns	0.0877	554 B
master	`StartFinishScope`	net6.0	680ns	1.02ns	3.95ns	0.00946	672 B
master	`StartFinishScope`	netcoreapp3.1	850ns	1.39ns	5.39ns	0.00888	672 B
master	`StartFinishScope`	net472	1.05μs	2.41ns	9.35ns	0.101	634 B
#5379	`StartFinishSpan`	net6.0	575ns	0.761ns	2.95ns	0.00755	552 B
#5379	`StartFinishSpan`	netcoreapp3.1	735ns	1.86ns	7.2ns	0.00724	552 B
#5379	`StartFinishSpan`	net472	750ns	2.22ns	8.59ns	0.0879	554 B
#5379	`StartFinishScope`	net6.0	665ns	1.13ns	4.36ns	0.00932	672 B
#5379	`StartFinishScope`	netcoreapp3.1	813ns	1.18ns	4.58ns	0.00912	672 B
#5379	`StartFinishScope`	net472	920ns	1.18ns	4.55ns	0.1	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	641ns	1.4ns	5.41ns	0.00933	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	982ns	1.92ns	7.42ns	0.00887	672 B
master	`RunOnMethodBegin`	net472	1.11μs	2.85ns	11ns	0.1	634 B
#5379	`RunOnMethodBegin`	net6.0	673ns	1.5ns	5.81ns	0.00951	672 B
#5379	`RunOnMethodBegin`	netcoreapp3.1	947ns	0.95ns	3.68ns	0.00924	672 B
#5379	`RunOnMethodBegin`	net472	1.09μs	2.08ns	8.06ns	0.1	634 B

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 2, 2024

andrewlock requested a review from a team as a code owner April 2, 2024 12:11

bouwkast approved these changes Apr 2, 2024

View reviewed changes

robertpi approved these changes Apr 2, 2024

View reviewed changes

andrewlock force-pushed the andrew/ci/dont-print-snapshot-diff-by-default branch from a9151c3 to 2b23f04 Compare April 2, 2024 13:05

Don't print snapshots diff unless running in CI

30440cf

andrewlock force-pushed the andrew/ci/dont-print-snapshot-diff-by-default branch from 2b23f04 to 30440cf Compare April 3, 2024 11:39

andrewlock merged commit ad9f44c into master Apr 3, 2024
53 of 56 checks passed

andrewlock deleted the andrew/ci/dont-print-snapshot-diff-by-default branch April 3, 2024 14:03

github-actions bot added this to the vNext milestone Apr 3, 2024

andrewlock modified the milestones: vNext, vNext-v2 Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't print snapshots diff unless running in CI #5379

Don't print snapshots diff unless running in CI #5379

andrewlock commented Apr 2, 2024

datadog-ddstaging bot commented Apr 2, 2024 •

edited

Loading

andrewlock commented Apr 2, 2024 •

edited

Loading

andrewlock commented Apr 2, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #5379

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5379

Faster 🎉 in #5379

Raw results

Raw results

Don't print snapshots diff unless running in CI #5379

Don't print snapshots diff unless running in CI #5379

Conversation

andrewlock commented Apr 2, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Apr 2, 2024 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Apr 2, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Apr 2, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #5379

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5379

Faster 🎉 in #5379

Raw results

Raw results

datadog-ddstaging bot commented Apr 2, 2024 •

edited

Loading

andrewlock commented Apr 2, 2024 •

edited

Loading

andrewlock commented Apr 2, 2024 •

edited

Loading