Exclude an SSIS service from auto-tracing #5813

andrewlock · 2024-07-26T13:18:46Z

Summary of changes

Excludes some SQL Server Integration Services (SSIS) services from tracing

Reason for change

We have seen a case of injecting into MsDtsSrvr.exe causing it to break.

Implementation details

Exclude some of the processes defined here. There are more services we could avoid, but ReportingServicesService seems sufficiently general that we could accidentally not instrument something we're supposed to.

In contrast, I think these are all pretty safe:

MsDtsSrvr.exe"
sqlagent.exe"
sqlbrowser.exe"

Test coverage

Nope

datadog-ddstaging · 2024-07-26T13:38:27Z

Datadog Report

Branch report: andrew/exclude-more-processes
Commit report: cacf57e
Test service: dd-trace-dotnet

✅ 0 Failed, 350590 Passed, 2525 Skipped, 24h 32m 54.11s Total Time

andrewlock · 2024-07-26T13:49:02Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (75ms)  : 65, 84
     .   : milestone, 75,
    master - mean (72ms)  : 62, 82
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (1,062ms)  : 1042, 1082
     .   : milestone, 1062,
    master - mean (1,057ms)  : 1037, 1078
     .   : milestone, 1057,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (110ms)  : 106, 115
     .   : milestone, 110,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (754ms)  : 734, 775
     .   : milestone, 754,
    master - mean (758ms)  : 735, 781
     .   : milestone, 758,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (94ms)  : 90, 97
     .   : milestone, 94,
    master - mean (93ms)  : 89, 97
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (698ms)  : 678, 717
     .   : milestone, 698,
    master - mean (702ms)  : 680, 724
     .   : milestone, 702,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (194ms)  : 188, 200
     .   : milestone, 194,
    master - mean (192ms)  : 189, 196
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (1,169ms)  : 1125, 1213
     .   : milestone, 1169,
    master - mean (1,165ms)  : 1122, 1207
     .   : milestone, 1165,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (276ms)  : 272, 280
     .   : milestone, 276,
    master - mean (276ms)  : 271, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (933ms)  : 874, 991
     .   : milestone, 933,
    master - mean (915ms)  : 881, 949
     .   : milestone, 915,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5813) - mean (267ms)  : 260, 273
     .   : milestone, 267,
    master - mean (266ms)  : 262, 269
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (5813) - mean (929ms)  : 820, 1038
     .   : milestone, 929,
    master - mean (901ms)  : 880, 922
     .   : milestone, 901,

lucaspimentel

Wow, it's been a very long time since I've heard of SQL Server Reporting Services.

andrewlock · 2024-07-26T19:25:46Z

Benchmarks Report for tracer 🐌

Benchmarks for #5813 compared to master:

1 benchmarks are faster, with geometric mean 1.320
1 benchmarks are slower, with geometric mean 1.134
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.78μs	41.9ns	233ns	0.0115	0.00382	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	54.8ns	310ns	0.0195	0.00975	0	5.62 KB
master	`StartStopWithChild`	net472	16.1μs	56ns	217ns	1.02	0.3	0.0972	6.07 KB
#5813	`StartStopWithChild`	net6.0	7.8μs	43ns	276ns	0.0159	0.00796	0	5.43 KB
#5813	`StartStopWithChild`	netcoreapp3.1	9.74μs	49.6ns	243ns	0.014	0.00467	0	5.62 KB
#5813	`StartStopWithChild`	net472	16μs	36.1ns	130ns	1.01	0.301	0.0871	6.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	473μs	282ns	1.09μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	645μs	488ns	1.89μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	875ns	3.28μs	0.425	3.3 KB
#5813	`WriteAndFlushEnrichedTraces`	net6.0	473μs	253ns	948ns	0	2.7 KB
#5813	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	634μs	506ns	1.96μs	0	2.7 KB
#5813	`WriteAndFlushEnrichedTraces`	net472	852μs	701ns	2.62μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	201μs	1.17μs	10.7μs	0.195	18.45 KB
master	`SendRequest`	netcoreapp3.1	224μs	1.32μs	12.2μs	0.235	20.61 KB
master	`SendRequest`	net472	0.00348ns	0.00114ns	0.00426ns	0	0 b
#5813	`SendRequest`	net6.0	197μs	1.14μs	9.77μs	0.191	18.45 KB
#5813	`SendRequest`	netcoreapp3.1	221μs	1.27μs	10.2μs	0.206	20.61 KB
#5813	`SendRequest`	net472	0.00676ns	0.00168ns	0.0065ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	562μs	2.64μs	10.9μs	0.558	0	0	41.56 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	689μs	3.46μs	16.6μs	0.34	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	net472	849μs	3.26μs	12.6μs	8.39	2.52	0.419	53.27 KB
#5813	`WriteAndFlushEnrichedTraces`	net6.0	567μs	2.42μs	8.73μs	0.566	0	0	41.67 KB
#5813	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	701μs	3.47μs	15.1μs	0.331	0	0	41.76 KB
#5813	`WriteAndFlushEnrichedTraces`	net472	878μs	4.1μs	15.9μs	8.36	2.64	0.44	53.31 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.37μs	1.11ns	4.32ns	0.0144	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.66μs	1.14ns	4.26ns	0.0132	1.02 KB
master	`ExecuteNonQuery`	net472	2.04μs	2.59ns	10ns	0.156	987 B
#5813	`ExecuteNonQuery`	net6.0	1.29μs	1.14ns	4.41ns	0.0142	1.02 KB
#5813	`ExecuteNonQuery`	netcoreapp3.1	1.71μs	2.24ns	8.69ns	0.0136	1.02 KB
#5813	`ExecuteNonQuery`	net472	1.97μs	2.3ns	8.6ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.15μs	0.723ns	2.7ns	0.0136	0	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.59μs	0.58ns	2.17ns	0.0132	0	976 B
master	`CallElasticsearch`	net472	2.51μs	1.36ns	5.26ns	0.158	0	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	6.34ns	26.1ns	0.0134	0	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.14ns	4.43ns	0.014	0	1.02 KB
master	`CallElasticsearchAsync`	net472	2.7μs	2.1ns	7.85ns	0.166	0.00134	1.05 KB
#5813	`CallElasticsearch`	net6.0	1.27μs	0.717ns	2.58ns	0.0134	0	976 B
#5813	`CallElasticsearch`	netcoreapp3.1	1.53μs	1.44ns	5.39ns	0.0129	0	976 B
#5813	`CallElasticsearch`	net472	2.46μs	2.32ns	8.68ns	0.157	0	995 B
#5813	`CallElasticsearchAsync`	net6.0	1.31μs	0.768ns	2.66ns	0.0132	0	952 B
#5813	`CallElasticsearchAsync`	netcoreapp3.1	1.76μs	0.97ns	3.63ns	0.0142	0	1.02 KB
#5813	`CallElasticsearchAsync`	net472	2.62μs	6.52ns	25.2ns	0.166	0.0013	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5813

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.134	1,189.21	1,348.08

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.19μs	0.492ns	1.91ns	0.0132	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.689ns	2.58ns	0.0127	952 B
master	`ExecuteAsync`	net472	1.74μs	0.916ns	3.43ns	0.145	915 B
#5813	`ExecuteAsync`	net6.0	1.35μs	0.56ns	2.1ns	0.0135	952 B
#5813	`ExecuteAsync`	netcoreapp3.1	1.63μs	0.917ns	3.31ns	0.013	952 B
#5813	`ExecuteAsync`	net472	1.71μs	0.833ns	3.23ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.29μs	2.3ns	8.3ns	0.0301	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.07μs	2.09ns	8.08ns	0.0362	2.76 KB
master	`SendAsync`	net472	7.69μs	2.92ns	11.3ns	0.496	3.15 KB
#5813	`SendAsync`	net6.0	4.08μs	2.24ns	8.38ns	0.0307	2.22 KB
#5813	`SendAsync`	netcoreapp3.1	5.11μs	1.59ns	6.16ns	0.0359	2.76 KB
#5813	`SendAsync`	net472	7.77μs	1.71ns	6.39ns	0.496	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.572ns	2.14ns	0.0227	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.06μs	0.551ns	1.99ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.68μs	4.24ns	16.4ns	0.249	1.57 KB
#5813	`EnrichedLog`	net6.0	1.56μs	1.16ns	4.35ns	0.0229	1.64 KB
#5813	`EnrichedLog`	netcoreapp3.1	2.23μs	1.45ns	5.6ns	0.0222	1.64 KB
#5813	`EnrichedLog`	net472	2.57μs	0.939ns	3.64ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	154ns	598ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	139ns	540ns	0	0	4.28 KB
master	`EnrichedLog`	net472	147μs	97.1ns	376ns	0.657	0.219	4.46 KB
#5813	`EnrichedLog`	net6.0	116μs	245ns	949ns	0	0	4.28 KB
#5813	`EnrichedLog`	netcoreapp3.1	120μs	105ns	407ns	0	0	4.28 KB
#5813	`EnrichedLog`	net472	149μs	247ns	957ns	0.672	0.224	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.06μs	1.46ns	5.48ns	0.0309	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.21μs	1.39ns	5.37ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.76μs	1.1ns	4.25ns	0.321	2.02 KB
#5813	`EnrichedLog`	net6.0	2.99μs	1.01ns	3.91ns	0.0299	2.2 KB
#5813	`EnrichedLog`	netcoreapp3.1	4.11μs	1.91ns	7.16ns	0.0288	2.2 KB
#5813	`EnrichedLog`	net472	4.77μs	0.93ns	3.6ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.33μs	1.12ns	4.34ns	0.0158	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.71μs	1.16ns	4.51ns	0.0156	0	1.14 KB
master	`SendReceive`	net472	2.11μs	0.868ns	3.36ns	0.183	0.00105	1.16 KB
#5813	`SendReceive`	net6.0	1.24μs	0.647ns	2.42ns	0.016	0	1.14 KB
#5813	`SendReceive`	netcoreapp3.1	1.71μs	0.63ns	2.27ns	0.0153	0	1.14 KB
#5813	`SendReceive`	net472	2.1μs	1.84ns	7.11ns	0.183	0.00106	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.984ns	3.81ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	1.85ns	6.68ns	0.0222	1.65 KB
master	`EnrichedLog`	net472	4.45μs	4.31ns	16.7ns	0.323	2.04 KB
#5813	`EnrichedLog`	net6.0	2.79μs	0.862ns	3.34ns	0.0224	1.6 KB
#5813	`EnrichedLog`	netcoreapp3.1	3.91μs	4.13ns	14.9ns	0.0215	1.65 KB
#5813	`EnrichedLog`	net472	4.42μs	2.39ns	8.95ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5813

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.320	634.19	480.53

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	403ns	0.111ns	0.432ns	0.00811	576 B
master	`StartFinishSpan`	netcoreapp3.1	588ns	0.482ns	1.87ns	0.00768	576 B
master	`StartFinishSpan`	net472	582ns	0.169ns	0.655ns	0.0918	578 B
master	`StartFinishScope`	net6.0	634ns	0.18ns	0.674ns	0.00991	696 B
master	`StartFinishScope`	netcoreapp3.1	760ns	0.45ns	1.74ns	0.00945	696 B
master	`StartFinishScope`	net472	909ns	1.73ns	6.47ns	0.104	658 B
#5813	`StartFinishSpan`	net6.0	413ns	0.261ns	0.942ns	0.00801	576 B
#5813	`StartFinishSpan`	netcoreapp3.1	548ns	0.143ns	0.534ns	0.00775	576 B
#5813	`StartFinishSpan`	net472	639ns	0.505ns	1.96ns	0.0915	578 B
#5813	`StartFinishScope`	net6.0	481ns	0.251ns	0.972ns	0.0099	696 B
#5813	`StartFinishScope`	netcoreapp3.1	692ns	0.75ns	2.9ns	0.00918	696 B
#5813	`StartFinishScope`	net472	849ns	0.423ns	1.52ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	599ns	0.162ns	0.605ns	0.00982	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	961ns	1.15ns	4.47ns	0.00951	696 B
master	`RunOnMethodBegin`	net472	1.14μs	0.303ns	1.17ns	0.104	658 B
#5813	`RunOnMethodBegin`	net6.0	651ns	0.231ns	0.893ns	0.00987	696 B
#5813	`RunOnMethodBegin`	netcoreapp3.1	949ns	0.419ns	1.57ns	0.00902	696 B
#5813	`RunOnMethodBegin`	net472	1.17μs	1.04ns	4.04ns	0.104	658 B

## Summary of changes Excludes some [SSIS](https://learn.microsoft.com/en-us/sql/integration-services/sql-server-integration-services?view=sql-server-ver16) services from tracing ## Reason for change We have seen a case of injecting into MsDtsSrvr.exe causing it to break. ## Implementation details Exclude some of the [processes defined here](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-windows-service-accounts-and-permissions?view=sql-server-ver16). There are more services we _could_ avoid, but `ReportingServicesService` seems sufficiently general that we could accidentally _not_ instrument something we're supposed to. In contrast, I think these are all pretty safe: - `MsDtsSrvr.exe"` - `sqlagent.exe"` - `sqlbrowser.exe"` ## Test coverage Nope

Exclude an SSIS service from auto-tracing

cacf57e

andrewlock added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:native-library Automatic instrumentation native C++ code (Datadog.Trace.ClrProfiler.Native) type:reliability labels Jul 26, 2024

andrewlock requested review from a team as code owners July 26, 2024 13:18

kevingosse approved these changes Jul 26, 2024

View reviewed changes

lucaspimentel approved these changes Jul 26, 2024

View reviewed changes

lucaspimentel reviewed Jul 26, 2024

View reviewed changes

andrewlock merged commit b1dfcfe into master Jul 30, 2024
78 of 80 checks passed

andrewlock deleted the andrew/exclude-more-processes branch July 30, 2024 10:23

github-actions bot added this to the vNext-v3 milestone Jul 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exclude an SSIS service from auto-tracing #5813

Exclude an SSIS service from auto-tracing #5813

andrewlock commented Jul 26, 2024 •

edited by lucaspimentel

Loading

datadog-ddstaging bot commented Jul 26, 2024 •

edited

Loading

andrewlock commented Jul 26, 2024

lucaspimentel left a comment

andrewlock commented Jul 26, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5813

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5813

Raw results

Raw results

Exclude an SSIS service from auto-tracing #5813

Exclude an SSIS service from auto-tracing #5813

Conversation

andrewlock commented Jul 26, 2024 • edited by lucaspimentel Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jul 26, 2024 • edited Loading

Datadog Report

andrewlock commented Jul 26, 2024

Execution-Time Benchmarks Report ⏱️

lucaspimentel left a comment

Choose a reason for hiding this comment

andrewlock commented Jul 26, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5813

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5813

Raw results

Raw results

andrewlock commented Jul 26, 2024 •

edited by lucaspimentel

Loading

datadog-ddstaging bot commented Jul 26, 2024 •

edited

Loading