Automatic instrumentation vs manual - when and how?

[dominikjeske]

Open telemetry is changing rapidly, and I'm confused about how to configure instrumentation of my apps. For a while we have option of Automatic instrumentation but I can't find anywhere when this scenario is better then manual. For example how it affects performance? Another question is how exactly is this mechanism working? I see docs about configuration but I can't find any on how automatic instrumentation actually works. How it magically reads metrics from process, what techniques it uses, is it working out of process etc. Is it applicable to all scenarios? For example documentations about IIS https://opentelemetry.io/docs/instrumentation/net/automatic/ says about .NET Framework but nothing about .NET Core?

[pellared]

For a while we have option of Automatic instrumentation but I can't find anywhere when this scenario is better then manual.

When you do not want to change the code, update the code when API changes or when you want to instrument another library. TL;DR; when you prefer magic instead of maintaining the instrumentation manually.

I do not say that it is better. It is a different approach.

For example how it affects performance?

That is a though question. In short: it affects performance as any kind of instrumentation. The manual instrumentation should be a little faster, but everything depends on the environment. In usual business applications the performance overhead should be tolerable.

Another question is how exactly is this mechanism working?

See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/design.md

says about .NET Framework but nothing about .NET Core?

It says about .NET (.NET Core was renamed to .NET). None .NET Core version is currently supported by Microsoft.

[pjanotti]

@dominikjeske let me add my take on some of @pellared answer.

Auto-Instrumentation for .NET exists for a few reasons:

• Some users that want to use OTel don't want to, or can't, change their source code. The manual setup of the SDK is small, but, for several reasons (lack of resources, lack of access to source code, design decision given that O11y may be an orthogonal concern, etc) some teams are unwilling or unable to change their source code. This is especially true for users of APM/O11y based on agents that don't require any changes to their application code.
• While .NET provides an excellent support for OTel (i.e.: the core abstractions are part of the standard libraries) not all libraries are going to be instrumented with OTel on their sources. Support to bytecode instrumentation provides a way to instrument libraries that are not instrumented using OTel (or the corresponding standard abstractions). Hopefully in the future most of the libraries and users are using code that is already instrumented and the need for bytecode instrumentation gets diminished but that's not the case right now.

The main performance difference that one should expect from auto-instrumentation is at start-up time since the code bootstrapping the SDK and applying any configuration needs to yet injected into the application. After that other than a bug that somehow only affects auto-instrumentation you should expect the same performance.

[experimenti]

Hi Robert, Played with both extensively, and yes, it's in flux. The auto-instrumentation relies (mostly) on env variables and a script to inject the libraries. It gets messy quickly if you are in moderately complex ci/cd context (kubernetes or compose) and that's where we got off the path. The manual instrumentation and extensions (EF, Asp.net core, grpc, etc.) are working well, but there's some artistry to getting the config/service injection right. If you're rolling multiple projects into one container, you'll likely fight dup. Instrumentation. Another downside here is that if you get really granular with Activities/Spans, you get some crowded/not-so-pretty code to trap the traces. Likely an attribute-based solution in the future. Let me know if I can provide more info/examples. Make no mistake though, in complex systems, the span contexts alone make the effort worthwhile. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Pająk ***@***.***> Sent: Saturday, November 4, 2023 12:41:46 AM To: open-telemetry/opentelemetry-dotnet-instrumentation ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [open-telemetry/opentelemetry-dotnet-instrumentation] Automatic instrumentation vs manual - when and how? (Discussion #3068) For a while we have option of Automatic instrumentation but I can't find anywhere when this scenario is better then manual. When you do not want to change the code, update the code when API changes or when you want to instrument another library. TL;DR; when you prefer magic instead of maintaining the instrumentation manually. Another question is how exactly is this mechanism working? See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/design.md says about .NET Framework but nothing about .NET Core? It says about .NET (.NET Core was renamed to .NET). None .NET Core version is currently supported by Microsoft. — Reply to this email directly, view it on GitHub<#3068 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEYBK7TM7FLDUDY64XMDQDYCUUFVAVCNFSM6AAAAAA64YXWJSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TINRZGY2TE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pjanotti]

@experimenti depending on your ci/cd context you may want to try Using the OpenTelemetry.AutoInstrumentation NuGet packages - not applicable in all cases, but, suitable to some.

We strive to have the env vars used for customizations, ie.: we expect that most users don't need to change them. Are there any defaults that caused you trouble and you had to change them?

[dominikjeske]

@pellared thanks for your answer. I had to be blind not seeing docs about design. Now after reading I have more questions - I will be graitfoul for some answers ;)

I'm Aware that manual and automatic are different approach to sam think but the question is - apart of different configuration injection do we have other differences

1. When we are using manual approach - is there any byte code instrumentation?
2. About side by side CLR Profiles - can I use automatic instrumentation on same machine with Dynatrace agent?
3. When we are talking with side by side - we had issues with our code using EventSource to generate custom metrics. When other app like PerfView was changing EventSource filter (it is global for app) we had performance issues of handling all those events in app code. Do you know such problems with OpenTelemetry auto or manual approach?
4. You mention .NET is now one platform. I'm aware of this but my question was about https://opentelemetry.io/docs/instrumentation/net/automatic/ -> "The following instructions apply to .NET Framework applications." so docs are saying about .NET Framework not .NET. But here we have .NET Core https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/iis-instrumentation.md so it is little confusing. I don't know what this warning is about when apparently, we can use this in .NET/.Net Core?
5. When auto instrumentation is sending data to Collector how it is done? Is this same as manual so application code is sending data to collector or auto instrumentation is doing the job so

[pjanotti]

1. When we are using manual approach - is there any byte code instrumentation?

No, there isn't.

2. About side by side CLR Profiles - can I use automatic instrumentation on same machine with Dynatrace agent?

No, only a single CLR Profiler can be used for .NET/.NET Framework application. There are some projects that allow multiplexing CLR profilers into a single project but it is hard to guarantee that they won't interfere with each other. If you Dynatrace agent deploys a CLR Profiler (and I think it does) you can't have the bytecode instrumentation from this project. However, you can still use the StartupHook to automatically add OTel instrumentations to your application. Once more: in this case none of the instrumentations that depend on bytecode instrumentation will work but things like ASP.NET Core, HTTP calls, etc will be automatically instrumented - the StartupHook is not available for .NET Framework applications.

3. ... Do you know such problems with OpenTelemetry auto or manual approach?

I'm not aware of such problems, but, by the description these will be the same with manual or auto-instrumentation: the auto-instrumentation project aims to use the same components as much as possible as the manual instrumentation. So in general, if it is not related to a bytecode instrumentation the behavior should be the same as manual instrumentation, you can see which instrumentations use bytecode or not at Instrumentations.

4. You mention .NET is now one platform. I'm aware of this but my question was about https://opentelemetry.io/docs/instrumentation/net/automatic/ -> "The following instructions apply to .NET Framework applications." so docs are saying about .NET Framework not .NET. But here we have .NET Core https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/iis-instrumentation.md so it is little confusing. I don't know what this warning is about when apparently, we can use this in .NET/.Net Core?

There are small differences in configuration and supported instrumentations between .NET and .NET Framework. Instrumentations list the differences for instrumentations. The doc is trying to make clear the cases that the config is different between .NET and .NET Framework. Please, let's know the exact parts that are giving a different interpretation than that we will be happy to look at those. cc @theletterf

5. When auto instrumentation is sending data to Collector how it is done? Is this same as manual so application code is sending data to collector or auto instrumentation is doing the job so

The auto-instrumentation hooks up the SDK for the app, but, it is using the same components as the manual instrumentation, so the same SDK and exporter. So in the end it is the same code.

[dominikjeske]

@pjanotti Thanks ;)