Auto Instrumentation - Monitor multiple processes

[dineshg13]

Is your feature request related to a problem? Please describe.
Our auto instrumentation system is currently limited to monitoring a single process. This approach poses an issue when dealing with a large cluster containing 1000 pods, for example, as it would necessitate deploying an instrumentation service as a sidecar for each pod. We can reduce our resource utilization by allowing a single instance of the auto instrumentation service to supervise numerous services.

Describe the solution you'd like
We want the auto instrumentation service to be able to monitor multiple processes. To achieve this

• Make OTEL_GO_AUTO_TARGET_EXE optional
• Use https://github.com/google/gops to monitor for new processes on the machine.
• Instrument the new process
• Have an exclude list to ignore processes for auto instrumentation

This approach involves significant code refactoring. We need to approach this in multiple PRs. Some concerns from slack

• Ensure we have a single OTel exporter for all the processes vs. running one instance per process
• Watching for terminated processes and clean up the eBPF programs/maps for the terminated process
• How can we make env vars like OTEL_SERVICE_NAME work when dealing with multiple processes?

[dineshg13]

Existing code:

Proposed Changes:

• Add orchestrator that splits the responsibility

Plan for getting this across

1. Add PR for bare metal using existing code.
2. Add orchestrator API - with no functionality changes.
3. Making orchestrator Changes by adding functionality for monitoring GO process.
4. [Stretch] Introduce breaking changes - remove the need to define TARGET_EXE in the auto instrumentation process.

[RonFed]

Sounds like a good plan, I suggest designing the orchestrator by using the existing APIs we expose in instrumentation.go like NewInstrumentation, Instrumentation.Run and Instrumentation.Close without introducing additional breaking changes.
This will allow users to control instrumentation via 3 different methods:

• Env vars for a single process
• Exposed APIs in instrumentation.go for a single process
• Orchestrator for multiple processes (which uses the API)

[dianashevchenko]

@edeNFed @RonFed I hope you're doing well. I wanted to check in and ask if there are any plans to prioritize this issue in the near future. Your work is greatly appreciated, and any updates would be helpful 🙇