Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document overhead of Performance monitoring #7262

Closed
danielkhan opened this issue Jun 26, 2023 · 3 comments · Fixed by #7387
Closed

Document overhead of Performance monitoring #7262

danielkhan opened this issue Jun 26, 2023 · 3 comments · Fixed by #7387
Assignees

Comments

@danielkhan
Copy link
Contributor

danielkhan commented Jun 26, 2023

Core or SDK?

Platform/SDK

Which part? Which one?

SDKs

Description

We are frequently asked about performance overhead, and while it's hard to put a number to it, we should help our users to understand possible performance implications and what we are doing to mitigate them.

Customer Feedback (paraphrased)

“I don’t need benchmarking statistics. I want to know how the sdk works-under-the hood, and then I can analyze my app and decide if it’s something to be worried about, because only I know the intricacies and architecture of my app”

“I’m scared to let the sentry sdk run because I don’t know what it’s doing. For analogy, if your trusted auto technician said your vehicle needs 5 hours of maintenance work and problem fixing, you’d ask for a description of the required work, just so you’re aware. And then you’d agree (or not). I trust you Sentry but I’m still looking for the same awareness regarding your SDK for Performance Monitoring - what is it doing?”

Suggested Solution

  • Focus on talking about how the sdk is designed and works rather than referencing lab-like benchmarking experiments which are expensive to run.
  • Something similar to our existing documentation for perf impact of Profiling and Session Replay, see examples below.
  • Something that addresses the concern in the above 2 customer feedback quotes.
  • Consider including examples of harmful things the sdk is-not-doing, to assure it's okay to use.

Examples:

  • https://docs.sentry.io/product/profiling/performance-overhead mentions "1 to 5%" at the bottom but even if you remove this line, this documentation is still helpful.
  • https://docs.sentry.io/product/session-replay/performance-overhead makes 0 mention of a quantified performance impact like "3% CPU" and this is still a helpful page of documentation.
  • “trace generation overhead” by Google’s Dapper Tracing whitepaper sections 4.1, 4.2, 4.3
  • New Relic's browser monitoring and performance impact documentation. Notice its general speak about web apps + sdk's. You could remove the "overhead of less than 15ms per page." at the bottom and it is still helpful documentation.
  • UX Cam's documentation also does not give CPU/benchmarking statistics on this subject, for their Session Replay product.
  • Datadog's documentation does not mention CPU/benchmarking statistics for their Session Replay other than a network bandwidth aspect, "Datadog also reduces the load on a browser’s UI thread by delegating most of the CPU-intensive work (such as compression) to a dedicated web worker. "
  • Fullstory's documentation divides it into 1. script is small 2. script is edge-cached by CDN to reduce time 3. UI thread utilization 4. Upload bandwdith, and do we do the following at sentry, "FullStory uses Event Handlers that simply add all user interaction events to an in-memory queue to be processed later, rather than processing them in real time. " ?

Not The Solution / What Is Not Being Asked

Benchmarking experiments, scientific analysis, answers like, “3% of CPU”. The Session Replay docs don’t mention this and the Profiling docs do at the bottom but you could remove that line and it’d still be a useful page of docs.

cc @lizokm

@danielkhan danielkhan changed the title Performance overhead documentation for Performance Monitoring Document overhead of Performance monitoring Jun 26, 2023
@smeubank
Copy link
Member

@sl0thentr0py FYI what we talked about during planning

we can discuss in more detail this week, if an outcome of your research could be summary that could be used for this

@thinkocapo
Copy link
Contributor

the updated Performance Overhead documentation is:
https://docs.sentry.io/product/performance/performance-overhead/
image

@thinkocapo
Copy link
Contributor

not asking for numbers but here's an instance where we did document numbers for sentry-dotnet sdk. looks like it's regarding span generation and memory.
getsentry/sentry-dotnet#2633
getsentry/sentry-dotnet#2632
getsentry/sentry-dotnet#2621
getsentry/sentry-dotnet#2619

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants