traces_sample_rate default is set to 1.0 - risking exhaustion of Sentry performance transactions

[dparrella]

Environment

What version are you running?
24.8.0 24.1.0

Steps to Reproduce

Deploy latest release with sentry enabled and a DSN supplied to the environment.

Expected Result

No performance transactions. traces_sample_rate should be set to 0.0, or at least configurable through the environment.

Actual Result

All of our Sentry transactions for the month (100K) were consumed in a matter of days.

This will be a serious problem for any Sentry customer that deploys this service.

[loewenheim]

Hi, thank you for the report. I'm afraid I don't quite understand the issue; we don't set traces_sample_rate in Symbolicator. Instead, we use traces_sampler to sample 5% of transactions (see here), is this what you would like to be configurable?

[dparrella]

Hi, @loewenheim. Thank you for getting to this so quickly. We misunderstood how the configuration was applied, but yes what you suggested makes sense. In our case, the service handles a large volume of events, so that explains why even a 5% sample rate would have generated so many transactions.

The main issue though is that before, we were not seeing any transactions. Then we started seeing a huge spike in transactions because this configuration was presumably changed. However, the most recent changes to this file were 7 months ago so I can't explain why we didn't see this problem sooner.

Should traces be disabled by default to avoid this problem for unsuspecting Sentry customers? We will also be happy with configuration options or environment variable support.

[loewenheim]

Are you running your own Symbolicator?

[dparrella]

Are you running your own Symbolicator?

Yes we are. Maybe we are unique in doing that :). But we have been running it for years at this point.

Also I'm sorry but I was mistaken on the version we are running - it's actually 24.1.0. Updated the issue description to reflect that.

[loewenheim]

Are you running your own Symbolicator?

Yes we are. Maybe we are unique in doing that :). But we have been running it for years at this point.

Right, I wanted to make sure—it is fairly rare.

We could make the tracing sample rate configurable so that you can turn it off, would that work for you?

[dparrella]

Are you running your own Symbolicator?

Yes we are. Maybe we are unique in doing that :). But we have been running it for years at this point.

Right, I wanted to make sure—it is fairly rare.

We could make the tracing sample rate configurable so that you can turn it off, would that work for you?

Yes making it configurable would be great. Thank you.

[tobias-wilfert]

Hi, this has been addressed by #1517 the latest version of Symbolicator will release in a couple of days.