release-22.2: sql: reduce the overhead of EXPLAIN ANALYZE

[yuzefovich]

Backport 1/1 commits from #91117.

/cc @cockroachdb/release

---

In order to propagate the execution stats across the distributed query plan we use the tracing infrastructure, where each stats object is added as "structured metadata" to the trace. Thus, whenever we're collecting the exec stats for a statement, we must enable tracing. Previously, in many cases we would enable it at the highest verbosity level which has non-trivial overhead. In some cases this was an overkill (e.g. in EXPLAIN ANALYZE we don't really care about the trace containing all of the gory details - we won't expose it anyway), so this is now fixed by using the less verbose "structured" verbosity level. As a concrete example of the difference: for a stmt that without EXPLAIN ANALYZE takes around 190ms, with EXPLAIN ANALYZE it would previously run for about 1.8s and now it takes around 210ms.

This required some minor changes to the row-by-row outbox and router
setups to collect thats even if the recording is not verbose.

Addresses: #90739.

Epic: None

Release note (performance improvement): The overhead of running EXPLAIN ANALYZE and EXPLAIN ANALYZE (DISTSQL) has been significantly reduced. The overhead of EXPLAIN ANALYZE (DEBUG) didn't change.

Release justification: performance fix.

[cockroach-teamcity]

This change is 

[michae2]

Reviewed 5 of 5 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DrewKimball)

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[DrewKimball]

Reviewable status: complete! 2 of 0 LGTMs obtained (waiting on @yuzefovich)

[yuzefovich]

Hm, we got some failures here because some of the execution stats are getting dropped from the trace. On master things work well due to recent @andreimatei work. In particular, once I revert 2d29175 on master I get a failure in explain_analyze_plans logic test.

Andrei, do you think it's feasible that we'll backport #89785 (which I assume will also pull in #88414) to 22.2? Those changes on their own would help with getting better tracing on 22.2 and would also unblock backporting the current change of lowering the overhead of EXPLAIN ANALYZE.

[yuzefovich]

@andreimatei do you have thoughts on #91208 (comment)?

[andreimatei]

I'm reticent to backport those patches; they're a bit big and I'm not all that confident that they don't have unintended consequences.
I would also question why this patch here is being backported - was there a regression or otherwise is there good reason for it to be?
If need be, I'd rather we increase the per-span structured tracing limit in 22.2 and see if that unblocks you.

[yuzefovich]

It's not a regression, but I'd argue that the fact that EXPLAIN ANALYZE slows the execution down by a factor of 10 in some cases when comparing to the vanilla query is a bug - the effect of enabling verbose tracing will skew the execution time of the operations that log a lot of stuff when comparing to less verbose operations which might deem the "profile" not very representative. So my justification for the backport is that it's a fix to a long-standing bug.

I'll play around with the constant limits and see whether at least simple tests in CI pass.

[yuzefovich]

I don't think I'll get to spending more time on this.