compaction: async deletion of files can result in many goroutines

[nicktrav]

When using the deletion pacer, we kick off the deletion in a goroutine, here. This can result in many goroutines being spawned, only to be put to sleep by the pacer, here.

Here's an example summary I saw, which showed 500+ goroutines all sleeping, waiting their turn to delete the files from their respective compactions. This accounted for ~5% of the total goroutine count of this process:

When using a deletion pacer, rather than spawning a goroutine per compaction, we could consider having a single goroutine that is responsible for deletion of files. When a compaction completes, it puts the files that need to be deleted in a queue (guarded by some mutex). Then a second goroutine could pull from this queue in order to satisfy the deleting pacing constraints.

Note that when running without a pacer, this problem isn't apparent, as the files are deleted synchronously as part of the compaction.

[jbowens]

Was the high goroutine count present with and without encryption-at-rest? cockroachdb/cockroach#98051 has the potential to slow Remove FS calls and exacerbate the issue.

[nicktrav]

Was the high goroutine count present with and without encryption-at-rest?

Correct. I saw it in both. The test run using a master binary doesn't seem to have this issue. It may take some time to manifest though.

[bananabrick]

When a compaction completes, it puts the files that need to be deleted in a queue (guarded by some mutex). Then a >second goroutine could pull from this queue in order to satisfy the deleting pacing constraints.

We already have this queue, versionSet.obsoleteTables. We're not using it in the way which is proposed in this issue, though.

[jbowens]

This was resolved in #2641.