[internal] Add a benchmark for Snapshot capture #12569
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ci skip-build-wheels]
[ci skip-build-wheels]
stuhood
commented
Aug 13, 2021
| (20, 10_000_000, true, 10), | ||
| (1, 200_000_000, true, 10), | ||
| ] { | ||
| let (count, size, _immutable, captures) = params; |
There was a problem hiding this comment.
The _immutable flag becomes relevant in #12563.
There was a problem hiding this comment.
I'm not sure I understand the point of captures > 1? If we expect them to be no-ops, why are we benchmarking them?
There was a problem hiding this comment.
Anytime we Snapshot the filesystem, if we discover that we've already captured a file in the LMDB store, we'll skip re-storing it. But that still involves hashing it to determine that we don't need to store it again. So it's a noop in terms of storage, but not in terms of hashing: LMDB::put will skip storing the value, and in #12563, LMDB::reserve will fail with an "exists" error and prevent the second pass over the data.
The captures ratio here is supposed to align with usecases (I'll add a comment in #12563):
(100, 100, false, 100)is supposed to represent Snapshotting the workspace: lots of small hand written files that change relatively infrequently (100 captures is 1 write and 99 noops)(20, 10_000_000, true, 10)and(1, 200_000_000, true, 10)both represent capturing out of local process execution sandboxes: immutable data that is more unique, so more writes and fewer noops.
illicitonion
approved these changes
Aug 14, 2021
| (20, 10_000_000, true, 10), | ||
| (1, 200_000_000, true, 10), | ||
| ] { | ||
| let (count, size, _immutable, captures) = params; |
There was a problem hiding this comment.
I'm not sure I understand the point of captures > 1? If we expect them to be no-ops, why are we benchmarking them?
stuhood
added a commit
that referenced
this pull request
Aug 14, 2021
A private repository experienced OOM issues in CI with a large number of tests. Allocation profiling highlighted that a large batch of `32MB`/`64MB` allocations was occurring around the same time, all caused by the fact that `PosixFS::read_file` slurps an entire file into memory as `FileContent`. Because local process execution uses `PosixFS::read_file` to capture sandbox outputs, a bunch of processes finishing at once could lead to memory usage spikes.
To fix that, this change removes `PosixFS::read_file` (which was only ever used to digest files), and replaces it with `{Store,local::ByteStore,ShardedLmdb}::store*` methods which make two-ish passes over a `Read` instance in order to digest and capture it.
The methods take an `immutable_data` parameter so that callers can indicate that they know that data will not change, which allows for avoiding hashing it on the second pass. Captures from process sandboxes are treated as immutable, while memoized captures from the workspace are not.
----
#12569 adds relevant benchmarks. Profiling allocation during those benchmarks shows peak memory usage of `945MB` on `main`, and `362MB` on the branch. Additionally, the benchmarks all run faster (likely due to reduced allocation):
```
snapshot_capture/snapshot_capture((100, 100, false, 100))
time: [1.8909 s 1.9000 s 1.9109 s]
change: [-27.459% -25.841% -24.085%] (p = 0.00 < 0.05)
Performance has improved.
snapshot_capture/snapshot_capture((20, 10000000, true, 10))
time: [1.2539 s 1.2655 s 1.2782 s]
change: [-62.138% -61.318% -60.505%] (p = 0.00 < 0.05)
Performance has improved.
snapshot_capture/snapshot_capture((1, 200000000, true, 10))
time: [3.5281 s 3.5773 s 3.6299 s]
change: [-13.420% -11.985% -10.434%] (p = 0.00 < 0.05)
Performance has improved.
```
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor the test harness to allow for creating larger files (without allocating their full contents in memory), and add benchmarks for a series of file counts, sizes, and repetitions.
Preparation for #12563.
[ci skip-build-wheels]