Support generating PRs for rollup-merged perf-testing #701
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds two new commands to rust-timer, both of which support being listed multiple times per comment, and can be made anywhere:
@rust-timer make-pr-for <sha>
@rust-timer update-branch-for <sha>
Both commands should be passed a rollup merge SHA, e.g.,
https://github.com/rust-lang/rust/commit/105cd4955425de2613ac2ae6c2d2d1baf17ebd2b
(with or without a full https URL prefix, thoughrust-lang/rust@SHA
is not supported).We create a pull request on rust-lang/rust and immediately invoke try on it to gather perf statistics. The pull request is created such that it contains two commits:
This should have the effect of mostly faithfully representing what it would look like to merge the PR into that old master by itself (rather than in the rollup), though it isn't absolutely perfect -- there's a number of things that could go wrong to make this infeasible.
One of the easiest ways for this to be problematic is if there have been CI-fixing changes landing in master since the creation of the rollup. Currently, those changes would likely mean that the pull request we've created is essentially useless, as it'll lack those changes and that means it cannot be easily run on its own. In such a scenario though nothing should stop the human operator from manually pushing some commits to the created PR.
An alternative to this is to generate a revert of the merge of just the commit passed onto current master, and benchmark that. The results would then be considered an "inverse" (i.e., good means the original PR was bad). This is much less prone to problems caused by external changes but is a less faithful and is somewhat harder to interpret the results of.
The second is basically solely for the edge case of something landing in master in the (short) amount of time the first one takes to work; this should be quite rare in practice. Eventually we may automatically detect this case, but the current PR doesn't do so as this is relatively hard.