Spurious timeout in the asmjs builder #48826
Labels
A-spurious
Area: Spurious failures in builds (spuriously == for no apparent reason)
O-asmjs
Target: asm.js - http://asmjs.org/
O-emscripten
Target: 50% off wasm32-unknown-musl. the savings come out of stdio.h, but hey, you get SDL!
T-infra
Relevant to the infrastructure team, which will review and decide on the PR/issue.
Comments
alexcrichton
added
A-spurious
|
For reference, the rough timing breakdown according to the log in the recent successful builds are: (Unit = seconds)
We see that finishing the We could compare with those of android...x86_64...wasm32...and we see that the run time of the |
|
FWIW pre-LLD the wasm32-unknown run-pass test suite took 2700s, wheras after LLD it took 850s, a transition from 45m to ~15m. That's I think definitely confirming that Emscripten is so slow because of how it's linker works, it's just LLVM's LTO linker. Once my machine frees up from timing other things I'll run some tests to see how much optimizing tests costs us here. |
|
Disabling optimizations in tests unfortunately broke a bunch of tests and shaved 5 minutes off the total build time locally. I'm not sure if that'll actually translate to wins upstream... |
5 tasks
bors
added a commit
that referenced
this issue
Mar 19, 2018
Try to reduce amount of time on the asmjs builder This PR has two commits for two separate strategies: * First it disables optimizations for all tests, hopefully saving time by not optimizing the test code. This caused a number of run-pass tests to fail which are switched to being ignored here. * Next it disables a number of test suites which aren't asm.js specific and already run elsewhere cc #48826
|
Looks like with #49095 our builds have stabilized at from 2.5 hours to 2 hours on the asmjs builder. That should hopefully buy us at least a few months! |
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Mar 26, 2018
This commit disables building documentation on cross-compiled compilers, for example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out of these documentation artifacts and they often take 10-15 minutes total to build as it requires building rustdoc/rustbook and then also generating all the documentation, especially for the reference and the book itself. In an effort to cut down on the amount of work that we're doing on dist CI builders in light of recent timeouts this was some relatively low hanging fruit to cut which in theory won't have much impact on the ecosystem in the hopes that the documentation isn't used too heavily anyway. While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly a half-hour impact on the bots. In that sense I'm hoping that we can land this and test out what happens on CI to see how it affects timing. Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to generate documentation.
kennytm
added a commit
to kennytm/rust
that referenced
this issue
Mar 26, 2018
rustbuild: Disable docs on cross-compiled builds This commit disables building documentation on cross-compiled compilers, for example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out of these documentation artifacts and they often take 10-15 minutes total to build as it requires building rustdoc/rustbook and then also generating all the documentation, especially for the reference and the book itself. In an effort to cut down on the amount of work that we're doing on dist CI builders in light of recent timeouts this was some relatively low hanging fruit to cut which in theory won't have much impact on the ecosystem in the hopes that the documentation isn't used too heavily anyway. While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly a half-hour impact on the bots. In that sense I'm hoping that we can land this and test out what happens on CI to see how it affects timing. Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to generate documentation.
TimNN
added a commit
to TimNN/rust
that referenced
this issue
Mar 26, 2018
rustbuild: Disable docs on cross-compiled builds This commit disables building documentation on cross-compiled compilers, for example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out of these documentation artifacts and they often take 10-15 minutes total to build as it requires building rustdoc/rustbook and then also generating all the documentation, especially for the reference and the book itself. In an effort to cut down on the amount of work that we're doing on dist CI builders in light of recent timeouts this was some relatively low hanging fruit to cut which in theory won't have much impact on the ecosystem in the hopes that the documentation isn't used too heavily anyway. While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly a half-hour impact on the bots. In that sense I'm hoping that we can land this and test out what happens on CI to see how it affects timing. Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to generate documentation.
workingjubilee
added
the
O-emscripten
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The asm.js builder has been creeping up in time it takes to run ever since it was originally added, and clocking in at around 2.5 hrs now this builder is ripe for timing out on occasion. The suspected cause here is the fact that the run-pass test suite takes >1hr to run and the suspected cause there is that we're effectively compiling every test with LTO which can take quite some time.
This issue is intended to track possible solutions to bring the builder back under control. Today in a discussion a few possible avenues arose:
cc @kennytm
The text was updated successfully, but these errors were encountered: