Fix Windows test flakiness

[brson]

The flakiness of the Windows test suite is out of hand. Most PR's fail multiple times. It feels like it's getting worse. Lots of weird I/O failures.

I don't know if I am comfortable making another release in this state.

[Diggsey]

It's probably rust-lang/rust#29497 in various forms. 😞

[Diggsey]

Let's collect examples of spurious failures here. I'm assuming this is one:
https://ci.appveyor.com/project/brson/rustup-rs/build/1.0.846/job/d39vkpcr22q2j0ke

I've been trying to reproduce this locally, but the test suite is stubbornly passing every time 😛 I've tried 32-bit and 64-bit, maybe it's specifically -gnu builds? If we have some more examples it may be possible to spot a pattern.

[Diggsey]

Hm, I left the test suite running on loop for several hours in 3 different configurations, and didn't get a single failure...

[brson]

FWIW, someone is working on a remove_dir_all impl here: rust-lang-deprecated/tempdir#15 (comment)

[brson]

cc @Aaronepower once you've got that remove_dir_all crate you might plug it into rustup too.

[Diggsey]

Here's a table of recent failures:

Configuration	Failing Test	Error
64-bit GNU	uninstall_deletes_cargo_home	Access is denied during uninstall.
64-bit GNU	uninstall_doesnt_leave_gc_file	Panic: tests\cli-self-upd.rs:294
64-bit MSVC	uninstall_doesnt_leave_gc_file	Access is denied during uninstall.
64-bit GNU	uninstall_doesnt_leave_gc_file	Panic: tests\cli-self-upd.rs:294
32-bit MSVC	uninstall_deletes_bins	Access is denied during uninstall.
64-bit MSVC	uninstall_deletes_bins	Access is denied during uninstall.
32-bit GNU	remove_default_toolchain_err_handling	Access is denied during uninstall.
64-bit MSVC	windows_handle_empty_path_registry_key	Access is denied during uninstall.
64-bit GNU	multirust_upgrade_works_with_proxy	unable to hard link fallback exe (already exists)

It's clearly not related to the configuration. All of them seem related to file or directory removal. However, the last one would not be fixed by an improved remove_dir_all. It looks like it might be caused by tests running in parallel?

[Diggsey]

Update:

• It's not caused by tests running in parallel
• It's not caused by any virus scanners, the indexing service, or shadow copy
• Running handle.exe to find handles to files being deleted shows no results over many, many test runs

[Diggsey]

• It doesn't seem to be caused by any kind of network file system weirdness

[Diggsey]

@brson Updating to the newer appveyor image seems to have reduced the failure rate quite a bit. Over 40 builds there was only 1 failure, which would be fixed by the remove_dir_all changes, so in combination, I think we should be back down to an acceptable failure rate...

I'll work on getting that PR into a state we can merge (atm it installs more than is strictly required, and increases the build time quite a bit. I might see if it can cache the msys install across builds.) and then I think we can close this.

[XAMPPRocky]

@brson The remove_dir_all crate is now available, sorry I didn't finish it before you wrote that module. 😅

[mati865]

Another victim.
Toolchain: 32-bit MSVC
Test: updater_is_deleted_after_running_rustup

error: could not remove 'setup' file: 'C:\Users\appveyor\AppData\Local\Temp\1\rustup-cargo.2PgHSjcklvu2\ch\bin/rustup-init.exe'
info: caused by: Access is denied. (os error 5)

[rbtcollins]

Just noting that we do still see the occasional unexplained windows failure, but rare vs common. I'm going to close this as long in the tooth and suggest we open specific tickets when we feel we have a pattern emerging again.