x.py fails (timeout) to download components

[L117]

I just cloned this rust repo and attemped to build it by calling

./x.py build

It initialized submodules, started downloading rust-std and failed repeatedly:

downloading https://static.rust-lang.org/dist/2019-04-11/rust-std-beta-x86_64-unknown-linux-gnu.tar.gz
##################################################                                                                       42.3%
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
Throwing away 33504800 bytes
##############################################                                                                           38.9%
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
Throwing away 30821397 bytes
##############################################                                                                           39.2%
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
Throwing away 30941837 bytes
###################################################                                                                      43.0%
curl: (28) Operation too slow. Less than 10 bytes/sec transferred the last 30 seconds

Waiting longer causes messages above to repeat.

It's not connection issue, because it always fails after ~30 seconds of downloading (Messages above repeat in ~2 minutes).

I also tried changing 30 to 9999 here:

rust/src/bootstrap/bootstrap.py

Line 82 in c5295ac

"-y", "30", "-Y", "10", # timeout if speed is < 10 bytes/sec for > 30 seconds

What caused download to succeed in a few (1-2) minutes.

So I assume it's curl to blame.
curl version:

$ curl --version          
curl 7.65.0 (x86_64-pc-linux-gnu) libcurl/7.65.0 OpenSSL/1.1.1b zlib/1.2.11 libidn2/2.1.1
Release-Date: 2019-05-22
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL TLS-SRP UnixSockets

[AaronKutch]

I got x.py to test libcore once, but after I updated my local clone of rust, I started getting this repeatedly:

$ ./x.py test --stage 0 src/libcore
Updating only changed submodules
Submodules updated in 0.34 seconds
Exception calling "DownloadFile" with "2" argument(s): "The operation has timed out"
At line:1 char:82
+ ... pe]::Tls12; (New-Object System.Net.WebClient).DownloadFile('https://s ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException


spurious failure, trying again

[AaronKutch]

I am nuking my entire rust clone in case changes to the bootstrap are incompatible to previous artifacts produced by previous versions. Why does x.py need to download submodules like rls and rustfmt if I am only trying to test libcore? edit: apparently, this is expected #61667

[AaronKutch]

Nope, I am still getting the DownloadFile exception after refreshing everything, including updating MSYS2 to get curl to 7.65.1, trying the MSYS2 that comes along with git, and messing with bootstrap.py

[mati865]

@AaronKutch your issue is different and there is no curl involved (it downloads file through PowerShell). You can see this MSYS2 patch how to make it use curl instead: https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-rust/force-curl-rust.patch

[AaronKutch]

Thank you, it works now

[mati865]

@Mark-Simulacrum original issue (from comment 0) is not resolved. AaronKutch reported another issue that isn't solved either, there is only workaround available.

[Mark-Simulacrum]

We should probably fallback to trying curl if the powershell invocation fails. Reopening. I think a PR would work best here.

[AaronKutch]

For some reason, the powershell download is working again on both my MSYS2 install and from git bash. We still want that fallback though. I couldn't PR a fix though, because I don't know how the exception handling is working in that Python script.

[AaronKutch]

Actually, since I should learn Python anyway, should I PR a fix? Is it as simple as detecting an exception occurred and refactoring the if-else statement in a way that it would also run curl?

[Mark-Simulacrum]

That's what I would suggest -- we can make it pretty simple and if anything goes wrong with the powershell call then fallback to curl

[jyn514]

It might also make sense to try curl before using powershell, as suggested by @ChrisDenton in #95170 (comment).

@rustbot label: +E-easy