Go back to using curl to download snapshots

[alexcrichton]

We've had too many problems using urllib to make it worth it.

Closes #1346
Closes #1088
cc #1319 (using a proxy through curl)

[rust-highfive]

r? @huonw

(rust_highfive has picked a reviewer for you, use r? to override)

[huonw]

Why this change?

[alexcrichton]

@huonw ah good catch, updated to keep the relevant changes of aceba88.

[huonw]

@bors r+ 8c05

[bors]

⌛ Testing commit 8c0525c with merge caa5b60...

[bors]

⌛ Testing commit 8c0525c with merge adafb16...

[bors]

💔 Test failed - cargo-win-64

[alexcrichton]

@bors: retry force

[bors]

⚡ Previous build results are reusable. Rebuilding only cargo-win-64...

[bors]

☀️ Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-32, cargo-win-64

[retep998]

This change back to curl has broken the downloading of snapshots for me on Windows using Msys2.

$ make
/mingw64/bin/python src/etc/dl-snapshot.py x86_64-pc-windows-gnu
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
Traceback (most recent call last):
  File "src/etc/dl-snapshot.py", line 75, in <module>
    raise Exception("failed to fetch url")
Exception: failed to fetch url
Makefile:90: recipe for target 'target/snapshot/bin/cargo.exe' failed
make: *** [target/snapshot/bin/cargo.exe] Error 1

[alexcrichton]

Hm, do you have curl install specially? I'm curious why it doesn't have access to the same certificates that other msys2 installations do?

[bombless]

Hmmm same issue as @retep998
I thought it was something wrong in my machine or something.

How did you fix it, @retep998 ?

FWIW, setting proxy on curl doesn't work for my MSYS2.
Not sure if it is about MSYS2 pacman tool/repository.

[retep998]

I just manually edited the scripts to pass the -k option to curl. The problem is that with msys2, the msys2 curl works fine, but the mingw64 curl is broken, and to build cargo you need cmake and cmake depends on the mingw64 curl. You could also install an external cmake so you don't need the mingw64 curl installed.

[bombless]

Thanks for the info, @retep998 !
Maybe we could use python urllib2 for Windows, and use curl for other systems.
I'll try make that happen by modifying this python script.