Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fetchurl: add user agent #17757

Merged
merged 5 commits into from
Jun 11, 2018
Merged

fetchurl: add user agent #17757

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0cb623c
fetchurl: add user agent
copumpkin Aug 15, 2016
16ed09a
Merge branch 'master' into fetchurl-user-agent
lukateras May 18, 2018
a26d9e3
Readd --retry 3
lukateras Jun 11, 2018
5a232b0
Leverage lib.release
lukateras Jun 11, 2018
f44012b
lib.release -> lib.trivial.release
lukateras Jun 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions pkgs/build-support/fetchurl/builder.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,23 @@ source $stdenv/setup

source $mirrorsFile

curlVersion=$(curl -V | head -1 | cut -d' ' -f2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm doing this instead of curl.version to support native stdenv (e.g. on Darwin).


# Curl flags to handle redirects, not use EPSV, handle cookies for
# servers to need them during redirects, and work on SSL without a
# certificate (this isn't a security problem because we check the
# cryptographic hash of the output anyway).
curl="curl \
--location --max-redirs 20 \
--retry 3 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was removing --retry 3 intentional?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks!

--disable-epsv \
--cookie-jar cookies \
--insecure \
$curlOpts \
$NIX_CURL_FLAGS"

curl=(
curl
--location
--max-redirs 20
--disable-epsv
--cookie-jar cookies
--insecure
--user-agent "curl/$curlVersion Nixpkgs/$nixpkgsVersion"
$curlOpts
$NIX_CURL_FLAGS
)

downloadedFile="$out"
if [ -n "$downloadToTemp" ]; then downloadedFile="$TMPDIR/file"; fi
Expand All @@ -32,7 +35,7 @@ tryDownload() {
# if we get error code 18, resume partial download
while [ $curlexit -eq 18 ]; do
# keep this inside an if statement, since on failure it doesn't abort the script
if $curl -C - --fail "$url" --output "$downloadedFile"; then
if "${curl[@]}" -C - --fail "$url" --output "$downloadedFile"; then
success=1
break
else
Expand Down Expand Up @@ -61,7 +64,7 @@ tryHashedMirrors() {

for mirror in $hashedMirrors; do
url="$mirror/$outputHashAlgo/$outputHash"
if $curl --retry 0 --connect-timeout "${NIX_CONNECT_TIMEOUT:-15}" \
if "${curl[@]}" --retry 0 --connect-timeout "${NIX_CONNECT_TIMEOUT:-15}" \
--fail --silent --show-error --head "$url" \
--write-out "%{http_code}" --output /dev/null > code 2> log; then
tryDownload "$url"
Expand Down
4 changes: 2 additions & 2 deletions pkgs/build-support/fetchurl/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@ in
assert sha512 != "" -> builtins.compareVersions "1.11" builtins.nixVersion <= 0;

let

urls_ =
if urls != [] && url == "" then
(if lib.isList urls then urls
Expand All @@ -107,7 +106,6 @@ let
else if sha256 != "" then { outputHashAlgo = "sha256"; outputHash = sha256; }
else if sha1 != "" then { outputHashAlgo = "sha1"; outputHash = sha1; }
else throw "fetchurl requires a hash for fixed-output derivation: ${lib.concatStringsSep ", " urls_}";

in

stdenvNoCC.mkDerivation {
Expand Down Expand Up @@ -135,6 +133,8 @@ stdenvNoCC.mkDerivation {

impureEnvVars = impureEnvVars ++ netrcImpureEnvVars;

nixpkgsVersion = lib.fileContents ../../../.version;
Copy link
Member

@bobvanderlinden bobvanderlinden May 31, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might be able to use lib.version lib.release now?


# Doing the download on a remote machine just duplicates network
# traffic, so don't do that.
preferLocalBuild = true;
Expand Down