Distfile caching on failed fetch stage

[tomalexander]

I think the current behavior for poudriere is that it does not save any dist files when the fetch stage fails, which is leaving me stuck unable to build graphics/librsvg2-rust.

For a while now I've had graphics/librsvg2-rust stuck as a failed build in my poudriere instance. I've attached 3 log files of my failed build of that port. Looking at the logs, it is downloading a bunch of crates, alternating between two distinct behaviors:

First, 3-4 crates successfully download from crates.io:

=> librsvg-2.58.0.tar.xz doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://download.gnome.org/sources/librsvg/2.58/librsvg-2.58.0.tar.xz
librsvg-2.58.0.tar.xz                                 5592 kB   16 MBps    01s
=> rust/crates/anes-0.1.6.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/anes/0.1.6/download?dummy=/rust/crates/anes-0.1.6.crate
rust/crates/anes-0.1.6.crate                            23 kB   13 MBps    00s
=> rust/crates/approx-0.5.1.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/approx/0.5.1/download?dummy=/rust/crates/approx-0.5.1.crate
rust/crates/approx-0.5.1.crate                          14 kB  136 MBps    00s
=> rust/crates/assert_cmd-2.0.13.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/assert_cmd/2.0.13/download?dummy=/rust/crates/assert_cmd-2.0.13.crate
rust/crates/assert_cmd-2.0.13.crate                     23 kB  108 kBps    00s

Then, there is a block of crates that fail to download from crates.io with Connection reset by peer, so they pull from distcache.FreeBSD.org:

=> rust/crates/bit-set-0.5.3.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit-set/0.5.3/download?dummy=/rust/crates/bit-set-0.5.3.crate
fetch: https://crates.io/api/v1/crates/bit-set/0.5.3/download?dummy=/rust/crates/bit-set-0.5.3.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit-set-0.5.3.crate
rust/crates/bit-set-0.5.3.crate                         14 kB 1671 kBps    00s
=> rust/crates/bit-vec-0.6.3.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit-vec/0.6.3/download?dummy=/rust/crates/bit-vec-0.6.3.crate
fetch: https://crates.io/api/v1/crates/bit-vec/0.6.3/download?dummy=/rust/crates/bit-vec-0.6.3.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit-vec-0.6.3.crate
rust/crates/bit-vec-0.6.3.crate                         19 kB 2250 kBps    00s
=> rust/crates/bit_field-0.10.2.crate doesn't seem to exist in /portdistfiles/.
=> Attempting to fetch https://crates.io/api/v1/crates/bit_field/0.10.2/download?dummy=/rust/crates/bit_field-0.10.2.crate
fetch: https://crates.io/api/v1/crates/bit_field/0.10.2/download?dummy=/rust/crates/bit_field-0.10.2.crate: Connection reset by peer
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/rust/crates/bit_field-0.10.2.crate
rust/crates/bit_field-0.10.2.crate                      10 kB  193 MBps    00s

Then the downloads from crates.io start working again briefly.

So looking at this pattern, I think I'm getting rate-limited by crates.io. This doesn't cause problems when the crates also exist in distcache.FreeBSD.org but not all the crates exist there.

There are two interesting bits from this:

1. The crate that the build eventually fails on is different between runs. This is down to random chance, failing the first time I am both rate-limited and the crate doesn't exist on distcache.FreeBSD.org. So if I'm lucky, my non-rate-limited requests happen to be the crates missing from distcache.FreeBSD.org.
2. The crates do not end up in /usr/ports/distfiles/rust/crates/. For example, in 3.log (attached), the build fails on color_quant-1.1.0.crate but in both my earlier builds of 1.log and 2.log that crate successfully downloaded.

Running find / -type f -name '*color_quant*crate*' turns up no files, nor does find /usr/ports/distfiles/rust/crates/ | grep -i quant. But ls -l /usr/ports/distfiles/rust/crates | wc -l shows I have 541 crates cached.

So what I suspect is happening is poudriere only moves the downloaded distfiles into /usr/ports/distfiles if it succeeds in fetching ALL of the crates. Is this statement correct?

And assuming that is correct, this presents a problem because if each individual successfully downloaded crate ended up cached in /usr/ports/distfiles then over a series of builds, the rate-limiting RNG gods would slowly grant me a couple new crates each build which would eventually let me build graphics/librsvg2-rust.

So I've posted this as a discussion instead of filing a bug report because I don't know if my assumptions on poudriere's behavior are correct. I'm hoping someone here can confirm?

Logs:
1.log
2.log
3.log

[garrettboone]

I don't know why this failure to download the patches happens, but you could download the files manually into /usr/local/poudriere/distfiles and then run the build again. It was happening to me with gitlab-ce.