Network retry issue 1602 #2396

sbeckeriv · 2016-02-19T05:40:20Z

Dearest Reviewer,

This branch resolves #1602 which relates to retrying network
issues automatically. There is a new utility helper for retrying
any network call.

There is a new config called net.retry value in the .cargo.config
file. The default value is 2. The documentation has also been
updated to reflect the new value.

Thanks
Becker

alexcrichton · 2016-02-20T01:31:20Z

Thanks for the PR @sbeckeriv! Some thoughts of mine:

Could this logic be centralized as much as possible? Currently the retry loop is encoded in two separate places and ideally it'd only be defined in one.
I'm pretty wary of retrying on any error, I've generally had bad experiences with tools that have that kind of behavior. Most of the time an error is impossible to resolve until something changes, it's only pretty rarely that an intermittent error can be retried to fix behavior. Can we whitelist specific errors to retry on?
I think that this logic needs to be attached directly to the calls to the network themselves, right now it's placed very high up in the chain and far away from where some updates may happen. There's also network updates throughout the rest of cargo (e.g. when publishing or installing) which won't get affected in the locations this is added.
Can you be sure to add some tests for this? It may be easiest to just mirror the HTTPS tests where some local sever is spun up, connected to, and immediately dropped to generate an error.
Can you be sure to document any new options in src/doc/config.md?

sbeckeriv · 2016-02-20T19:22:09Z

Thanks for the reply.

I will move the net_retry logic in to the utils config object. For the logic it self I believe I know how to do this. I am going to write a function that takes a closure and handles the logic.

I will figure out a white list enum too.

logic needs to be attached directly to the calls to the network themselves

I think with the closure concept I could push the logic down the stack, however, that would not blanket all network calls. Do you have some thoughts on how I might go about that? I think most network calls are else where crates. I could update curl-rust to support a retry option pretty easy. Make it an http setting like proxy and timeout. I read that libcurl does not directly support it. I need to read more around the ssh side. Let me know your thoughts.

alexcrichton · 2016-02-21T02:31:44Z

Do you have some thoughts on how I might go about that? I think most network calls are else where crates. I could update curl-rust to support a retry option pretty easy.

Nah I can help audit after the infrastructure in place, and then I can also help to ensure that we always push network requests through the same location. I'd also recommend that even if this were an option in curl-rust that we'd still do it in Cargo as we'd likely want to give a UI update as part of it as well anyway.

sbeckeriv · 2016-02-22T00:24:25Z

Before I get to far thoughts on these changes:
net retry value from the config with a default value:
https://github.com/rust-lang/cargo/pull/2396/files#diff-4559bc8f75ebf0a4d42be6e4e7fe9eaaR200

Wrapped retry logic:
https://github.com/rust-lang/cargo/pull/2396/files#diff-f9445846d299388561a3972b309c9d9eR1
and an example call:
https://github.com/rust-lang/cargo/pull/2396/files#diff-591901395b44f65a638e1c81531878d2R173

Thanks! It was fun to figure out the with_retry.

alexcrichton · 2016-02-22T18:25:34Z

Yeah, that looks great to me!

I think we may want to have another callback for "was this error one we can retry from", but other than that it looks like the right infrastructure to me

sbeckeriv · 2016-02-24T05:57:44Z

Hey,
I have updated to the new callback. I updated the docs and added in some comments in the code. I copied a https test and I am checking the stdout.

For the "was this error one we can retry from"; error handling is new to me. Should I pass it the CargoResult value or the unwrapped Err? What might that callback look like? How do I check its a Err I care about vs any other error?

Thanks!

alexcrichton · 2016-02-25T18:08:57Z

src/cargo/core/registry.rs

@@ -134,7 +140,7 @@ impl<'cfg> PackageRegistry<'cfg> {
            Some(&(ref previous, _)) => {
                if previous.precise() == namespace.precise() {
                    debug!("load/match    {}", namespace);
-                    return Ok(())
+                    return Ok(());


Perhaps you have rustfmt on auto-run? Could you back out these changes for now? They tend to clutter up the diff unfortunately :(

alexcrichton · 2016-02-25T18:12:35Z

I think that it may be good to introduce a new trait:

trait NetworkError {
    pub fn may_be_spurious(&self) -> bool;
}

That way you can define the retry function with a NetworkError bound on the E type coming out. I would only implement NetworkError for types like git2::Error and curl::Error rather than a blanket CargoError because that'll help target the errors individually.

bors · 2016-02-26T21:12:50Z

☔ The latest upstream changes (presumably #2397) made this pull request unmergeable. Please resolve the merge conflicts.

sbeckeriv · 2016-03-03T04:29:31Z

I pushed down the network call in to the checkout command.

I added the NetworkError trait. I am unsure how to apply the bounds in the retry method. I added it as a trait bound to T but the T is the return Ok value for the cargo result not the error. Any advice or examples of how to do this would be great!

Thanks again
Becker

alexcrichton · 2016-03-03T18:11:22Z

src/cargo/util/network.rs

+/// Example:
+/// use util::network;
+/// cargo_result = network.with_retry(&config, || something.download());
+pub fn with_retry<T, F>(config: &Config, mut callback: F) -> CargoResult<T>


I think the signature you want for this method is something along the lines of:

pub fn with_retry<T, E, F>(config: &Config, mut callback: F) -> CargoResult<T> where F: FnMut() -> Result<T, E>, E: NetworkError + CargoError

This would then match on the result of callback and call maybe_spurious to see whether it is indeed a spurious error or not.

alexcrichton · 2016-03-04T18:19:44Z

@sbeckeriv yeah the point of the bounds was to highlight the fact that Box<CargoError> is opaque and very difficult to see if we need to retry it or not. It basically just means that the retry needs to be pushed farther down in the stack.

sbeckeriv · 2016-03-06T21:59:15Z

The bounds make perfect sense now. I pushed the config down to the bottom fetch call. I am still getting "mismatched type" errors on the result I am returning. The generics at this level are beyond me. The error text https://gist.github.com/sbeckeriv/3866bd2ab7241da32b7c I have tried many combinations. I thought the error came from how I initialized the result value but I removed that. The normal human err is does not impl network error. I then thought it was because the shell command internally could cause an error but I removed the try with no luck.

Do i need to rebuild the result or implement a from?

Thanks again,
Becker

alexcrichton · 2016-03-07T07:43:54Z

No worries! Here's how I might write this function:

pub fn with_retry<T, E, F>(config: &Config, mut callback: F) -> CargoResult<T>
    where F: FnMut() -> Result<T, E>,
          E: NetworkError
{
    let mut remaining = try!(config.net_retry());
    loop {
        match callback() {
            Ok(ret) => return Ok(ret),
            Err(ref e) if e.maybe_spurious() && remaining > 0 => {
                // print to shell
                remaining -= 1;
            }
            Err(e) => return Err(Box::new(e)),
        }
    }
}

Perhaps that can help you out?

sbeckeriv · 2016-03-07T15:29:24Z

That does! When I first started I decomposed but I went the other way for reasons I do not remember. I need to update my test but it compiles now. Thanks!

sbeckeriv · 2016-03-10T19:07:10Z

I have the retry at the lowest level fetch. I think this covers all the git actions. I am looking in to the interactions with the registry.

sbeckeriv · 2016-03-10T19:58:23Z

As I figure out the code I am not sure I want to extern cargo in to crates-io. I am guessing there are separated in to there own project for a reason. Now I would require Config and network to cross back in to creates. I could just copy the concept but there is no config system for cargo-io helpers. Maybe this is figured out in another pull request? Let me know your thoughts.

Thanks again!
Becker

alexcrichton · 2016-03-11T08:21:44Z

src/cargo/util/errors.rs

+}
+
+impl NetworkError for git2::Error {
+    fn maybe_spurious(&self) -> bool { true }


For git errors this may want to at least check the class of the error for Net, and perhaps look at the code as well to ensure it's not a known non-spurious error.

alexcrichton · 2016-03-11T08:23:07Z

Looking good! I think though that this may not catch where we download from the registry (using curl), just git so far. API calls with crates.io also look like they may need instrumentation as well, and you can probably just implement NetworkError for the error type coming out of the crates-io library

sbeckeriv · 2016-03-16T21:50:27Z

I updated the error checking.

For git2::ErrorCode its not clear which errors relate to network activity. Maybe EOF for a disconnect? In my test I found that not connecting to a host gave an git2::ErrorClass::Os error. I am checking for both Os and Net now. Maybe my test is invalid?

For curl errors I left a comment to what I believe the number maps to based off of this website https://curl.haxx.se/libcurl/c/libcurl-errors.html

Thanks again for all the help.
Becker

alexcrichton · 2016-03-17T00:34:48Z

tests/test_bad_config.rs

-    .file("src/lib.rs", "");
+    .file("src/lib.rs", "").file(".cargo/config", r#"
+    [net]
+    retry=0


In theory these modifications to tests shouldn't be necessary. All of these are indicative of errors that can never be resolved, so they shouldn't print out warnings anyway

Yes. These are leftover from before the trait. I should be able to remove these.

alexcrichton · 2016-05-11T17:22:04Z

src/cargo/util/errors.rs

@@ -1,3 +1,4 @@
+extern crate curl_sys;


Could this be added to the crate root?

I moved it to src/cargo/lib.rs I think thats the "root". Let me know if that is wrong.

Ah yeah that's it!

alexcrichton · 2016-05-11T17:25:08Z

Looking good @sbeckeriv, thanks!

alexcrichton · 2016-05-12T00:31:27Z

src/cargo/util/network.rs

+            Err(ref e) if e.maybe_spurious() && remaining > 0 => {
+                try!(config.shell()
+                           .warn(format!("{} {} tries remaining", e,
+                                         remaining)));


Perhaps this could be along the lines of:

let msg = format!("spurious network error ({} tries remaining): {}", remaining, e); try!(config.shell().warn(msg));

alexcrichton · 2016-05-12T00:32:50Z

Awesome! Just a minor tweak to some wording, and then perhaps some squashing? Other than that r=me

sbeckeriv · 2016-05-12T02:03:52Z

@alexcrichton I made it one commit. Thanks for working with me and all the help! I learned a lot.

Becker

alexcrichton · 2016-05-12T16:27:13Z

src/cargo/util/network.rs

+            Ok(ret) => return Ok(ret),
+            Err(ref e) if e.maybe_spurious() && remaining > 0 => {
+                let msg = format!("spurious network error ({} tries \
+                          remaining): {}", remaining, e);


I think the remaining count here is actually remaining - 1

The retry config is 2 by default.

count is 2 I run once and fail i output 2 tries remains subtract 1 count is 1 run a second time 1 tries remaining subtract 1 count is 0 retry again just error.

The retry count is the count of retries not the count of total tries. I can subtract one from the config or update the docs to be explicitly clear it is the count of retries and not the total network tries?

Ah right yeah that makes sense, nevermind!

alexcrichton · 2016-05-12T16:27:40Z

Looks like the Windows CI is failing for non-spurious reasons?

alexcrichton · 2016-05-12T16:47:35Z

@bors: r+ 2f04d00ae2b5ddce01b7f5890e5311e287ee198f

bors · 2016-05-12T16:47:37Z

⌛ Testing commit 2f04d00 with merge e2648ce...

bors · 2016-05-12T16:57:33Z

💔 Test failed - cargo-win-msvc-64

sbeckeriv · 2016-05-12T17:07:17Z

Sorry I dont have access to a windows box right now. I might tonight. Does windows not load the .cargo/config file or does it output extra information?

alexcrichton · 2016-05-12T17:38:15Z

It should load that, yeah, but I suspect the problem here is that the exact error message is slightly different. It should be possible to just use with_stderr_contains perhaps?

Dearest Reviewer, This branch resolves rust-lang#1602 which relates to retrying network issues automatically. There is a new utility helper for retrying any network call. There is a new config called net.retry value in the .cargo.config file. The default value is 2. The documentation has also been updated to reflect the new value. Thanks Becker

sbeckeriv · 2016-05-12T18:27:39Z

Works locally. Lets see!

alexcrichton · 2016-05-12T18:40:54Z

@bors: r+ 7b03532

Network retry issue 1602 Dearest Reviewer, This branch resolves #1602 which relates to retrying network issues automatically. There is a new utility helper for retrying any network call. There is a new config called net.retry value in the .cargo.config file. The default value is 2. The documentation has also been updated to reflect the new value. Thanks Becker

bors · 2016-05-12T18:45:04Z

⌛ Testing commit 7b03532 with merge a4a9491...

bors · 2016-05-12T20:50:51Z

☀️ Test successful - cargo-cross-linux, cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64

rust-highfive assigned alexcrichton Feb 19, 2016

alexcrichton reviewed Feb 25, 2016
View reviewed changes

alexcrichton reviewed Mar 3, 2016
View reviewed changes

alexcrichton reviewed Mar 11, 2016
View reviewed changes

alexcrichton reviewed Mar 17, 2016
View reviewed changes

alexcrichton reviewed May 11, 2016
View reviewed changes

alexcrichton reviewed May 12, 2016
View reviewed changes

sbeckeriv force-pushed the network-retry-1602 branch from 4fc800d to b3de4c2 Compare May 12, 2016 01:51

alexcrichton reviewed May 12, 2016
View reviewed changes

sbeckeriv force-pushed the network-retry-1602 branch from b3de4c2 to 2f04d00 Compare May 12, 2016 16:42

sbeckeriv force-pushed the network-retry-1602 branch from 2f04d00 to 7b03532 Compare May 12, 2016 18:27

bors merged commit 7b03532 into rust-lang:master May 12, 2016

bors mentioned this pull request May 12, 2016

Command alias or Alias command #1091 #2679

Merged

sbeckeriv deleted the network-retry-1602 branch May 12, 2016 21:07

metajack mentioned this pull request May 13, 2016

Update Cargo to 2016-05-12 servo/servo#11177

Merged

5 tasks

This was referenced Apr 27, 2017

crates.io responded with 500 instead of 200 when downloading crate rust-lang/rust#41545

Closed

Retry if S3 returns 500 #3962

Closed

Network retry issue 1602 #2396

Network retry issue 1602 #2396

Conversation

sbeckeriv commented Feb 19, 2016 • edited Loading

alexcrichton commented Feb 20, 2016

sbeckeriv commented Feb 20, 2016

alexcrichton commented Feb 21, 2016

sbeckeriv commented Feb 22, 2016

alexcrichton commented Feb 22, 2016

sbeckeriv commented Feb 24, 2016

Choose a reason for hiding this comment

alexcrichton commented Feb 25, 2016

bors commented Feb 26, 2016

sbeckeriv commented Mar 3, 2016

Choose a reason for hiding this comment

alexcrichton commented Mar 4, 2016

sbeckeriv commented Mar 6, 2016

alexcrichton commented Mar 7, 2016

sbeckeriv commented Mar 7, 2016

sbeckeriv commented Mar 10, 2016

sbeckeriv commented Mar 10, 2016

Choose a reason for hiding this comment

alexcrichton commented Mar 11, 2016

sbeckeriv commented Mar 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented May 11, 2016

Choose a reason for hiding this comment

alexcrichton commented May 12, 2016

sbeckeriv commented May 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented May 12, 2016

alexcrichton commented May 12, 2016

bors commented May 12, 2016

bors commented May 12, 2016

sbeckeriv commented May 12, 2016

alexcrichton commented May 12, 2016

sbeckeriv commented May 12, 2016

alexcrichton commented May 12, 2016

bors commented May 12, 2016

bors commented May 12, 2016

sbeckeriv commented Feb 19, 2016 •

edited

Loading