Is it correct that watcher does not reload credentials after the stream is interrupted?

[cdhowie]

I don't see anything in the documentation about watcher reloading credentials after a stream error, which leads me to believe this doesn't happen. As a result, if:

1. I have an active watcher, and
2. The watcher encounters an error and is restarted, but
3. In the meantime, the credentials of the service account were rotated...

Then the stream will never be able to reconnect. This results in needing to manually implement credential reloading on a stream error, which looks a bit ugly -- though it can be wrapped in a function, at least.

pub fn recreate_on_err<F, U, S, T, E>(f: F) -> impl Stream<Item = Result<T, E>>
where
    F: FnMut() -> U,
    U: Future<Output = S>,
    S: Stream<Item = Result<T, E>>,
{
    futures::stream::repeat_with(f)
        .then(|i| i)
        .map(|s| {
            let mut take = true;
            s.take_while(move |i| {
                ready({
                    let r = take;
                    take = take && !i.is_err();
                    r
                })
            })
        })
        .flatten()
}

This allows one to do:

recreate_on_err(|| async {
  watcher(
    Api::all(Client::try_default().await.unwrap()),
    &watcher::Config::default()
  )
})

So my questions are:

1. Is this indeed necessary, and
2. If so, is this something that this library should handle in some way, such as by making the first argument take impl FnMut() -> impl Future<Output = Api<K>> instead of Api<K>?

[clux]

watcher does not do any error handling like this itself, but Client does, and a watcher is ultimately an abstraction on the Client.

Client has an AuthLayer (in its tower stack) in which the RefreshableToken will handle updating updated credentials silently behind the seams (i.e. before the actual business logic of making the watcher http call happens).

So, if you get an error, the watcher will reset the state to default/empty (and you can log that error event), but when you poll the next watcher event it will go through the auth layer before actually making the relist call, and if there are updates needed, they should happen then.

[cdhowie]

Ah, perfect, thanks! I tried to follow the chain of events from Client::try_default() but found it somewhat confusing, so couldn't verify either way.

Do you know if this behavior is documented anywhere? I looked for documentation on this topic before asking the question and I couldn't find any.

[clux]

There's not a whole lot tbh. We have some for contributors in the architecture doc, particularly https://kube.rs/architecture/#client
but it's also pretty old.

Other than that, client/builder.rs has the most generic make_generic_builder fn that sets up all the tower layers. There are simpler examples of setting them up yourself, in examples/custom_client* which are less dense and less feature-loaded.

We don't really document the tower::Layers much ourselves because it's a complexity most users do not need to concern themselves with unless they need to override layers, and the upstream tower docs and tower-http docs are pretty good for filling the gaps. They provide a bunch of nicely documented modular components that work in a decorator style before the actual http call goes out on the wire on our end.

we probably need a better doc for this in kube.rs long term though. EDIT: kube-rs/website#70