Skip to content
This repository has been archived by the owner on May 26, 2022. It is now read-only.

Add Backoff Cache Discovery #26

Merged
merged 14 commits into from
Oct 30, 2019
Merged

Add Backoff Cache Discovery #26

merged 14 commits into from
Oct 30, 2019

Conversation

aschmahmann
Copy link
Contributor

@aschmahmann aschmahmann commented Sep 6, 2019

To deal with repeated discovery requests (as seems likely to occur in libp2p/go-libp2p-pubsub#184) we need some sort of caching + backoff scheme.

These are a few configurable schemes to get us moving.

This also relates to libp2p/go-libp2p#707

backoff.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
Copy link
Contributor

@vyzo vyzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a cursory first pass.

backoff.go Outdated Show resolved Hide resolved
backoff.go Outdated Show resolved Hide resolved
backoff.go Outdated Show resolved Hide resolved
backoff.go Outdated Show resolved Hide resolved
backoffcache.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
backoffcache.go Outdated Show resolved Hide resolved
Copy link
Contributor

@vyzo vyzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need docstrings here, and also do something about the funky names of some procedures.
Also, we should add utility methods with known good coefficients that reduce the burden of using this API.

backoff.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
backoff.go Show resolved Hide resolved
backoffcache.go Show resolved Hide resolved
backoffcache.go Show resolved Hide resolved
backoffconnector.go Show resolved Hide resolved
@aschmahmann
Copy link
Contributor Author

aschmahmann commented Oct 25, 2019

@vyzo not sure why Github won't show the outdated/the updated comments in the conversation tab, but you can see them in the files tab.

Also, I moved the mock discovery used by this and the previous PR into a new file both can use.

@vyzo
Copy link
Contributor

vyzo commented Oct 25, 2019

Can you add some utility procedures for known good coefficients and the such?
I don't have the first clue what's a good coefficient for the polynomial, and I expect the same from users of the api.

@aschmahmann
Copy link
Contributor Author

aschmahmann commented Oct 25, 2019

Would you like me to emulate the one from the dialer https://github.com/libp2p/go-libp2p-swarm/blob/99831444e78c8f23c9335c17d8f7c700ba25ca14/swarm_dial.go which is

// BackoffBase is the base amount of time to backoff (default: 5s).
var BackoffBase = time.Second * 5

// BackoffCoef is the backoff coefficient (default: 1s).
var BackoffCoef = time.Second

// BackoffMax is the maximum backoff time (default: 5m).
var BackoffMax = time.Minute * 5

BackoffBase + BakoffCoef * PriorBackoffs^2

This is literally just math, what the right backoff times/rates are is a whole separate thing. The dialer says it uses quadratic instead of exponential backoff, but no reasons are given. All over the internet you can find resources on designing/benchmarking backoff strategies (e.g. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/).

For each service we use backoff in (e.g. pubsub, dialer, etc.) we should have a default. Don't see why that should be here though.

Side note: @raulk would you have any interest using this in the dialer? There are some issues about changing/upgrading the backoff e.g. libp2p/go-libp2p#1554

@vyzo
Copy link
Contributor

vyzo commented Oct 26, 2019

For each service we use backoff in (e.g. pubsub, dialer, etc.) we should have a default. Don't see why that should be here though.

Because the API is unusable without it? People using the API should have a simple way to get reasonable defaults.

backoff.go Outdated Show resolved Hide resolved
backoff.go Outdated Show resolved Hide resolved
Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM. I'm concerned about the complexity of these features but I can't think of a significantly simpler way to implement them.

backoffcache.go Show resolved Hide resolved
backoffcache.go Outdated Show resolved Hide resolved
backoffcache.go Show resolved Hide resolved
backoffconnector.go Show resolved Hide resolved
@aschmahmann
Copy link
Contributor Author

aschmahmann commented Oct 28, 2019

I'm concerned about the complexity of these features but I can't think of a significantly simpler way to implement them.

@Stebalien Me too. Idk how frequently we're going to need a caching structure wrapped around an event system that needs to catch up on previous events. However, if something similar comes up again we should just turn a lot of these into interface{} (yay golang's missing generics) so we don't need to maintain this kind of code in multiple locations.

@Stebalien Stebalien merged commit be2ad52 into master Oct 30, 2019
@Stebalien Stebalien deleted the feat/backoff branch October 30, 2019 01:09
@Stebalien
Copy link
Member

🚀

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants