common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10177

ph · 2019-01-18T17:03:37Z

This PR add a new interface called backoff.Backoff, this can be used to
generalize any backoff interaction. It move the current Backoff strategy
under an ExpBackoff type.

ExpBackoff is the same as before on every wait we just
exponentially increase the duration of the wait and sleep for that
amount.

EqualJitterBackoff uses an exponential increment of the duration but
will take half of that value as fixed sleep time and the other half
as a jitter. This will help distribute the new request when a cluster is
done instead of having all the beats trying to reconnect at once.

The Redis implementations and any clients wrapped with a backoff will
now use the EqualJitterBackoff, any other code will keep using the same
exponential strategy.

Fixes: #10172

libbeat/common/backoff.go

ph · 2019-01-21T17:12:34Z

@urso I've added the new interface and move any reference of *common.Backoff to use that interface, I haven't made it configurable by the user for this we probably want a bigger discussion so for now I left it to the implementer. Currently, Redis and output/backoff will now use the EqualJitterBackoff strategy as the default.

This mean that the default pipeline, the monitoring pipeline or any user of Redis will now have some random in it.

@kvch Journalbeat will now use the interface instead of the type for the backoff field but will still use the ExpBackoff strategy.

ph · 2019-01-21T17:14:35Z

Note: I have added a minimal test suite for both the implementation that tests:

can be interrupted by the close channel
can wait for some time

This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff times is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as a fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: elastic#10172

ph · 2019-01-21T17:18:16Z

@urso Also this is not a full jitter, more like an EqualJitter in that case, this ensures that the connection will at least sleep a period of time before retrying again, looking at the numbers from the article they look pretty similar with the rates.

urso · 2019-01-21T17:36:53Z

libbeat/common/backoff/equal_jitter.go

+	b.duration *= 2
+	if b.duration > b.max {
+		b.duration = b.max
+	}


nit: By moving this block before the computation of temp you don't need the init*2 on reset. In the end it doesn't matter :)

ph · 2019-01-21T18:30:25Z

Let's merge into 6.6 and 6.x

…amount of time (elastic#10177) This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: elastic#10172 (cherry picked from commit 4cb9bd7)

…ad of sleeping for a fixed amount of time (#10229) Cherry-pick of PR #10177 to 6.x branch. Original message: This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: #10172

…amount of time (#10177) (#10230) This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: #10172 (cherry picked from commit 4cb9bd7)

…amount of time (elastic#10177) This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: elastic#10172

…r instead of sleeping for a fixed amount of time (elastic#10229) Cherry-pick of PR elastic#10177 to 6.x branch. Original message: This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: elastic#10172

…amount of time (elastic#10177) (elastic#10230) This PR add a new interface called backoff.Backoff, this can be used to generalize any backoff interaction. It move the current Backoff strategy under an ExpBackoff type. ExpBackoff is the same as before on every wait we just exponentially increase the duration of the wait and sleep for that amount. EqualJitterBackoff uses an exponential increment of the duration but will take half of that value as fixed sleep time and the other half as a jitter. This will help distribute the new request when a cluster is done instead of having all the beats trying to reconnect at once. The Redis implementations and any clients wrapped with a backoff will now use the EqualJitterBackoff, any other code will keep using the same exponential strategy. Fixes: elastic#10172 (cherry picked from commit 7e68306)

ph added review libbeat needs_backport PR is waiting to be backported to other branches. labels Jan 18, 2019

ph requested a review from urso January 18, 2019 17:03

ph requested a review from a team as a code owner January 18, 2019 17:03

ph changed the title ~~common.Backoff now implements jitter instead of sleeping for a fixed~~ common.Backoff now implements jitter instead of sleeping for a fixed amount of time Jan 18, 2019

urso reviewed Jan 20, 2019

View reviewed changes

libbeat/common/backoff.go Outdated Show resolved Hide resolved

ph added in progress Pull request is currently in progress. and removed review labels Jan 21, 2019

ph force-pushed the feature/backoff-jitter branch 2 times, most recently from da6225f to 3f8f06e Compare January 21, 2019 17:11

ph force-pushed the feature/backoff-jitter branch from 3f8f06e to 5d7864b Compare January 21, 2019 17:16

Moving the licenser to EqualJitter

601f830

urso reviewed Jan 21, 2019

View reviewed changes

urso approved these changes Jan 21, 2019

View reviewed changes

ph added review and removed in progress Pull request is currently in progress. labels Jan 21, 2019

ph merged commit 4cb9bd7 into elastic:master Jan 21, 2019

ph mentioned this pull request Jan 21, 2019

Cherry-pick #10177 to 6.x: common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10229

Merged

ph added v6.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 21, 2019

ph mentioned this pull request Jan 21, 2019

Cherry-pick #10177 to 6.6: common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10230

Merged

ph added the v6.6.0 label Jan 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10177

common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10177

ph commented Jan 18, 2019 •

edited

Loading

ph commented Jan 21, 2019

ph commented Jan 21, 2019

ph commented Jan 21, 2019

urso Jan 21, 2019

ph commented Jan 21, 2019

common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10177

common.Backoff now implements jitter instead of sleeping for a fixed amount of time #10177

Conversation

ph commented Jan 18, 2019 • edited Loading

ph commented Jan 21, 2019

ph commented Jan 21, 2019

ph commented Jan 21, 2019

urso Jan 21, 2019

Choose a reason for hiding this comment

ph commented Jan 21, 2019

ph commented Jan 18, 2019 •

edited

Loading