Add backoff support to the redis output #7781

urso · 2018-07-27T15:53:07Z

The PR adds backoff support to the redis output, similar to the Elasticsearch/Logstash output. 2 Settings backoff.init and backoff.max are available. On failure we will wait for backoff.init before trying to reconnect. If reconnect fails, the wait times increase exponentially. The backoff.max configures the maximum duration until retry.

We are forced to reconnect to redis on error. Even for redis internal
errors. The custom backoff client wrapper detects if the error has been
an redis internal error, or any other kind of error. If the actual error
comes from redis, a reconnect must not reset the backoff timer. The
backoff client wrapper makes sure that the backoff timer is only reset
after a successfull publish.
In case of IO errors during connect (without having published any event
yet), we track these as failOther. A successfull connect will reset
the backoff timer.

We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer.

ph

LGTM, a few typos in the comments to fix.

ph · 2018-07-30T13:49:54Z

libbeat/outputs/redis/backoff.go

+
+// failReason is used to track the cause of an error.
+// The redis client forces us to reconnect on any error (even for redis
+// internal errors). The backoff timer must not be reset on a successfully


s/successfully/successful

ph · 2018-07-30T13:50:31Z

libbeat/outputs/redis/backoff.go

+// failReason is used to track the cause of an error.
+// The redis client forces us to reconnect on any error (even for redis
+// internal errors). The backoff timer must not be reset on a successfully
+// reconnect after publishing failed with  a redis internal


double space between 'with' and 'a'

ph · 2018-07-30T13:51:45Z

libbeat/outputs/redis/backoff.go

+		// give the client a chance to promote an internal error to a network error.
+		b.updateFailReason(err)
+		b.backoff.Wait()
+	} else if b.reason != failRedis { // Only reset backoff durartion if failure was due to IO errors.


s/durartion/duration

* Add custom backoff support to redis client We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer. (cherry picked from commit eaf516a)

Cherry-pick #7781 to 6.4: Add backoff support to the redis output

Cherry-pick #7781 to 6.x: Add backoff support to the redis output

ph · 2018-08-03T14:43:29Z

This PR need a 6.3 backport.

* Add custom backoff support to redis client We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer. (cherry picked from commit eaf516a)

We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer. (cherry picked from commit eaf516a)

We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer. (cherry picked from commit 1415dc1)

* Add custom backoff support to redis client We are forced to reconnect to redis on error. Even for redis internal errors. The custom backoff client wrapper detects if the error has been an redis internal error, or any other kind of error. If the actual error comes from redis, a reconnect should reset the backoff timer. The backoff client wrapper makes sure that the backoff timer is only reset after a successfull publish. In case of IO errors during connect (without having published any event yet), we track these as `failOther`. A successfull connect will reset the backoff timer. (cherry picked from commit 1415dc1)

Cherry-pick elastic#7781 to 6.4: Add backoff support to the redis output

urso added 4 commits July 27, 2018 14:58

Add backoff settings to redis output

4acbacd

Add redis backoff settings to reference configs

af7ce00

Add redis backoff settings to docs

500b29e

urso added review needs_backport PR is waiting to be backported to other branches. labels Jul 27, 2018

urso mentioned this pull request Jul 27, 2018

Beats common backoff behavior #6791

Closed

adrianlzt mentioned this pull request Jul 30, 2018

Implements backoff mechanism in redis client #7780

Closed

Use interface types in redis tests

a5820f2

ph self-requested a review July 30, 2018 13:48

ph added the libbeat label Jul 30, 2018

ph approved these changes Jul 30, 2018

View reviewed changes

Typos

7ae1030

ph approved these changes Jul 31, 2018

View reviewed changes

ph merged commit eaf516a into elastic:master Jul 31, 2018

urso mentioned this pull request Jul 31, 2018

Cherry-pick #7781 to 6.x: Add backoff support to the redis output #7817

Merged

urso added v6.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 31, 2018

urso mentioned this pull request Jul 31, 2018

Cherry-pick #7781 to 6.4: Add backoff support to the redis output #7819

Merged

urso added the v6.4.0 label Jul 31, 2018

andrewvc added a commit that referenced this pull request Jul 31, 2018

Merge pull request #7819 from urso/backport_7781_6.4

bc7bd2a

Cherry-pick #7781 to 6.4: Add backoff support to the redis output

andrewvc added a commit that referenced this pull request Jul 31, 2018

Merge pull request #7817 from urso/backport_7781_6.x

028bac7

Cherry-pick #7781 to 6.x: Add backoff support to the redis output

ph added the needs_backport PR is waiting to be backported to other branches. label Aug 3, 2018

urso mentioned this pull request Aug 3, 2018

Cherry-pick #7781 to 6.3: Add backoff support to the redis output #7852

Merged

urso added v6.3.3 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 3, 2018

urso deleted the output-redis-backoff branch February 19, 2019 18:40

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Merge pull request elastic#7819 from urso/backport_7781_6.4

a89c067

Cherry-pick elastic#7781 to 6.4: Add backoff support to the redis output

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backoff support to the redis output #7781

Add backoff support to the redis output #7781

urso commented Jul 27, 2018

ph left a comment

ph Jul 30, 2018

ph Jul 30, 2018

ph Jul 30, 2018

ph commented Aug 3, 2018

Add backoff support to the redis output #7781

Add backoff support to the redis output #7781

Conversation

urso commented Jul 27, 2018

ph left a comment

Choose a reason for hiding this comment

ph Jul 30, 2018

Choose a reason for hiding this comment

ph Jul 30, 2018

Choose a reason for hiding this comment

ph Jul 30, 2018

Choose a reason for hiding this comment

ph commented Aug 3, 2018