You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The text was updated successfully, but these errors were encountered:
alok87
changed the title
S3 batcher on restart should consider loader offset for robustness
RedshiftSink: Batcher on restart should consider loader offset for robustness
Aug 15, 2020
alok87
changed the title
RedshiftSink: Batcher on restart should consider loader offset for robustness
RedshiftBatcher: on restart should consider loader offset for robustness
Aug 17, 2020
S3 batcher does shutdown gracefully but every start should consider the last uploaded offset of batcher n the loader topic last message.
This will ensure regardless of what goes wrong, the system never looses data, never reprocess same data and thus becomes Idempotent.
Idea credits: Yelp experience https://engineeringblog.yelp.com/2016/10/redshift-connector.html
The text was updated successfully, but these errors were encountered: