Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix unable to capture changes while concurrent writes #39

Merged
merged 1 commit into from
Oct 21, 2019

Conversation

lasanthaS
Copy link
Contributor

Purpose

When concurrent/out-of-order writes for the auto-increment column occurs in the database, the source cannot capture such events in the polling mode.

Resolves #31

Goals

This PR introduces a strategy to handle such events by waiting on missing records.

Approach

When the wait.on.missed.record parameter is set to true, the source expects polling.column values in (numeric) sequential order. For each record, the source generates an event. When reading and processing records in batches, if the source identifies a missed record it pauses the processing and waits until the missing record appears. If the missing record does not appear for missed.record.waiting.timeout time period (in seconds), it ignores the record and resumes processing the rest.

Security checks

@mohanvive mohanvive merged commit 0d8a557 into siddhi-io:master Oct 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to capture all data changes due to the out of order of the auto incremental column of the database.
2 participants