Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear Event Hub Device Watermarks when source event hub changed #114

Merged
merged 4 commits into from
Jun 25, 2021

Conversation

pallar-ms
Copy link
Contributor

@pallar-ms pallar-ms commented Jun 18, 2021

Description -
When customer changes their source of events (source event hub) the IoT Connector watermark should be reset and process all data available from the earliest available point in the event hub.

Changes -

  • Append event hub name to the checkpoint blob path
  • Delete checkpoints corresponding to an older source event hub
  • Note - This change is applicable to the event hubs for both the app types (i.e. watermarks will be handled in the same fashion for input event hub or the normalization event hub, but they are independently maintained.)

Testing -

  • Manual testing by running the Fhir.Ingest console app and verifying the various scenarios
    • Change event hub namespace
    • Change event hub within the same event hub namespace
    • Same event hub
  • Added UTs for reset checkpoints

PENDING -
Once this change takes effect, the checkpoints in the older path will not be honored, which will trigger reprocessing of the data. To avoid this, we can copy over the older checkpoints to the new location. Dustin is helping gauge how many active Gen2 connectors we have and the size of data ingested, to assess the impact and take the call accordingly. Will not be closing this PR until then, but putting this out now for first pass of review.

* Clear checkpoints stored in blob paths pertaining to older event hub
@pallar-ms pallar-ms requested review from dustinburson, wi-y, rogordon01 and a team June 18, 2021 00:19
Copy link
Member

@dustinburson dustinburson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕐

Copy link
Member

@dustinburson dustinburson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Added a few more suggestions.

@pallar-ms pallar-ms merged commit a4480db into master Jun 25, 2021
@dustinburson dustinburson deleted the personal/pallar/clear_eventhub_watermarks branch November 8, 2022 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants