Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MQTT subscription #361

Merged
merged 11 commits into from
Oct 22, 2024
Merged

MQTT subscription #361

merged 11 commits into from
Oct 22, 2024

Conversation

timcowlishaw
Copy link
Contributor

@timcowlishaw timcowlishaw commented Sep 25, 2024

Addresses #355, allowing us to scale the mqtt task through parallel processing. We have two named "main" subscribers which ensure that messages are not dropped in the event of downtime (they have consistent client ids and persistent sessions), and a secondary subscriber role (with transient ids and clean sessions) which can be scaled with load. Needs thorough testing on staging before deployment!

⚠️ on deployment, we need to: ⚠️

  • bring down the app entirely, including the old mqtt-task
  • delete the old "mqtt subscriber" client in emqx, so that no messages are retained for it, as the client id changes in this release
  • run migrations BEFORE bringing up the mqtt-task again, as otherwise duplicate components will be created
  • bring up the entire stack again

@timcowlishaw timcowlishaw linked an issue Sep 25, 2024 that may be closed by this pull request
6 tasks
@timcowlishaw
Copy link
Contributor Author

(deployed to staging on branch staging-2024-01-01, although it was actually deployed on the 25th of sept, sorry for the confusion)

@timcowlishaw
Copy link
Contributor Author

#TODO: ensure that mqtt-task gets an exclusive lock on the components table before creating a component - it only happens in "unrealistic" testing conditions currently, but it is possible for two identical components to be created at once

@timcowlishaw timcowlishaw force-pushed the mqtt-task-shared-subscription branch from 871f074 to a728506 Compare October 22, 2024 11:00
@timcowlishaw timcowlishaw merged commit 9430b5f into master Oct 22, 2024
2 checks passed
@timcowlishaw timcowlishaw deleted the mqtt-task-shared-subscription branch October 22, 2024 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log a sentry error when the MQTT internal queue reaches saturation point Shared subscription in MQTT
2 participants