Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RabbitMQ Queues grow endlessly when running RabbitMQ in HA Mode #2738

Closed
Schnitzel opened this issue Jun 30, 2021 · 2 comments
Closed

RabbitMQ Queues grow endlessly when running RabbitMQ in HA Mode #2738

Schnitzel opened this issue Jun 30, 2021 · 2 comments

Comments

@Schnitzel
Copy link
Contributor

Describe the bug

By default Lagoon Core is deployed with a broker that has 3 rabbitmq nodes, this seems to create some issues with the GraphQL subscriptions that are used to update the UI in realtime.
It seems that the queues are not correctly removed and the amount of queues goes to around 3000, which then the rabbitmq channels go to 5000+ at which point rabbitmq blocks new requests. If this happens we have errors like

[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: Error: No channels left to allocate 
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: Error: No channels left to allocate 
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 
[lagoon-core-api-5f589fb46f-hnb92] [winston] Unknown logger level: error 
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325". 

Not sure if the UnhandledPromiseRejection is connected to the Error: No channels left to allocate but it's probably likely.

As soon as we change the rabbitmq to a single node, the system seems to work correctly: queues are not growing that crazy.

@rocketeerbkw
Copy link
Member

Related #1344

@tobybellwood
Copy link
Member

We no longer run RabbitMQ in HA mode, but reopening #1344 to address subscription, which I've reopened

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants