You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
By default Lagoon Core is deployed with a broker that has 3 rabbitmq nodes, this seems to create some issues with the GraphQL subscriptions that are used to update the UI in realtime.
It seems that the queues are not correctly removed and the amount of queues goes to around 3000, which then the rabbitmq channels go to 5000+ at which point rabbitmq blocks new requests. If this happens we have errors like
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: Error: No channels left to allocate
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: Error: No channels left to allocate
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
[lagoon-core-api-5f589fb46f-hnb92] [winston] Unknown logger level: error
[lagoon-core-api-5f589fb46f-hnb92] [2021-06-30 16:07:33] [pubsub-error]: uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
[lagoon-core-api-5f589fb46f-hnb92] UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "5325".
Not sure if the UnhandledPromiseRejection is connected to the Error: No channels left to allocate but it's probably likely.
As soon as we change the rabbitmq to a single node, the system seems to work correctly: queues are not growing that crazy.
The text was updated successfully, but these errors were encountered:
Describe the bug
By default Lagoon Core is deployed with a broker that has 3 rabbitmq nodes, this seems to create some issues with the GraphQL subscriptions that are used to update the UI in realtime.
It seems that the queues are not correctly removed and the amount of queues goes to around 3000, which then the rabbitmq channels go to 5000+ at which point rabbitmq blocks new requests. If this happens we have errors like
Not sure if the
UnhandledPromiseRejection
is connected to theError: No channels left to allocate
but it's probably likely.As soon as we change the rabbitmq to a single node, the system seems to work correctly: queues are not growing that crazy.
The text was updated successfully, but these errors were encountered: