Intermittent error on multi-tenancy: "Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details"

[pierre-yves-monnet]

Expected Behavior

No error should pop up

Current Behavior

A cluster is created, multi-tenancy enabled. 2 tenants are created, blue and green.

A script is created:

• deploying the same simple process on both tenants. This process contains a service task
• Starting a worker, with a filter on each tenant (so two workers are started)
• Every 5 seconds, a new process instance is created on the green tenant
• every 20 seconds, a new process instance is created on the red tenant

Process instances are created, and workers execute jobs, but from time to time, an Error: 16 UNAUTHENTICATED: is visible in the log. Even so
, all service tasks are completed.

Possible Solution

No solution

Steps to Reproduce

Create a cluster, with two tenants red and green
create a client ID / Client Secret
Run the script

Context (Environment)

Multi-tenancy usage

Detailed Description

Log visible:

 ##### Started/Job RED: 0/0, GREEN: 1/2
------  Green process instance started: [2251799813686010]
>>>> Execute GreenJob PI=[2251799813686010] tenant=[green]
15:42:07.412 | zeebe |  [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details


 ##### Started/Job RED: 0/0, GREEN: 2/3
------  Green process instance started: [4503599627371180]
>>>> Execute GreenJob PI=[4503599627371180] tenant=[green]
15:42:12.524 | zeebe |  [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details


 ##### Started/Job RED: 0/0, GREEN: 3/4
(node:226) UnhandledPromiseRejectionWarning: Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
    at callErrorFromStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
    at listener.onReceiveStatus.processedStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:97:36)
    at InterceptingListenerImpl.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at process.nextTick (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78)
    at process._tickCallback (internal/process/next_tick.js:61:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:160:32)
    at ServiceClientImpl.<anonymous> (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at Promise (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:272:47)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:226) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:226) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
--

Possible Implementation

red-green.txt
simple-bpmn.txt

[jwulf]

This is likely caused by resource starvation in the Zeebe gateway or Identity. Allocating more resources to them should mitigate this.

@pierre-yves-monnet what resources did you allocate for the cluster?

Do you have a cluster that I can test against?

[jwulf]

See this issue in Zeebe-land: camunda/camunda#18697

That's an attempt to mitigate this on the gateway.

Package 8.5.4 and 8.6.6 have a fix for #163 in it that may solve this.

[jwulf]

Notes for myself:

See here: https://jira.camunda.com/browse/SUPPORT-21985?focusedCommentId=350685&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-350685

and here: https://jira.camunda.com/browse/SUPPORT-22321

[jwulf]

"to work around the problem, please increase the maximum allowed number of threads via zeebe.gateway.threads.grpcMaxThreads. The default value is 2*numberOfCores"