[Feature] CosmosClient multi-tenant enablement

[kirankumarkolli]

Context

One of popular multi-tenant pattern, is to partition data per application tenant. Partitioning can be by Container OR container/PK (later is very prominent). Scale of multi-tenants might be very high.

Least privilege access principle demands application to use very constrained access credentials.
Azure Cosmos DB has ResourceTokens which enables fine grained access credentials.

Challenges/Gaps

Client instances ~ #tokens

Tokens are created on-demand (when new tenant created etc...), and new CosmosClient instances needs to be created to access data. Resulting in #clients approx. #tokens.

• Anti-pattern: Cosmos recommends Singleton pattern
• Non-optimal (result of multiple instances) and might result in throttling

Maintaining a valid token

Each token generated has a temporal validity after which are invalid. Application needs to keep re-generate new replacement tokens for the ones which are expiring.

• Re-generation demands CosmosClient re-creation
• Re-generation demands service interaction NW call
• Maintain map of token -> validity

This issue is trying to address the CosmosClient gaps only. Hence token generation gaps are out-of-scope.

Possibilities

Authorization provider API

Abstract authorization into a plug-in MasterKey plug-in will be out-of-box, enable customization for ResourceToken based implementation. Draft below

• Fully extensible: Can be even leveraged for credential swap/rotation
• Advanced usage: Depth of knowledge to map cosmos addressing -> tenant. Get-started sample might help, still user need to maintain it.
• Overloaded with non-user concerns: ex: Background calls, new resource addresses

abstract class ICosmosAuthorizationProvider
{
    string GetAuthorization(OperationContext context);
}

class MasterKeyAuthorizationProvider : ICosmosAuthorizationProvider
{
    public MasterKeyAuthorizationProvider(string masterKey)
    {
        ...
    }

    public override string GetAuthorization(OperationContext context)
    {
        return AuthorizationHelper.Generate..(context);
    }
}

class TokenAuthorizationProvider : ICosmosAuthorizationProvider
{
    private Dictionay<string, string> tenantToTokens;

    public MasterKeyAuthorizationProvider(string masterKey)
    {
        ...
    }

    public override string GetAuthorization(OperationContext context)
    {
        string tenant = GetTenant(context);
        return tenantToTokens[tenant];
    }
}

Request level credential

Application has full context of tenant and easier/intutive to pass required context into the API call.
One possibility is to have it as ReqeustOption, draft below.

What about non-user invoke API concerns like background calls etc.. few possibilities are

• Use master key & assert its non-use through custom handler
• Enable client-option to start with no-key & lazy bound initialization to API invocation

class RequestOptions
{
    ...
    public string Authorization { get; set; }
}

Recommendation

ReqeustOptions is simple and very intutive. And get started with "MasterKey" based constructor.
Its a kind of override mechanism. Assertion of always override can be asserted through CustomHandler.

References:

https://docs.microsoft.com/en-us/azure/cosmos-db/secure-access-to-data#resource-tokens
#622 ResourceTokens support

[christopheranderson]

For Resource Tokens, JS and Java already have something for this scenario:

JS supports TokenProvider: https://github.com/Azure/azure-cosmos-js/blob/master/src/auth.ts#L15

• One down side of this approach is that you need to muck up custom headers to pass any additional context to the lambda
• this also technically supports master key, but we leak internals more than is preferrable (we get away with it because yoloJS)

Java supports TokenResolver: https://github.com/Azure/azure-cosmosdb-java/blob/master/commons/src/main/java/com/microsoft/azure/cosmosdb/TokenResolver.java
- Here we fixed the downside that JS hit by providing a custom properties bag that doesn't get resolved as headers on to the request.
- This does not work for master key.

I think that we should consider deprecating both of these once we onboard and GA the pipeline pattern from the azure-sdk folks.

For master key, for Java, we're merging a PR today that supports the Credential pattern from azure-sdk team. We're tweaking it to be mutable so that you can rotate without killing the client. This was an ask from a customer who didn't want to pay the tax of restarting their Bean in their Spring Boot app. Azure/azure-sdk-for-java#4885 (/cc @kushagraThapar)

We've talked about adopting the credential pattern for ResourceTokens later so that you can rotate them without restarting your client (which in many ways is MORE critical since they expire and thus MUST be rotated)

I'd like to see us adopt more of the Credential pattern as I think it simplifies adding new auth mechanisms. Handlers/plugins should be used when needing to add business logic in the middle of the request pipeline for deciding which resource token to use.

[kirankumarkolli]

Thanks @christopheranderson for details.

With brevity generalizing that TokenResolver and Credential maps to plug-in model. This pattern fully abstract credentials/lifetimes and is fully extensible.

Broadly classifying the scenario's

• Single cred for all requests: Very simple and straight forward
• Single cred with lifetime (token refresh/master-key recycle): Same as above need mechanism to change/mute them
• Multiple creds and SDK will choose: V2 SDK has this options. Selection is non-trivial (least privilege, cross-partition, RID vs named invisibility)
• Per request/tenant cred: operationContext -> tenant mapping is business/application logic.

In all these cases caller already knows exactly what to use. So having it as a ReqeustOptions is simple, intuitive and straight forward also.

On Key-rotation:
Key rotation typically involves most of dependent services (or at-least >1) and is high impact for the application. Client recycle during that period can be an exception.

[FrankGagnon]

@kirankumarkolli Chiming in to say that I'm currently working on a multitenant application in which somewhere between tens of thousands and millions of resource tokens will be live in the system.

Adding the capacity to set the resource token in the request options would be a game changer in our application, making Cosmos Db a viable option. We can cache the keys/handle rotations ourselves, as long as we can specify the token for each request. Changing the key for the entire client would not solve the problem since we still couldn't have a client for all tenants.

Do you have any idea when this is planned to be implemented?

[vunvulear]

We fill the pressure to have a solution for this. I would be happy to see the ability to send the permission token when the request is made. In such a way we would have the ability to use only one instance of the client.

[aarondcoleman]

@kirankumarkolli bumping this as well. We're in need of this functionality and opening and closing a CosmosClient is causing major connection issues in our application at scale.

[vibeeshan025]

I think the requested functionality is invalid (Please correct me if I am wrong).

I have a multi-tenant app where each tenant is given a separate DB (All the DB is created under single Azure Cosmos DB account ). I have designed the app in a way that it's very stateless (No data is stored in memory, so I can spawn as many instances as I want when the usage is high ),

Each instance of the app directly connects to the account using CosmosClient. The app instance will just have one CosmosClient object throughout its life.

this.cosmosClient = new CosmosClient(EndpointUri, PrimaryKey);

In my case, one instance of the app will serve multiple tenants. JWT token will contain all the authorized tenants for a user in its JSON. (Stateless everywhere).

Once an API is requested, JWT is opened and we will see, for which tenant the request is for and create a Container object specific for the request, see below code. The key is, container objects are very lightweight and they are not checked with the server for their validity (No network trips) on creation. You might have to check for the validity when you create JWT tokens using cosmosClient.CreateDatabaseIfNotExistsAsync().

var container = this.cosmosClient.GetContainer("tenant-specific-db-name", "container-that-exisits-in-all-of-your-dbs");

The benchmarks are very promising. I was able to create 1,000,000 container objects per second on the fly without multithreading (single-core) and without async-await in debug mode in .net core 3.1 (HP Elitebook 840 G6, i7 8565U). I did not dig further as this was more than sufficient for my use-case. Wired, that only 12.5% of my CPU was used while benchmarking, I did not have much time to find out the reason.

Now you can use the container and do any operation specific to the tenant.

Warning: I could not find any official documentation giving the assurance that the container object will always be lightweight in the future releases, however, the current benchmarks are very promising and it has not changed in the last 3 major updates.

[tomasdeml]

If I am not mistaken, the situation with multiple CosmosClient instances is exacerbated by the fact that the client owns an instance of the DocumentClient class which in turn owns an HttpClient. The http client is disposed with the DocumentClient when the CosmosClient is disposed. And we all know what this can lead to...

[SzymonSmykala]

Any progress on that?

[MikeNicholls]

Although it's currently only in preview, using the new RBAC support is one way of achieving this if you're able to use AD identities and don't explicitly require resource tokens.

[WestDiscGolf]

bump

The ability for the client to handle multiple different connections in a multi tenant system is an important functional requirement. If data has to be separated by database going through a single code base then the connection string, db name, containers etc. have to be per request and resolved at runtime depending on claims etc. Due to this it appears to go against the "cosmos client should be registered as a singleton" recommendation/best practice and then we'd have to get into the world of manually working with HttpClientFactory which feels like you're going to have a bad day :-(

Any help/suggestions would be appreciated.

[LeszekKalibrate]

bump
2023 - is this implemented yet?