-
Notifications
You must be signed in to change notification settings - Fork 451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CosmosDB emulator: set default number of partitions to 1 #5641
Conversation
@radical it might be that 2 is the minimum so may as well set it there to avoid confusion? |
Would be good to add a test to ensure that if someone does this - they can override the value we provides:
|
When I set it to |
Interesting failure:
|
That message came from the emulator? |
aspire/playground/CosmosEndToEnd/CosmosEndToEnd.ApiService/Program.cs Lines 51 to 59 in 6df3fef
Error json expanded: {
"Summary" : {
"GatewayCalls" : {
"(503, 1007)" : 1
}
},
"children" : [
{
"children" : [
{
"children" : [
{
"children" : [
{
"children" : [
{
"children" : [
{
"children" : [
{
"data" : {
"Client Side Request Stats" : {
"AddressResolutionStatistics" : [],
"ContactedReplicas" : [],
"FailedReplicas" : [],
"HttpResponseStats" : [
{
"ActivityId" : "1a191f5e-2c77-4731-88bc-20e35c4c16dc",
"DurationInMs" : 141.8954,
"HttpMethod" : "POST",
"ReasonPhrase" : "Service Unavailable",
"RequestUri" : "https://127.0.0.1:44089/dbs/ef/colls",
"ResourceType" : "Collection",
"StartTimeUTC" : "2024-09-10T04:04:57.2625039Z",
"StatusCode" : "ServiceUnavailable"
}
],
"Id" : "AggregatedClientSideRequestStatistics",
"RegionsContacted" : [],
"StoreResponseStatistics" : []
},
"Point Operation Statistics" : {
"ActivityId" : "1a191f5e-2c77-4731-88bc-20e35c4c16dc",
"BELatencyInMs" : null,
"ErrorMessage" : "Microsoft.Azure.Documents.DocumentClientException:
{"code":"ServiceUnavailable","message":"Sorry, we are currently experiencing high demand in this region South Central US, and cannot fulfill your request at this time Tue, 10 Sep 2024 04:04:57 GMT.\\r\
ActivityId: 1a191f5e-2c77-4731-88bc-20e35c4c16dc, Microsoft.Azure.Documents.Common/2.14.0"}
RequestUri: https://127.0.0.1:44089/dbs/ef/colls;
RequestMethod: POST;
Header: Authorization Length: 80;
Header: x-ms-date Length: 29;
Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1;
Header: x-ms-activity-id Length: 36;
Header: Cache-Control Length: 8;
Header: User-Agent Length: 112;
Header: x-ms-version Length: 10;
Header: Accept Length: 16;
Header: traceparent Length: 55;
, Request URI: /dbs/ef/colls, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.34.4
at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)
at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)",
"Id" : "PointOperationStatistics",
"RequestCharge" : 0,
"RequestSessionToken" : null,
"RequestUri" : "dbs/ef",
"ResponseSessionToken" : null,
"ResponseTimeUtc" : "2024-09-10T04:04:57.4640883Z",
"StatusCode" : 503,
"SubStatusCode" : 1007
}
},
"duration in milliseconds" : 190.3148,
"name" : "Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request"
}
],
"duration in milliseconds" : 203.5346,
"name" : "Microsoft.Azure.Cosmos.Handlers.TransportHandler"
}
],
"duration in milliseconds" : 203.5386,
"name" : "Microsoft.Azure.Cosmos.Handlers.RouterHandler"
}
],
"duration in milliseconds" : 203.8547,
"name" : "Microsoft.Azure.Cosmos.Handlers.RetryHandler"
}
],
"duration in milliseconds" : 203.869,
"name" : "Microsoft.Azure.Cosmos.Handlers.TelemetryHandler"
}
],
"data" : {
"System Info" : {
"systemHistory" : [
{
"cpu" : 0,
"dateUtc" : "2024-09-10T04:04:53.6506560Z",
"memory" : 1860140,
"numberOfOpenTcpConnection" : 0,
"threadInfo" : {
"availableThreads" : 32765,
"isThreadStarving" : "no info",
"maxThreads" : 32767,
"minThreads" : 2
}
}
]
}
},
"duration in milliseconds" : 203.8816,
"name" : "Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler"
}
],
"duration in milliseconds" : 203.9043,
"name" : "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler"
}
],
"data" : {
"Client Configuration" : {
"Client Created Time Utc" : "2024-09-10T04:04:57.0562050Z",
"ConnectionConfig" : {
"gw" : "(cps:50, urto:120, p:False, httpf: False)",
"other" : "(ed:True, be:False)",
"rntbd" : "(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)"
},
"ConnectionMode" : "Gateway",
"ConsistencyConfig" : "(consistency: NotSet, prgns:[], apprgn: )",
"MachineId" : "vmId:69b1b058-5394-499f-a350-3837a3edeeef",
"NumberOfActiveClients" : 2,
"NumberOfClientsCreated" : 2,
"ProcessorCount" : 2,
"User Agent" : "cosmos-netstandard-sdk/3.42.0|2|X64|Ubuntu 22.04.4 LTS|.NET 8.0.7|DN| Microsoft.EntityFrameworkCore.Cosmos/8.0.8",
"VM Region" : "westus"
}
},
"duration in milliseconds" : 204.6975,
"name" : "CreateContainerStreamAsync",
"start datetime" : "2024-09-10T04:04:57.261Z"
} This is happening during |
The failure is unrelated to this PR, IIUC, so shouldn't block the merge. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
It would be good to hear from @Pilchie @kirankumarkolli on this change. |
My understanding is emulator AZURE_COSMOS_EMULATOR_PARTITION_COUNT impacts the concurrent #RUs supported. Checking with SME. |
@kirankumarkolli Any update on this? |
Apologies for delay. Recommendation is to use with /disableRateLimiting without it collections created with > 10K result in throttling (for devops it might not be an issue but good practice to set-up). |
@davidfowl do we want to use |
@kirankumarkolli What does |
Cosmos DB Request Units (RUs). 10,000 is the maximum for a single partition. |
Yes. Lets do it. |
This helps improve the container startup time for the emulator.
Note that log shows
n+1
partitions being started. For example withAZURE_COSMOS_EMULATOR_PARTITION_COUNT=1
the container logs show:Fixes #2177 .