Skip to content

Commit

Permalink
DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY now can be set by env
Browse files Browse the repository at this point in the history
  • Loading branch information
dkershner6 committed Oct 23, 2023
1 parent 4764fc1 commit 6cbbca3
Show file tree
Hide file tree
Showing 3 changed files with 35 additions and 8 deletions.
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ AWS link: https://d1gwt3w78t4dm3.cloudfront.net

Vercel link: https://open-next.vercel.app

## Configuration

### Environment variables

- DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY: The number of concurrent batch write commands to DynamoDB. Defaults to 4 in an effort to leave plenty of DynamoDB write request capacity for the production load.

## Contribute

To run `OpenNext` locally:
Expand Down
24 changes: 24 additions & 0 deletions packages/open-next/src/adapters/constants.ts
Original file line number Diff line number Diff line change
@@ -1 +1,25 @@
export const MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT = 25;

/**
* Sending to dynamo X commands at a time, using about X * 25 write units per batch to not overwhelm DDB
* and give production plenty of room to work with. With DDB Response times, you can expect about 10 batches per second.
*/
const DEFAULT_DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY = 4;

export const getDynamoBatchWriteCommandConcurrency = (): number => {
const dynamoBatchWriteCommandConcurrencyFromEnv =
process.env.DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY;
const parsedDynamoBatchWriteCommandConcurrencyFromEnv =
dynamoBatchWriteCommandConcurrencyFromEnv
? parseInt(dynamoBatchWriteCommandConcurrencyFromEnv)
: undefined;

if (
parsedDynamoBatchWriteCommandConcurrencyFromEnv &&
!isNaN(parsedDynamoBatchWriteCommandConcurrencyFromEnv)
) {
return parsedDynamoBatchWriteCommandConcurrencyFromEnv;
}

return DEFAULT_DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY;
};
13 changes: 5 additions & 8 deletions packages/open-next/src/adapters/dynamo-provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ import {
import { CdkCustomResourceEvent, CdkCustomResourceResponse } from "aws-lambda";
import { readFileSync } from "fs";

import { MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT } from "./constants.js";
import {
getDynamoBatchWriteCommandConcurrency,
MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT,
} from "./constants.js";
import { chunk } from "./util.js";

const PHYSICAL_RESOURCE_ID = "dynamodb-cache";
Expand Down Expand Up @@ -36,12 +39,6 @@ export async function handler(
}
}

/**
* Sending to dynamo X commands at a time, using about X * 25 write units per batch to not overwhelm DDB
* and give production plenty of room to work with. With DDB Response times, you can expect about 10 batches per second.
*/
const DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY = 4;

async function insert(): Promise<CdkCustomResourceResponse> {
const tableName = process.env.CACHE_DYNAMO_TABLE!;

Expand All @@ -65,7 +62,7 @@ async function insert(): Promise<CdkCustomResourceResponse> {

const paramsChunks = chunk(
batchWriteParamsArray,
DYNAMO_BATCH_WRITE_COMMAND_CONCURRENCY,
getDynamoBatchWriteCommandConcurrency(),
);

for (const paramsChunk of paramsChunks) {
Expand Down

0 comments on commit 6cbbca3

Please sign in to comment.