Skip to content

Commit

Permalink
fix: limit max upload size (#224)
Browse files Browse the repository at this point in the history
Adds limit based on previous decisions, considering that S3 Put command
has hard-limit of 5GiB.

- By limiting CAR size to 127*(1<<25), we guarantee max-4GiB-padded
Filecoin pieces and have better utilization of Fil sector space.
- By receiving one more byte, we would immediatly get to 8GiB padded
piece.

Closes storacha/w3up#827
  • Loading branch information
vasco-santos authored Sep 13, 2023
1 parent ff92360 commit 1466f0d
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 11 deletions.
16 changes: 8 additions & 8 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion upload-api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"@web-std/fetch": "^4.1.0",
"@web3-storage/access": "^14.0.0",
"@web3-storage/capabilities": "^9.0.0",
"@web3-storage/upload-api": "^5.3.0",
"@web3-storage/upload-api": "^5.3.1",
"@web3-storage/w3infra-ucan-invocation": "*",
"multiformats": "^11.0.1",
"nanoid": "^4.0.2",
Expand Down
8 changes: 6 additions & 2 deletions upload-api/service.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ Sentry.AWSLambda.init({

export const createServiceRouter = UploadAPI.createService

export const MAX_S3_PUT_SIZE = 5_000_000_000
// S3 Put command has hard-limit of 5GiB.
// By limiting CAR size to 127*(1<<25), we guarantee max-4GiB-padded Filecoin pieces
// and have better utilization of Fil sector space.
// By receiving one more byte, we would immediatly get to 8GiB padded piece.
export const MAX_UPLOAD_SIZE = 127*(1<<25)

/**
* @param {import('@ucanto/interface').Signer} servicePrincipal
Expand All @@ -19,7 +23,7 @@ export const createUcantoServer = (servicePrincipal, context) =>
UploadAPI.createServer({
...context,
id: servicePrincipal,
maxUploadSize: MAX_S3_PUT_SIZE,
maxUploadSize: MAX_UPLOAD_SIZE,
errorReporter: {
catch: (/** @type {string | Error} */ err) => {
console.warn(err)
Expand Down

0 comments on commit 1466f0d

Please sign in to comment.