Skip to content

Commit

Permalink
feat: uploadCarWithStat avoids copying stat.carBytes via Blob constru…
Browse files Browse the repository at this point in the history
…ction (#2543)

… and instead wraps the stat.carBytes Uint8Array as a BlobLike

Context
* trying to make changes that help these code paths avoid the 'out of
memory' others saw when testing on staging

Motivation:
* removing this `new Blob` could get rid of a memory allocation that
could be contributing to 'out of memory'
* I think by passing a BlobLike to `uploadCAR` here and not a whole
blob, there will not be downstream copies of the underlying bytes
because this car BlobLike, once passed to `uploadCAR` will get passed to
a `CAR.BlockStream` that streams out blocks without re-buffering the
whole car bytes in memory
https://github.com/web3-storage/w3up/blob/main/packages/upload-client/src/index.js#L107
  • Loading branch information
gobengo authored Apr 1, 2024
1 parent bdde10a commit bcb67a3
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion packages/api/src/routes/nfts-upload.js
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,16 @@ export async function uploadCarWithStat(
// should only be 1 - shard size in w3up is > max upload size in CF
/** @type {import('@web3-storage/w3up-client/types').CARLink[]} */
const shards = []
await w3up.uploadCAR(new Blob([stat.carBytes]), {
const carBytesBlobLike = {
stream: () =>
new ReadableStream({
start(c) {
c.enqueue(stat.carBytes)
c.close()
},
}),
}
await w3up.uploadCAR(carBytesBlobLike, {
onShardStored: ({ cid }) => {
shards.push(cid)
},
Expand Down

0 comments on commit bcb67a3

Please sign in to comment.