Skip to content

Commit

Permalink
fix: memory usage (#1227)
Browse files Browse the repository at this point in the history
Previously `parallel-transform-web` was running up to N transform tasks
concurrently, but while those tasks were running it buffered the rest of
the stream data internally 🤦 .

This was resolved in
alanshaw/parallel-transform-web#1 so just
pulling in the latest code here.

I was seeing memory usage in w3cli for a 12GB file grow to ~10GB but now
it stays at around at round ~4GB (which is the same usage I see when I
remove `parallel-transform-web` and replace it with a regular serial
`TransformStream`).
  • Loading branch information
Alan Shaw authored Dec 7, 2023
1 parent e5c86e0 commit c9e7a49
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 7 deletions.
4 changes: 2 additions & 2 deletions packages/upload-client/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -78,16 +78,16 @@
"ipfs-utils": "^9.0.14",
"multiformats": "^12.1.2",
"p-retry": "^5.1.2",
"parallel-transform-web": "^1.0.0",
"parallel-transform-web": "^1.0.1",
"varint": "^6.0.0"
},
"devDependencies": {
"@types/assert": "^1.5.6",
"@types/mocha": "^10.0.1",
"@types/varint": "^6.0.1",
"@web3-storage/eslint-config-w3up": "workspace:^",
"@ucanto/principal": "^9.0.0",
"@ucanto/server": "^9.0.1",
"@web3-storage/eslint-config-w3up": "workspace:^",
"assert": "^2.0.0",
"blockstore-core": "^3.0.0",
"c8": "^7.13.0",
Expand Down
10 changes: 5 additions & 5 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit c9e7a49

Please sign in to comment.