Skip to content

Commit

Permalink
fix: storefront events cron with max concurrency (storacha#1191)
Browse files Browse the repository at this point in the history
Running batches of concurrency 20 in my machine handled in 2.40s a batch
of updates. A page query of `4046` in parallel trying to crawl the
entire chain was being too much and mostly the CRON would timeout before
things getting updated, causing cron to be struggling to move on
  • Loading branch information
vasco-santos authored Nov 28, 2023
1 parent 97a7def commit 11010c9
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 6 deletions.
3 changes: 2 additions & 1 deletion packages/filecoin-api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,8 @@
"@ucanto/server": "^9.0.1",
"@ucanto/transport": "^9.0.0",
"@web3-storage/capabilities": "workspace:^",
"@web3-storage/data-segment": "^4.0.0"
"@web3-storage/data-segment": "^4.0.0",
"p-map": "^6.0.0"
},
"devDependencies": {
"@ipld/car": "^5.1.1",
Expand Down
11 changes: 8 additions & 3 deletions packages/filecoin-api/src/storefront/events.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import pMap from 'p-map'
import { Storefront, Aggregator } from '@web3-storage/filecoin-client'
import * as AggregatorCaps from '@web3-storage/capabilities/filecoin/aggregator'

Expand Down Expand Up @@ -137,8 +138,9 @@ export const handleCronTick = async (context) => {
}
}
// Update approved pieces from the ones resolved
const updatedResponses = await Promise.all(
submittedPieces.ok.map((pieceRecord) =>
const updatedResponses = await pMap(
submittedPieces.ok,
(pieceRecord) =>
updatePiecesWithDeal({
id: context.id,
aggregatorId: context.aggregatorId,
Expand All @@ -147,7 +149,10 @@ export const handleCronTick = async (context) => {
taskStore: context.taskStore,
receiptStore: context.receiptStore,
})
)
,
{
concurrency: 20
}
)

// Fail if one or more update operations did not succeed.
Expand Down
6 changes: 4 additions & 2 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 11010c9

Please sign in to comment.