Pipeline chain download - fetch and import data #1207
Conversation
…-download-fetch-data
…-download-fetch-data
…-download-fetch-data
| @@ -189,7 +189,7 @@ public PipelineBuilder( | |||
| pipeEnd, | |||
| maximumBatchSize, | |||
| outputCounter.labels(lastStageName + "_outputPipe", "batches")), | |||
| bufferSize / maximumBatchSize + 1, | |||
| (int) Math.ceil(((double) bufferSize) / maximumBatchSize), | |||
There was a problem hiding this comment.
Fixes the calculation so that a pipeline with buffer size 800, creating batches of 200 will hold 4 batches in the buffers (max 800 items in total). The previous integer calculation would wind up buffering up to 5 batches.
| @@ -59,4 +59,8 @@ public String toString() { | |||
| .add("end", end.getNumber()) | |||
| .toString(); | |||
| } | |||
|
|
|||
| public int getSegmentLength() { | |||
| return (int) (end.getNumber() - start.getNumber()); | |||
There was a problem hiding this comment.
As a general rule for cases like this, should we use Math.toIntExact or just use the simpler casting since we're not expecting large numbers?
There was a problem hiding this comment.
This case is safe because the gap between checkpoint headers is originally specified as an int but having the explicit check probably makes sense anyway. So I've changed it to use toIntExact.
| @@ -46,6 +46,10 @@ public void runPendingFutures() { | |||
| currentTasks.forEach(ExecutorTask::run); | |||
| } | |||
|
|
|||
| public int getPendingFuturesCount() { | |||
| return tasks.size(); | |||
There was a problem hiding this comment.
You need to filter this by ExecutorTask::isPending or change the method name to countFutures
| @@ -59,4 +59,8 @@ public String toString() { | |||
| .add("end", end.getNumber()) | |||
| .toString(); | |||
| } | |||
|
|
|||
| public int getSegmentLength() { | |||
There was a problem hiding this comment.
(optional) If you change this to getSegmentLengthExclusive you can avoid the - 1 below: https://github.com/PegaSysEng/pantheon/pull/1207/files#diff-e42d52272bdde3b38192060ce222bebbR66
There was a problem hiding this comment.
Tempting but it's only exclusive of one end, not both so could wind up being confusing.
There was a problem hiding this comment.
If you -1, its exclusive of both ends :) [0,1] : 1 - 0 - 1 = 0
There was a problem hiding this comment.
🤦♂️ I shouldn't try to do math so early in the morning...
…-download-fetch-data
…-download-fetch-data
…-download-fetch-data
PR description
Implements the pipeline steps to download headers, bodies and transaction receipts then fast import blocks.
Note that the pipeline is currently using
thenProcessAsyncwhich will need to be replaced withthenProcessAsyncPreservingOrderonce it's available.Toggle is still set to use the old chain download process.