-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unified protocol schedule - combine TimestampSchedule and MutableProtocolSchedule #5310
Conversation
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…erenceTests handling TODOs/comments tidy up now before going back to fixing tests as want to put on CI Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
|
|
||
import com.google.common.annotations.VisibleForTesting; | ||
|
||
public class UnifiedProtocolSchedule implements ProtocolSchedule { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Plan to rename this DefaultProtocolSchedule but think Unified helps with clarity for the first pass review
@Override | ||
public ProtocolSpec getByBlockHeader(final ProcessableBlockHeader blockHeader) { | ||
return this.timestampSchedule | ||
.getByTimestamp(blockHeader.getTimestamp()) | ||
.orElseGet( | ||
() -> | ||
transitionUtils.dispatchFunctionAccordingToMergeState( | ||
protocolSchedule -> protocolSchedule.getByBlockHeader(blockHeader))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getByTimestamp and getByBlockNumber are stitched together inside UnifiedProtocolSchedule now, so complexity is encapsulated in that one place.
public ProtocolSpec getByBlockNumber(final long number) { | ||
|
||
return Optional.ofNullable(protocolContext) | ||
.map(ProtocolContext::getBlockchain) | ||
.flatMap(blockchain -> blockchain.getBlockHeader(number)) | ||
.map(timestampSchedule::getByBlockHeader) | ||
.orElseGet( | ||
() -> | ||
transitionUtils.dispatchFunctionAccordingToMergeState( | ||
protocolSchedule -> protocolSchedule.getByBlockNumber(number))); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getByTimestamp and getByBlockNumber are stitched together inside UnifiedProtocolSchedule.getByBlockHeader now, so complexity is encapsulated in that one place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AbstractProtocolScheduleBuilder only has one subclass now, ProtocolScheduleBuilder but will merge them in subsequent PR to minimise this already rather large changeset :)
validateForkOrder("GrayGlacier", config.getGrayGlacierBlockNumber(), lastForkBlock); | ||
// Begin timestamp forks | ||
lastForkBlock = validateForkOrder("Shanghai", config.getShanghaiTime(), lastForkBlock); | ||
lastForkBlock = validateForkOrder("Cancun", config.getCancunTime(), lastForkBlock); | ||
lastForkBlock = validateForkOrder("FutureEips", config.getFutureEipsTime(), lastForkBlock); | ||
lastForkBlock = | ||
validateForkOrder("ExperimentalEips", config.getExperimentalEipsTime(), lastForkBlock); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
append timestamp forks to end of blockNumber forks
// Timestamp Forks | ||
createTimestampMilestone(config.getShanghaiTime(), specFactory.shanghaiDefinition(config)), | ||
createTimestampMilestone(config.getCancunTime(), specFactory.cancunDefinition(config)), | ||
createTimestampMilestone( | ||
config.getFutureEipsTime(), specFactory.futureEipsDefinition(config)), | ||
createTimestampMilestone( | ||
config.getExperimentalEipsTime(), specFactory.experimentalEipsDefinition(config)), | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
append timestamp forks to end of blockNumber forks.
Using createTimestampMilestone because the getBlockByHeader algorithm needs to know the difference between timestamp and blockNumber forks.
|
||
@Override | ||
public ProtocolSpec getByBlockHeader(final ProcessableBlockHeader blockHeader) { | ||
// checkArgument(number >= 0, "number must be non-negative"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think we can safely assume that a blockHeader will never have a negative number or timestamp
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's a reasonable assumption and the BlockHeaderBuilder
checks the number and timestamp are >= 0
// TODO SLD - do we need this or a getByBlockHeader equivalent? Could be why referenceTests are | ||
// failing? | ||
// @Override | ||
// public ProtocolSpec getByBlockNumber(final long number) { | ||
// final ProtocolSpec original = delegate.getByBlockNumber(number); | ||
// final BlockProcessor noRewardBlockProcessor = | ||
// new MainnetBlockProcessor( | ||
// original.getTransactionProcessor(), | ||
// original.getTransactionReceiptFactory(), | ||
// Wei.ZERO, | ||
// original.getMiningBeneficiaryCalculator(), | ||
// original.isSkipZeroBlockRewards(), | ||
// Optional.empty(), | ||
// delegate); | ||
// final BlockValidator noRewardBlockValidator = | ||
// new MainnetBlockValidator( | ||
// original.getBlockHeaderValidator(), | ||
// original.getBlockBodyValidator(), | ||
// noRewardBlockProcessor, | ||
// original.getBadBlocksManager()); | ||
// final BlockImporter noRewardBlockImporter = new | ||
// MainnetBlockImporter(noRewardBlockValidator); | ||
// return new ProtocolSpec( | ||
// original.getName(), | ||
// original.getEvm(), | ||
// original.getTransactionValidator(), | ||
// original.getTransactionProcessor(), | ||
// original.getPrivateTransactionProcessor(), | ||
// original.getBlockHeaderValidator(), | ||
// original.getOmmerHeaderValidator(), | ||
// original.getBlockBodyValidator(), | ||
// noRewardBlockProcessor, | ||
// noRewardBlockImporter, | ||
// noRewardBlockValidator, | ||
// original.getBlockHeaderFunctions(), | ||
// original.getTransactionReceiptFactory(), | ||
// original.getDifficultyCalculator(), | ||
// Wei.ZERO, // block reward | ||
// original.getMiningBeneficiaryCalculator(), | ||
// original.getPrecompileContractRegistry(), | ||
// original.isSkipZeroBlockRewards(), | ||
// original.getGasCalculator(), | ||
// original.getGasLimitCalculator(), | ||
// original.getFeeMarket(), | ||
// original.getBadBlocksManager(), | ||
// Optional.empty(), | ||
// original.getWithdrawalsValidator(), | ||
// original.getWithdrawalsProcessor(), | ||
// original.getDepositsValidator(), | ||
// original.isPoS()); | ||
|
||
// } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need this or a getByBlockHeader equivalent? Could be why referenceTests are failing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@shemnon I've essentially migrated getBlockByNumber to getBlockByHeader.
Do you know if the logic that I've commented out in here is actually used?
It's not covered by tests AFAICT, but I think it might be used if retesteth is run with certain inputs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will break retesteth and we won't be able to fill reference tests.
retesteth needs to have some blocks evaluated "without block reward" and this is how that feature is wired in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I can just migrate this code into the getByBlockHeader method above then. I think the break has already occurred during this commit actually, since that implementation of getBlockByNumber is no longer used: 62f4a51
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC this was the beyoncé rule one 😉 😆
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've just migrated it here, hope this makes sense! b459bc3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's the beyoncé rule??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
“If you liked it, then you shoulda put a test on it.”
https://abseil.io/resources/swe-book/html/ch11.html#:~:text=The%20Beyonc%C3%A9%20Rule&text=The%20straightforward%20answer%20is%3A%20test,an%20automated%20test%20for%20it.
String listMilestones(); | ||
|
||
void putMilestone( | ||
final boolean isTimestampMilestone, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would rather find a more encapsulated way to deal with this, ideally private to UnifiedProtocolSchedule (only used by getByBlockHeader). It does make UnifiedProtocolSchedule.getByBlockHeader implementation quite neat though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could move this into another interface like what was done with the mutable methods in the current implementation at least then it would only be exposed for the few places it's needed like the protocol schedule builder.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…hedule Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@shemnon There is a single referenceTest file that is breaking: Line 51 in 1665eb7
Now that I've combined the timestamp milestones into the same collection as the blockNumber milestone, there's a difficulty mismatch because the protocolSchedule retrieves Shanghai instead of the appropriate blockNumber milestone. This is because the test inputs have timestamps far into the future so it always matches Shanghai. For example, the first entry:
The timestamp Any thoughts about how we could fix this? |
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…hedule Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
This reverts commit 861ab1d. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@shemnon I've opened an issue for now ethereum/tests#1207 but is it possible/appropriate to temporarily disable these while we wait so it doesn't block the PR? |
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
public ProtocolSpec getByBlockHeader(final ProcessableBlockHeader blockHeader) { | ||
// TODO SLD to remove once it's been discussed in PR review | ||
// checkArgument(number >= 0, "number must be non-negative"); | ||
checkArgument( | ||
!protocolSpecs.isEmpty(), "At least 1 milestone must be provided to the protocol schedule"); | ||
checkArgument( | ||
protocolSpecs.last().milestone() == 0, "There must be a milestone starting from block 0"); | ||
|
||
// protocolSpecs is sorted in descending block order, so the first one we find that's lower than | ||
// the requested level will be the most appropriate spec | ||
for (final ScheduledProtocolSpec scheduledProtocolSpec : protocolSpecs) { | ||
if (scheduledProtocolSpec.isTimestampMilestone()) { | ||
if (blockHeader.getTimestamp() >= scheduledProtocolSpec.milestone()) { | ||
return scheduledProtocolSpec.spec(); | ||
} | ||
} else { | ||
if (blockHeader.getNumber() >= scheduledProtocolSpec.milestone()) { | ||
return scheduledProtocolSpec.spec(); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the most important change and represents a change to the algorithm for retrieving the protocol schedule: it still uses the original MutableScheduleProtocol.getByBlockNumber algorithm (running through the schedule in reverse order and returning as soon as our input milestone is beyond a spec), however it checks the timestamp if it's a timestampFork or else it checks the blockNumber.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the original MutableProtocolSchedule.getByBlockNumber algorithm for comparison:
@Override
public ProtocolSpec getByBlockNumber(final long number) {
checkArgument(number >= 0, "number must be non-negative");
checkArgument(
!protocolSpecs.isEmpty(), "At least 1 milestone must be provided to the protocol schedule");
checkArgument(
protocolSpecs.last().milestone() == 0, "There must be a milestone starting from block 0");
// protocolSpecs is sorted in descending block order, so the first one we find that's lower than
// the requested level will be the most appropriate spec
for (final ScheduledProtocolSpec s : protocolSpecs) {
if (number >= s.milestone()) {
return s.spec();
}
}
return null;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: you could make this closer to the original and remove a little bit of dupe by doing something like
for (final ScheduledProtocolSpec scheduledProtocolSpec : protocolSpecs) {
final long milestone =
scheduledProtocolSpec.isTimestampMilestone()
? blockHeader.getTimestamp()
: blockHeader.getNumber();
if (milestone >= scheduledProtocolSpec.milestone()) {
return scheduledProtocolSpec.spec();
}
}
Looks to like the tests for difficulty are not being maintained. In the interrum we can change the shanghai timstamp in the mainnet config
|
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…steth NoReward Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a few comments
.../src/main/java/org/hyperledger/besu/consensus/common/bft/BaseBftProtocolScheduleBuilder.java
Outdated
Show resolved
Hide resolved
...erledger/besu/ethereum/api/jsonrpc/internal/methods/engine/WithdrawalsValidatorProvider.java
Show resolved
Hide resolved
...erledger/besu/ethereum/api/jsonrpc/internal/methods/engine/WithdrawalsValidatorProvider.java
Show resolved
Hide resolved
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/ProtocolSchedule.java
Outdated
Show resolved
Hide resolved
...um/core/src/test/java/org/hyperledger/besu/ethereum/mainnet/UnifiedProtocolScheduleTest.java
Outdated
Show resolved
Hide resolved
...um/core/src/test/java/org/hyperledger/besu/ethereum/mainnet/UnifiedProtocolScheduleTest.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…hedule Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…net/UnifiedProtocolScheduleTest.java Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…net/UnifiedProtocolScheduleTest.java Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Instead of applying mergeSpecificModifications to every post-merge fork, rely on what's configured in the MainnetProtocolsSpec. The exception being MergeBlockProcessor which still needs wiring in as modification due to circular dependency. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…hedule Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/ProtocolSchedule.java
Show resolved
Hide resolved
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/ProtocolSchedule.java
Outdated
Show resolved
Hide resolved
|
||
@Override | ||
public ProtocolSpec getByBlockHeader(final ProcessableBlockHeader blockHeader) { | ||
// checkArgument(number >= 0, "number must be non-negative"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's a reasonable assumption and the BlockHeaderBuilder
checks the number and timestamp are >= 0
consensus/merge/src/main/java/org/hyperledger/besu/consensus/merge/MergeProtocolSchedule.java
Show resolved
Hide resolved
String listMilestones(); | ||
|
||
void putMilestone( | ||
final boolean isTimestampMilestone, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could move this into another interface like what was done with the mutable methods in the current implementation at least then it would only be exposed for the few places it's needed like the protocol schedule builder.
String listMilestones(); | ||
|
||
void putMilestone( | ||
final boolean isTimestampMilestone, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
really don't like this boolean, maybe enum would make it clearer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used inheritance instead as discussed 816137a
public ProtocolSpec getByBlockHeader(final ProcessableBlockHeader blockHeader) { | ||
// TODO SLD to remove once it's been discussed in PR review | ||
// checkArgument(number >= 0, "number must be non-negative"); | ||
checkArgument( | ||
!protocolSpecs.isEmpty(), "At least 1 milestone must be provided to the protocol schedule"); | ||
checkArgument( | ||
protocolSpecs.last().milestone() == 0, "There must be a milestone starting from block 0"); | ||
|
||
// protocolSpecs is sorted in descending block order, so the first one we find that's lower than | ||
// the requested level will be the most appropriate spec | ||
for (final ScheduledProtocolSpec scheduledProtocolSpec : protocolSpecs) { | ||
if (scheduledProtocolSpec.isTimestampMilestone()) { | ||
if (blockHeader.getTimestamp() >= scheduledProtocolSpec.milestone()) { | ||
return scheduledProtocolSpec.spec(); | ||
} | ||
} else { | ||
if (blockHeader.getNumber() >= scheduledProtocolSpec.milestone()) { | ||
return scheduledProtocolSpec.spec(); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: you could make this closer to the original and remove a little bit of dupe by doing something like
for (final ScheduledProtocolSpec scheduledProtocolSpec : protocolSpecs) {
final long milestone =
scheduledProtocolSpec.isTimestampMilestone()
? blockHeader.getTimestamp()
: blockHeader.getNumber();
if (milestone >= scheduledProtocolSpec.milestone()) {
return scheduledProtocolSpec.spec();
}
}
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Move timestamp or blockNumber complexity into ScheduledProtocolSpec subclasses Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…hedule Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -107,8 +107,7 @@ public Map<String, JsonRpcMethod> methods( | |||
blockchainQueries, protocolSchedule, transactionPool, privacyParameters), | |||
new ExecutionEngineJsonRpcMethods( | |||
miningCoordinator, | |||
MergeProtocolSchedule.createTimestamp( | |||
genesisConfigOptions, privacyParameters, false), | |||
MergeProtocolSchedule.create(genesisConfigOptions, privacyParameters, false), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that the MergeProtocolShedule already contains the timestamp forks this creation might not be needed anymore as discussed. Might be useful to try use the existing one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created a task on the other PR #5354
…ocolSchedule (hyperledger#5310) - Combine MutableProtocolSchedule and DefaultTimestampSchedule into UnifiedProtocolSchedule. - Implement getByBlockHeader taking into account both timestamp-based and blockNumber-based forks - Unstitch timestampSchedule from TransitionProtocolSchedule - BftProtocolSchedule extends UnifiedProtocolSchedule (instead of MutableProtocolSchedule) - In MergeProtocolSchedule, unapply mergeSpecificModifications from Shanghai onwards - Migrate getByBlockNumber modifications into getByBlockHeader for retesteth NoReward - Temporarily fix reference tests by artificially increasing shanghaiTime Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
…ocolSchedule (hyperledger#5310) - Combine MutableProtocolSchedule and DefaultTimestampSchedule into UnifiedProtocolSchedule. - Implement getByBlockHeader taking into account both timestamp-based and blockNumber-based forks - Unstitch timestampSchedule from TransitionProtocolSchedule - BftProtocolSchedule extends UnifiedProtocolSchedule (instead of MutableProtocolSchedule) - In MergeProtocolSchedule, unapply mergeSpecificModifications from Shanghai onwards - Migrate getByBlockNumber modifications into getByBlockHeader for retesteth NoReward - Temporarily fix reference tests by artificially increasing shanghaiTime Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
This is a necessarily large PR, but it's mostly type removal/renaming/unification changes.
I have added comments to highlight the interesting parts that require closer review.
Currently, one reference test file breaking due to invalid test inputs see #5310 (comment)PR description
UnifiedProtocolSchedule
.Optimise getByBlockHeader algorithm (once all tests are passing)Follow up PR will cover the more minor tidy up changes left out of this PR for brevity: #5354
Should end up with this hierarchy:
Fixed Issue(s)
#5260
Testing
Full sync goerliissues with full goerli syncEnsure EVM tool workssee conv with Danno: https://discord.com/channels/905194001349627914/943252457063075880/1098464329117995038Performance test? Since getByBlockHeader algorithm has significantly changed.Refactored algorithm is very similar to original now.