Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve newPayload and FCU logs #7961

Merged
merged 34 commits into from
Dec 5, 2024

Conversation

ahamlat
Copy link
Contributor

@ahamlat ahamlat commented Nov 29, 2024

PR description

This PR improves the default output for newPayload and FCU calls within the Engine API. The goal is to provide a matrix-like output format, making it easier to identify and analyze the relevant information at a glance. Also, we’ve included new log details, such as mgas/s and the percentage of parallelized transactions, which are displayed when Xbonsai-parallel-tx-processing-enabled is active.

The default output before this PR

2024-11-29T13:56:49,791 : VALID for fork-choice-update: head: 0x68976b7a9aad87107d3449e485b683692b186356bd753da109e68fd77eec9662, finalized: 0xfdf5ee580708a39cb753825e2296918f9199b00f59e03b5fd4c1d7550c870044, safeBlockHash: 0x5913a61f68d5fb31d223e96eacce18c02c1ba5cd93e1a47c9d880c2bac6b2517 
2024-11-29T13:57:00,946 : Imported #21,293,789 / 212 tx / 16 ws / 2 blobs / base fee 13.78 gwei / 19,512,289 (65.0%) gas / (0xdceca4e7b6f07886245b625d415fa3e8fe2d4f8cee2ae9cc8d9e2c5b530503fa) in 0.216s. Peers: 25 
2024-11-29T13:57:12,520 : Imported #21,293,790 / 220 tx / 16 ws / 1 blobs / base fee 14.29 gwei / 18,591,986 (62.0%) gas / (0xe3f16161ceacd20f36a324779f79daab8472008400f670b3b1c3d930b534aec1) in 0.197s. Peers: 25 
2024-11-29T13:57:24,794 : Imported #21,293,791 / 185 tx / 16 ws / 0 blobs / base fee 14.72 gwei / 14,585,670 (48.6%) gas / (0xd2521da4ee26b49dbc198e7397fb96f6468fabbdedb8dbb5a4237cea84ad1506) in 0.158s. Peers: 25 
2024-11-29T13:57:37,120 : Imported #21,293,792 / 213 tx / 16 ws / 0 blobs / base fee 14.67 gwei / 16,959,617 (56.5%) gas / (0xb0277d881486c2b33aab94117dec073c7864b2e01db2b5108bfdd6c0ebf1c070) in 0.221s. Peers: 25 
2024-11-29T13:57:49,219 : Imported #21,293,793 / 200 tx / 16 ws / 0 blobs / base fee 14.91 gwei / 13,948,112 (46.5%) gas / (0x9d0f78304138fc730da2ca314aa0795e04e794c913888fb73e5df08482703be8) in 0.145s. Peers: 25 
2024-11-29T13:58:00,693 : Imported #21,293,794 / 177 tx / 16 ws / 6 blobs / base fee 14.78 gwei / 11,173,858 (37.2%) gas / (0x3d03679bf65eea53cc813021c2a0718e0682ed66a32b248a824cd310bb221b27) in 0.117s. Peers: 25 
2024-11-29T13:58:00,960 : VALID for fork-choice-update: head: 0x3d03679bf65eea53cc813021c2a0718e0682ed66a32b248a824cd310bb221b27, finalized: 0xfdf5ee580708a39cb753825e2296918f9199b00f59e03b5fd4c1d7550c870044, safeBlockHash: 0x5913a61f68d5fb31d223e96eacce18c02c1ba5cd93e1a47c9d880c2bac6b2517 
2024-11-29T13:58:13,429 : Imported #21,293,795 / 293 tx / 16 ws / 5 blobs / base fee 14.31 gwei / 29,983,687 (99.9%) gas / (0xb2f425aa03e7c5b31f8019823896923ae017b0a76ee349a4e28e8cf425646164) in 0.451s. Peers: 25 
2024-11-29T13:58:24,601 : Imported #21,293,796 / 219 tx / 16 ws / 0 blobs / base fee 16.09 gwei / 15,268,772 (50.9%) gas / (0xe9162cb4be04dd15b8aad9fde24ddf7d7579845b1531697aa47a7e4e16a9ff13) in 0.172s. Peers: 25 
2024-11-29T13:58:36,822 : Imported #21,293,797 / 173 tx / 16 ws / 4 blobs / base fee 16.13 gwei / 11,657,572 (38.9%) gas / (0x0ec84d09d4833dcad161e421d70988d53e1f94bc951084b15e1679a0e1eb19ce) in 0.153s. Peers: 25 
2024-11-29T13:58:48,582 : Imported #21,293,798 / 139 tx / 16 ws / 5 blobs / base fee 15.68 gwei / 8,934,463 (29.8%) gas / (0x02146c3b17981ebe7d20fba50c23b6cf03674d47e4b7379f5aad37f728014913) in 0.081s. Peers: 25 
2024-11-29T13:59:01,275 : Imported #21,293,799 / 128 tx / 16 ws / 6 blobs / base fee 14.89 gwei / 8,913,285 (29.7%) gas / (0x357af341b3187e2b7218dfa5bcb1fe3966c0c45f1841924ebc817cb9e08b4332) in 0.099s. Peers: 25 
2024-11-29T13:59:01,515 : VALID for fork-choice-update: head: 0x357af341b3187e2b7218dfa5bcb1fe3966c0c45f1841924ebc817cb9e08b4332, finalized: 0xfdf5ee580708a39cb753825e2296918f9199b00f59e03b5fd4c1d7550c870044, safeBlockHash: 0x5913a61f68d5fb31d223e96eacce18c02c1ba5cd93e1a47c9d880c2bac6b2517 

With this PR (we can notice that each part of the log is padded to avoid having differences between the lines)

2024-11-29T13:56:49,455 : Imported #21,293,788  (0x6897.....c9662)|  215 tx| 16 ws| 6 blobs|base fee  13.56 gwei|gas used  16,914,471 ( 56.4%)|exec time 0.223s|mgas/s  75.85|Parallel txs  34.9%|peers: 25 
2024-11-29T13:56:49,756 : FCU(VALID) | head: 0x6897.....c9662 | finalized: 0xfdf5.....70044 | safeBlockHash: 0x5913.....b2517 
2024-11-29T13:57:00,972 : Imported #21,293,789  (0xdcec.....503fa)|  212 tx| 16 ws| 2 blobs|base fee  13.78 gwei|gas used  19,512,289 ( 65.0%)|exec time 0.249s|mgas/s  78.36|Parallel txs  36.8%|peers: 25 
2024-11-29T13:57:12,513 : Imported #21,293,790  (0xe3f1.....4aec1)|  220 tx| 16 ws| 1 blobs|base fee  14.29 gwei|gas used  18,591,986 ( 62.0%)|exec time 0.194s|mgas/s  95.83|Parallel txs  35.0%|peers: 25 
2024-11-29T13:57:24,855 : Imported #21,293,791  (0xd252.....d1506)|  185 tx| 16 ws| 0 blobs|base fee  14.72 gwei|gas used  14,585,670 ( 48.6%)|exec time 0.150s|mgas/s  97.24|Parallel txs  31.4%|peers: 25 
2024-11-29T13:57:37,077 : Imported #21,293,792  (0xb027.....1c070)|  213 tx| 16 ws| 0 blobs|base fee  14.67 gwei|gas used  16,959,617 ( 56.5%)|exec time 0.167s|mgas/s 101.55|Parallel txs  39.9%|peers: 25 
2024-11-29T13:57:49,281 : Imported #21,293,793  (0x9d0f.....03be8)|  200 tx| 16 ws| 0 blobs|base fee  14.91 gwei|gas used  13,948,112 ( 46.5%)|exec time 0.128s|mgas/s 108.97|Parallel txs  36.5%|peers: 25 
2024-11-29T13:58:00,567 : Imported #21,293,794  (0x3d03.....21b27)|  177 tx| 16 ws| 6 blobs|base fee  14.78 gwei|gas used  11,173,858 ( 37.2%)|exec time 0.103s|mgas/s 108.48|Parallel txs  35.0%|peers: 25 
2024-11-29T13:58:00,880 : FCU(VALID) | head: 0x3d03.....21b27 | finalized: 0xfdf5.....70044 | safeBlockHash: 0x5913.....b2517 
2024-11-29T13:58:13,211 : Imported #21,293,795  (0xb2f4.....46164)|  293 tx| 16 ws| 5 blobs|base fee  14.31 gwei|gas used  29,983,687 ( 99.9%)|exec time 0.309s|mgas/s  97.03|Parallel txs  27.0%|peers: 25 
2024-11-29T13:58:24,642 : Imported #21,293,796  (0xe916.....9ff13)|  219 tx| 16 ws| 0 blobs|base fee  16.09 gwei|gas used  15,268,772 ( 50.9%)|exec time 0.162s|mgas/s  94.25|Parallel txs  37.9%|peers: 25 
2024-11-29T13:58:36,773 : Imported #21,293,797  (0x0ec8.....b19ce)|  173 tx| 16 ws| 4 blobs|base fee  16.13 gwei|gas used  11,657,572 ( 38.9%)|exec time 0.141s|mgas/s  82.68|Parallel txs  41.0%|peers: 25 
2024-11-29T13:58:48,505 : Imported #21,293,798  (0x0214.....14913)|  139 tx| 16 ws| 5 blobs|base fee  15.68 gwei|gas used   8,934,463 ( 29.8%)|exec time 0.076s|mgas/s 117.56|Parallel txs  40.3%|peers: 25 
2024-11-29T13:59:01,106 : Imported #21,293,799  (0x357a.....b4332)|  128 tx| 16 ws| 6 blobs|base fee  14.89 gwei|gas used   8,913,285 ( 29.7%)|exec time 0.090s|mgas/s  99.04|Parallel txs  39.8%|peers: 25 
2024-11-29T13:59:01,343 : FCU(VALID) | head: 0x357a.....b4332 | finalized: 0xfdf5.....70044 | safeBlockHash: 0x5913.....b2517 

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests

@ahamlat ahamlat force-pushed the change-log-newpayload branch from c160af3 to 09cb8da Compare November 29, 2024 14:05
@ahamlat ahamlat marked this pull request as ready for review November 29, 2024 14:09
message.append(" / %d blobs / base fee %s / %,d (%01.1f%%) gas / (%s) in %01.3fs. Peers: %d");
double mgasPerSec = (timeInS != 0) ? block.getHeader().getGasUsed() / (timeInS * 1_000_000) : 0;
message.append(
"|%2d blobs|base fee %s|gas used %,11d (%5.1f%%)|exec time %01.3fs|mgas/s %6.2f");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's inconsistent spaces/no spaces between the pipes, I'd say with spaces are easier to read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inconsistencies related to space come from the padding mecanism, which depends on the represented data. I can add a space for cases that don't start with a number.

forkChoice.getHeadBlockHash(),
forkChoice.getFinalizedBlockHash(),
forkChoice.getSafeBlockHash());
AbstractEngineNewPayload.shortenString(forkChoice.getHeadBlockHash().toHexString()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment about making etherscan debugging more difficult, less important for FCU though I think since can cross ref with Imported log

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as for newPayload call.

ahamlat and others added 25 commits December 2, 2024 14:14
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
* Upgrade to Promethus java client 1.x and adapt the code to the new version

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* Update CHANGELOG.md

Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

---------

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
…7953)

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
…erledger#7956)

* fix TransactionLocation in DefaultBlockchain unsafeImportBlock() and make some readability improvements

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
* fast sync log warnings for deprecation

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* minor text change in changelog

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: daniellehrner <daniel.lehrner@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
…sonrpc/internal/methods/engine/AbstractEngineNewPayload.java

Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: ahamlat <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
@ahamlat ahamlat force-pushed the change-log-newpayload branch from 4d06966 to 01cf738 Compare December 2, 2024 13:14
Comment on lines +204 to +206
public void setIsProcessedInParallel(final Optional<Boolean> isProcessedInParallel) {
this.isProcessedInParallel = isProcessedInParallel;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we avoid to make this class mutable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that currently, when a transaction executes, it doesn't know itself if it si being executed in parallel or not, so the instance of TransactionProcessingResult is not aware if it was parallel or not. This information is only accessible later on. that's why we need to make the result mutable at this level only.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I see, fine for me, but maybe this is a sign that some refactor is needed, to postponed the creation of the result, of the data should be kept elsewhere

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Comment on lines +204 to +206
public void setIsProcessedInParallel(final Optional<Boolean> isProcessedInParallel) {
this.isProcessedInParallel = isProcessedInParallel;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I see, fine for me, but maybe this is a sign that some refactor is needed, to postponed the creation of the result, of the data should be kept elsewhere

Copy link
Contributor

@fab-10 fab-10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
@ahamlat ahamlat enabled auto-merge (squash) December 5, 2024 12:49
@ahamlat ahamlat merged commit d595eee into hyperledger:main Dec 5, 2024
43 checks passed
daniellehrner pushed a commit to daniellehrner/besu that referenced this pull request Dec 18, 2024
* Change the output log on newPayload and FCU executions

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants