Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Baseline interpreter #261

Merged
merged 20 commits into from
Dec 21, 2020
Merged

Baseline interpreter #261

merged 20 commits into from
Dec 21, 2020

Conversation

chfast
Copy link
Member

@chfast chfast commented Oct 15, 2020

The "baseline" is a new interpreter added to evmone. It has much simpler, "classic" design and performs minimal amount of code analysis (only required JUMPDEST analysis, the code is unmodified). EVM instructions implementations are shared with "advanced" VM where possible.

To use it pass O=0 option.

The initial benchmarking results look promising (see below). All done with Clang-11 compiler. The GCC-10 has lower performance, but I believe this can be fixed.
Moreover, the first PoC has some additional changes and tweaks (e.g. custom instruction table) and is up to 5% faster than this version. The old version is still available in baseline-old branch.

Solidity test suite

For solidity test suite it performs a tinny bit better.

 Performance counter stats for 'test/soltest -t ABIDecoderTest,ABIEncoderTest,SolidityAuctionRegistrar,SolidityFixedFeeRegistrar,SolidityWallet,GasMeterTests,GasCostTests,SolidityEndToEndTest,SolidityOptimizer -- 
--vm ./libevmone.so,O=0' (10 runs):

         20 061,36 msec task-clock                #    1,000 CPUs utilized            ( +-  0,07% )
                67      context-switches          #    0,003 K/sec                    ( +-  6,90% )
                 0      cpu-migrations            #    0,000 K/sec                    ( +- 50,92% )
            10 325      page-faults               #    0,515 K/sec                    ( +-  0,01% )
    88 233 333 889      cycles                    #    4,398 GHz                      ( +-  0,07% )
   132 744 158 912      instructions              #    1,50  insn per cycle           ( +-  0,00% )
    21 297 391 944      branches                  # 1061,613 M/sec                    ( +-  0,00% )
       263 104 779      branch-misses             #    1,24% of all branches          ( +-  0,02% )

           20,0619 +- 0,0135 seconds time elapsed  ( +-  0,07% )

 Performance counter stats for 'test/soltest -t ABIDecoderTest,ABIEncoderTest,SolidityAuctionRegistrar,SolidityFixedFeeRegistrar,SolidityWallet,GasMeterTests,GasCostTests,SolidityEndToEndTest,SolidityOptimizer -- 
--vm ./libevmone.so,O=2' (10 runs):

         20 099,19 msec task-clock                #    1,000 CPUs utilized            ( +-  0,10% )
                65      context-switches          #    0,003 K/sec                    ( +-  5,25% )
                 0      cpu-migrations            #    0,000 K/sec                    ( +- 50,92% )
            12 026      page-faults               #    0,598 K/sec                    ( +-  0,00% )
    88 400 104 940      cycles                    #    4,398 GHz                      ( +-  0,10% )
   133 004 916 548      instructions              #    1,50  insn per cycle           ( +-  0,00% )
    21 317 895 354      branches                  # 1060,635 M/sec                    ( +-  0,00% )
       264 350 780      branch-misses             #    1,24% of all branches          ( +-  0,20% )

           20,0998 +- 0,0197 seconds time elapsed  ( +-  0,10% )

Internal benchmark suite

Comparing advanced to baseline
Benchmark                             Time   Time Old   Time New
----------------------------------------------------------------
blake2b_huff/empty                 -0.4959         49         25
blake2b_huff/2805nulls             +0.2660        314        398
blake2b_huff/5610nulls             +0.3348        579        773
blake2b_huff/8415nulls             +0.3584        830       1128
blake2b_huff/65536nulls            +0.4093       6187       8719
blake2b_shifts/2805nulls           -0.0143       4028       3971
blake2b_shifts/5610nulls           -0.0146       8261       8141
blake2b_shifts/8415nulls           -0.0117      12360      12215
blake2b_shifts/65536nulls          -0.0123      97256      96057
sha1_divs/empty                    +0.0478         61         64
sha1_divs/1351                     +0.1307       1168       1321
sha1_divs/2737                     +0.1329       2274       2577
sha1_divs/5311                     +0.1346       4438       5035
sha1_divs/65536                    +0.1365      53961      61326
sha1_shifts/empty                  +0.1238         37         42
sha1_shifts/1351                   +0.2825        673        863
sha1_shifts/2737                   +0.2874       1307       1683
sha1_shifts/5311                   +0.2857       2550       3279
sha1_shifts/65536                  +0.2816      31008      39740
weierstrudel/0                     -0.1184        225        198
weierstrudel/1                     -0.0340        451        436
weierstrudel/3                     -0.0072        682        677
weierstrudel/9                     +0.0202       1367       1394
weierstrudel/14                    +0.0268       1938       1990
micro/beginsub_push1s_0xffff       -0.5347        154         72
micro/beginsubs_0xffff             -0.8990        254         26
micro/jumpdests_0xffff             -0.5911        539        220
micro/loop_with_many_jumpdests     -0.0092      16183      16033
micro/push1s_0xffff                -0.5178        157         76
micro/push32s_0xffff               -0.7140         20          6
micro/zeros_0xffff                 -0.9363        403         26

@chfast chfast force-pushed the baseline branch 3 times, most recently from 477ba74 to 3dc2b12 Compare October 19, 2020 09:53
@chfast chfast marked this pull request as ready for review October 19, 2020 09:54
@chfast chfast requested review from gumb0, axic, chriseth and holiman October 19, 2020 10:01
@chriseth
Copy link
Contributor

Please also test against the SemanticTests, that should be the largest group of tests that run against the EVM.

@chfast
Copy link
Member Author

chfast commented Oct 19, 2020

Please also test against the SemanticTests, that should be the largest group of tests that run against the EVM.

This is similar.

 Performance counter stats for 'test/soltest -t *semanticTest* -- --vm ./libevmone.so,O=0' (10 runs):

         63 676,76 msec task-clock                #    1,000 CPUs utilized            ( +-  0,06% )
               188      context-switches          #    0,003 K/sec                    ( +-  2,60% )
                 2      cpu-migrations            #    0,000 K/sec                    ( +- 33,95% )
            37 636      page-faults               #    0,591 K/sec                    ( +-  0,00% )
   280 020 514 526      cycles                    #    4,398 GHz                      ( +-  0,06% )
   410 641 632 184      instructions              #    1,47  insn per cycle           ( +-  0,00% )
    74 971 303 940      branches                  # 1177,373 M/sec                    ( +-  0,00% )
     1 056 888 261      branch-misses             #    1,41% of all branches          ( +-  0,02% )

           63,6779 +- 0,0390 seconds time elapsed  ( +-  0,06% )

 Performance counter stats for 'test/soltest -t *semanticTest* -- --vm ./libevmone.so,O=2' (10 runs):

         63 844,77 msec task-clock                #    1,000 CPUs utilized            ( +-  0,08% )
               199      context-switches          #    0,003 K/sec                    ( +-  2,59% )
                 2      cpu-migrations            #    0,000 K/sec                    ( +- 26,68% )
            37 782      page-faults               #    0,592 K/sec                    ( +-  0,00% )
   280 792 766 348      cycles                    #    4,398 GHz                      ( +-  0,08% )
   410 193 954 835      instructions              #    1,46  insn per cycle           ( +-  0,00% )
    74 766 270 230      branches                  # 1171,063 M/sec                    ( +-  0,00% )
     1 059 638 613      branch-misses             #    1,42% of all branches          ( +-  0,03% )

           63,8459 +- 0,0494 seconds time elapsed  ( +-  0,08% )

lib/evmone/baseline.cpp Outdated Show resolved Hide resolved
lib/evmone/baseline.cpp Outdated Show resolved Hide resolved
lib/evmone/baseline.cpp Outdated Show resolved Hide resolved
@chfast
Copy link
Member Author

chfast commented Dec 17, 2020

Some final benchmarks with Clang 11.

[advanced vs. baseline]/execute/main/blake2b_huff/empty_mean                         -0.5214         -0.5214            51            24            51            24
[advanced vs. baseline]/execute/main/blake2b_huff/2805nulls_mean                     +0.2390         +0.2390           314           389           314           389
[advanced vs. baseline]/execute/main/blake2b_huff/5610nulls_mean                     +0.3017         +0.3017           576           750           576           750
[advanced vs. baseline]/execute/main/blake2b_huff/8415nulls_mean                     +0.3302         +0.3302           825          1097           825          1097
[advanced vs. baseline]/execute/main/blake2b_huff/65536nulls_mean                    +0.3873         +0.3873          6109          8475          6109          8475
[advanced vs. baseline]/execute/main/blake2b_shifts/2805nulls_mean                   -0.0258         -0.0258          3986          3883          3986          3883
[advanced vs. baseline]/execute/main/blake2b_shifts/5610nulls_mean                   -0.0231         -0.0231          8156          7968          8156          7968
[advanced vs. baseline]/execute/main/blake2b_shifts/8415nulls_mean                   -0.0224         -0.0224         12225         11951         12225         11951
[advanced vs. baseline]/execute/main/blake2b_shifts/65536nulls_mean                  -0.0211         -0.0211         96244         94212         96244         94212
[advanced vs. baseline]/execute/main/sha1_divs/empty_mean                            +0.0233         +0.0233            62            63            62            63
[advanced vs. baseline]/execute/main/sha1_divs/1351_mean                             +0.1054         +0.1054          1166          1289          1166          1289
[advanced vs. baseline]/execute/main/sha1_divs/2737_mean                             +0.1093         +0.1093          2269          2518          2270          2518
[advanced vs. baseline]/execute/main/sha1_divs/5311_mean                             +0.1105         +0.1105          4425          4914          4425          4914
[advanced vs. baseline]/execute/main/sha1_divs/65536_mean                            +0.1121         +0.1121         53814         59848         53815         59848
[advanced vs. baseline]/execute/main/sha1_shifts/empty_mean                          +0.0861         +0.0861            37            40            37            40
[advanced vs. baseline]/execute/main/sha1_shifts/1351_mean                           +0.2164         +0.2164           675           821           675           821
[advanced vs. baseline]/execute/main/sha1_shifts/2737_mean                           +0.2211         +0.2211          1313          1603          1313          1603
[advanced vs. baseline]/execute/main/sha1_shifts/5311_mean                           +0.2217         +0.2217          2558          3126          2558          3126
[advanced vs. baseline]/execute/main/sha1_shifts/65536_mean                          +0.2247         +0.2247         31144         38144         31144         38144
[advanced vs. baseline]/execute/main/weierstrudel/0_mean                             -0.1537         -0.1537           228           193           228           193
[advanced vs. baseline]/execute/main/weierstrudel/1_mean                             -0.0644         -0.0644           453           424           453           424
[advanced vs. baseline]/execute/main/weierstrudel/3_mean                             -0.0327         -0.0327           683           661           683           661
[advanced vs. baseline]/execute/main/weierstrudel/9_mean                             -0.0053         -0.0053          1368          1360          1368          1360
[advanced vs. baseline]/execute/main/weierstrudel/14_mean                            +0.0058         +0.0058          1937          1948          1937          1948
[advanced vs. baseline]/execute/micro/beginsub_push1s_0xffff_mean                    -0.5362         -0.5362           155            72           155            72
[advanced vs. baseline]/execute/micro/beginsubs_0xffff_mean                          -0.8989         -0.8989           255            26           255            26
[advanced vs. baseline]/execute/micro/jumpdests_0xffff_mean                          -0.5990         -0.5990           549           220           549           220
[advanced vs. baseline]/execute/micro/loop_with_many_jumpdests_mean                  -0.0111         -0.0111         16223         16042         16223         16043
[advanced vs. baseline]/execute/micro/push1s_0xffff_mean                             -0.5184         -0.5184           157            76           157            76
[advanced vs. baseline]/execute/micro/push32s_0xffff_mean                            -0.7126         -0.7126            20             6            20             6
[advanced vs. baseline]/execute/micro/zeros_0xffff_mean                              -0.9387         -0.9387           420            26           420            26

@codecov-io
Copy link

codecov-io commented Dec 18, 2020

Codecov Report

Merging #261 (cfdcfc7) into master (578ce07) will increase coverage by 0.07%.
The diff coverage is 99.64%.

@@            Coverage Diff             @@
##           master     #261      +/-   ##
==========================================
+ Coverage   99.67%   99.75%   +0.07%     
==========================================
  Files          23       24       +1     
  Lines        3073     3634     +561     
==========================================
+ Hits         3063     3625     +562     
+ Misses         10        9       -1     
Flag Coverage Δ
unittests 99.75% <99.64%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
test/unittests/evm_fixture.cpp 85.71% <75.00%> (+5.71%) ⬆️
lib/evmone/baseline.cpp 99.81% <99.81%> (ø)
lib/evmone/evmone.cpp 100.00% <100.00%> (ø)
test/unittests/evmone_test.cpp 100.00% <100.00%> (ø)
test/unittests/evm_other_test.cpp 100.00% <0.00%> (+2.63%) ⬆️

Copy link
Member

@gumb0 gumb0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@chfast chfast merged commit afad0aa into master Dec 21, 2020
@chfast chfast deleted the baseline branch December 21, 2020 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants