Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql backend w/ receipt log indexing #883

Merged
merged 85 commits into from
May 31, 2023

Conversation

tgmichel
Copy link
Contributor

@tgmichel tgmichel commented Oct 28, 2022

Co-authored by @nbaztec

This PR adds an optional relational backend type for Frontier database. It can be enabled using the cli argument

--frontier-backend-type sql

When enabled it completely replaces RocksDB as Frontier DB meaning if you want to use it you will need to sync down to genesis.

The only engine currently supported is Sqlite, because is an embeded replacement for Rocks, although the crate of choice - sqlx - should allow us to add support for additional backends in the future (e.g. Postgresql).

The main motivation for this PR is to add proper, performant support for Ethereum Log indexing and querying. At Moonbeam we've been succesfully testing this for a while now although not over live traffic, so it is considered experimental.

Some comparison (Rocks VS Sqlite) tables for response times gathered using a random Log request generation over Moonbase Alpha database (+330M transactions, +16M logs) (click to expand):


100 to 1000 blocks range request
----------------------------------------------------------------------------------------------------
      ║ RocksDB                            SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║  | 0          | 2379ms        | 0          | 53ms         {"fromBlock":2284210,"toBlock":2284649,"address":["0x7161ACAEF4059Dd805Df35a4C57D30F2dc31741E","0x7161ACAEF4059Dd805Df35a4C57D30F2dc31741E"],"topics":[null,"0x0000000000000000000000007161acaef4059dd805df35a4c57d30f2dc31741e","0x00000000000000000000000090a70aae360e5e69c3cb466880f025985810f2c8"]}
----------------------------------------------------------------------------------------------------
2     ║  | 0          | 1942ms        | 0          | 53ms         {"fromBlock":1888556,"toBlock":1889080,"address":"0x79371FC247ea3780e3d53e10855D114877C8ea99"}
----------------------------------------------------------------------------------------------------
3     ║  | 0          | 3071ms        | 0          | 461ms        {"fromBlock":3474,"toBlock":4390}
----------------------------------------------------------------------------------------------------
4     ║  | 0          | 2598ms        | 0          | 57ms         {"fromBlock":791633,"toBlock":792520,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xF0322775DC5a0019d88ba301578770C637eBbBDb"],"topics":[["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0","0x4dec04e750ca11537cabcd8a9eab06494de08da3735bc8871cd41250e190bc04","0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x13ed6866d4e1ee6da46f845c46d7e54120883d75c5ea9a2dacc1c4ca8984ab80"],null,"0x000000000000000000000000f0322775dc5a0019d88ba301578770c637ebbbdb"]}
----------------------------------------------------------------------------------------------------
5     ║  | 0          | 1351ms        | 0          | 55ms         {"fromBlock":1404551,"toBlock":1404893,"address":["0x94C32B7BCB917036006659CE7D9358BFC0C60201","0x27D4fE01C690317F5584696dE2f815ab3FE87037"]}
----------------------------------------------------------------------------------------------------
6     ║  | 0          | 293ms         | 0          | 26ms         {"fromBlock":3584046,"toBlock":3584313,"address":"0xdEc1bc71bf91431D60eF2742f412DCd1c5A204B8"}
----------------------------------------------------------------------------------------------------
7     ║  | 0          | 465ms         | 0          | 26ms         {"fromBlock":1579515,"toBlock":1579627,"address":"0xcFCD4A09e5EB3A1b48cc399484990e9660e1F7A6"}
----------------------------------------------------------------------------------------------------
8     ║  | 984        | 1788ms        | 984        | 886ms        {"fromBlock":1221398,"toBlock":1221858}
----------------------------------------------------------------------------------------------------
9     ║  | 754        | 1041ms        | 754        | 944ms        {"fromBlock":3479005,"toBlock":3479295}
----------------------------------------------------------------------------------------------------
10    ║  | 392        | 3411ms        | 392        | 1981ms       {"fromBlock":677686,"toBlock":678417,"topics":["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0"]}
----------------------------------------------------------------------------------------------------
11    ║  | 0          | 1238ms        | 0          | 52ms         {"fromBlock":1780013,"toBlock":1780375,"address":["0xCc2C8b69d540Cb0AB3E02561a588436Df9c01910","0x4382DDE5C3DFbAD96087B87cA592436f95e94b54","0x820F510F1B3E468024761247aDf6e3Be270f3475"],"topics":["0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925",null,"0x0000000000000000000000008d49ea9cb3ad3b710f68be6f6514f41ca462b392"]}
----------------------------------------------------------------------------------------------------
12    ║  | 1573       | 3486ms        | 1573       | 2658ms       {"fromBlock":2352092,"toBlock":2352726}
----------------------------------------------------------------------------------------------------
13    ║  | 0          | 194ms         | 0          | 27ms         {"fromBlock":3652982,"toBlock":3653297,"address":["0x3AeEbD0Fa9701f4549753C4C8498291c1c8AF8d7","0xCD7A0D098E3A750126b0fec54BE401476812cfc0","0xCD7A0D098E3A750126b0fec54BE401476812cfc0"],"topics":[["0x0fe05e323c51ec64db29d65e4471f535fbed63586f534f452455b30e98542483","0xf05ac779af1ec75a7b2fbe9415b33a67c00294a121786f7ce2eb3f92e4a6424a","0x5b5af0622001a9b735a56357ddc1abd65e6a640126498674daf9d2fb05160725","0x29486b9e2ae569b440933a9b1b421467306fa21f3dcad439c262910a634963a9"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000000000000000000000000000000000000000000000000001","0x0000000000000000000000000000000000000000000000000000000000000009","0x0000000000000000000000000000000000000000000000000000000000001d67"],["0x0000000000000000000000000000000000000000000000000000000000fb3e6b","0x0000000000000000000000000000000000000000000000000000000000000131","0x000000000000000000000000d8b6645541aae0ed7ab2c182ab22b6505c5b18d5"],["0x78621d5d09d277d1793101e89a44ce538432318615bb50eb413dbfd54528923b","0x000000000000000000000000000000000000000000000000000000000000000a"]]}
----------------------------------------------------------------------------------------------------
14    ║  | 0          | 1958ms        | 0          | 25ms         {"fromBlock":707340,"toBlock":708089,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0x42B15a11cc295BE6f97Aa4518E850B064b64FB11"]}
----------------------------------------------------------------------------------------------------
15    ║  | 4539       | 3034ms        | 4539       | 3171ms       {"fromBlock":1277548,"toBlock":1278188}
----------------------------------------------------------------------------------------------------
1001 to 2500 blocks range request
----------------------------------------------------------------------------------------------------
      ║ RocksDB                            SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║  | 0          | 5271ms        | 0          | 71ms         {"fromBlock":2424496,"toBlock":2425513,"address":["0x2A330B171AA85b99810AC30E8DE16cf22b69b426","0x2A330B171AA85b99810AC30E8DE16cf22b69b426"]}
----------------------------------------------------------------------------------------------------
2     ║  | 3496       | 9818ms        | 3496       | 6124ms       {"fromBlock":3249247,"toBlock":3251175}
----------------------------------------------------------------------------------------------------
3     ║  | 790        | 4755ms        | 790        | 5291ms       {"fromBlock":1809259,"toBlock":1811157,"topics":["0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925","0x000000000000000000000000bd8d8650cf1fd991e8e7696d95536a6f7479c49c","0x000000000000000000000000faa8a4d6ac08e8e470d5f4ed771d645d5caf5957"]}
----------------------------------------------------------------------------------------------------
4     ║  | 0          | 3007ms        | 0          | 27ms         {"fromBlock":2665851,"toBlock":2667814,"address":"0x04a72797360815e33Fc6676Bf0b02F7D3E901c78"}
----------------------------------------------------------------------------------------------------
5     ║  | 4823       | 7263ms        | 4823       | 8344ms       {"fromBlock":3059915,"toBlock":3061926}
----------------------------------------------------------------------------------------------------
6     ║  | 0          | 10096ms       | 0          | 5968ms       {"fromBlock":2026368,"toBlock":2027935}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
7     ║  | 1          | 7963ms        | 1          | 919ms        {"fromBlock":531763,"toBlock":533514,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000a3dc12efa6477debc01d118f79451f471d7d3c20"],["0x000000000000000000000000a3dc12efa6477debc01d118f79451f471d7d3c20","0x0000000000000000000000000000000000000000000000000000000000000007"],"0x0000000000000000000000000000000000000000000000000000000000007200"]}
----------------------------------------------------------------------------------------------------
8     ║  | 1          | 908ms         | 1          | 660ms        {"fromBlock":3582211,"toBlock":3583548,"topics":[["0x127186556e7be68c7e31263195225b4de02820707889540969f62c05cf73525e","0x602f1aeac0ca2e7a13e281a9ef0ad7838542712ce16780fa2ecffd351f05f899"],"0x0000000000000000000000000000000000000000000000000000000000006331"]}
----------------------------------------------------------------------------------------------------
9     ║  | 9458       | 2877ms        | 9458       | 6761ms       {"fromBlock":3395715,"toBlock":3397528}
----------------------------------------------------------------------------------------------------
10    ║  | 1          | 3878ms        | 1          | 660ms        {"fromBlock":3484267,"toBlock":3485555,"topics":[["0x91632a5a137b662a3150dceee2472d910950953afbf5ac128f87be7aaef5c952","0x6c3543130e60b15dd1f1f886e61a65ec337da34b356e078fcfe585b5578f39ef","0x147f1c3f003f534adab0058fb3df9bd88c05c9b2830c725bba5c699ac2042446","0x6ab61a3b8c08b4c30ed9c9f7498d6d9760df2118cd386ebd2de31033791a4331","0x66cbca4f3c64fecf1dcb9ce094abcf7f68c3450a1d4e3a8e917dd621edb4ebe0","0x3ffdfc472f6c61636e8d2af87c313062b1e6b246a45d16a63553b26089281dc0"],["0x0000000000000000000000006cf00ae50def3fd2767ff1b63b4dfa1a006f41e9","0x0000000000000000000000000000000000000000000000000000000000000001"],["0x0000000000000000000000000000000000000000000000000000000000000fa2","0x000000000000000000000000576cfcf9be1daed9b7066c455d6c7ae17e070a91"],"0x00000000000000000000000006a19e6e274fdb241649ae0f4ae8e9d939ad6193"]}
----------------------------------------------------------------------------------------------------
11    ║  | 0          | 2898ms        | 0          | 28ms         {"fromBlock":3060142,"toBlock":3061891,"address":["0x366a64fa544034F6d538B413068E89709cd90cd6","0x2A330B171AA85b99810AC30E8DE16cf22b69b426","0x86dcB387fdab45f2f415c13aA72A61122e49d73E"]}
----------------------------------------------------------------------------------------------------
12    ║  | 9          | 6652ms        | 9          | 1247ms       {"fromBlock":3108209,"toBlock":3110354,"topics":["0x6d47968b2a13605f2dbe5e2e9f6513ad9c81fb14696d94ae9299a452ed62e0e6","0x0000000000000000000000000325fb902fd1b7aad06f9467f41b32913d316b2d"]}
----------------------------------------------------------------------------------------------------
13    ║  | 0          | 3702ms        | 0          | 26ms         {"fromBlock":3353489,"toBlock":3354934,"address":["0x2e1d90501C3173367ecC6a409Fb1b588Bf3C16A5","0x29893eEFF38C5D5A1B2F693e2d918e618CCFfdD8"]}
----------------------------------------------------------------------------------------------------
14    ║  | 1635       | 4770ms        | 1635       | 2293ms       {"fromBlock":3440054,"toBlock":3441087}
----------------------------------------------------------------------------------------------------
15    ║  | 2          | 5746ms        | 2          | 1047ms       {"fromBlock":501853,"toBlock":504113,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b","0x003da85d6058444eca57e3b88be14c734f4f239affe28b7629966e318de2a90e"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x00000000000000000000000016b9ae658e5c4bc4d58a7841b3b4050524b6265f","0x000000000000000000000000f8bad7d45f905c2e59a58840dddd73fbc55c23e3"],["0x00000000000000000000000016b9ae658e5c4bc4d58a7841b3b4050524b6265f","0x0000000000000000000000000000000000000000000000000000000000000009","0x000000000000000000000000f8bad7d45f905c2e59a58840dddd73fbc55c23e3","0x000000000000000000000000952b02ebada16f84f64e388a2030e230334f41ed","0x000000000000000000000000000000000000000000000000000000000000000d"],["0x00000000000000000000000000000000000000000000000000000000000035ff","0x000000000000000000000000000000000000000000000000000000000000377a"]]}
----------------------------------------------------------------------------------------------------
2501 to 5000 blocks range request
----------------------------------------------------------------------------------------------------
      ║ RocksDB                            SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║  | 0          | 10060ms       | 0          | 68ms         {"fromBlock":1752572,"toBlock":1755935,"address":"0x4bc0760e50eF8Fc2DC48052D1B11cc62578E2F6c"}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
2     ║  | 0          | 10048ms       | 0          | 50ms         {"fromBlock":2825227,"toBlock":2828956,"address":["0xB308eDAe61436B3a00EC6adD22a1F738064E8AED","0xd54C1b20Ffb06E9d229395abb45a064026FedEa8","0x33467aaEb8BF5Ad18984C50b9107801F6F12d07B"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
3     ║  | 1          | 8521ms        | 1          | 946ms        {"fromBlock":3011544,"toBlock":3014256,"topics":[["0x44e4f8f6bd682c5a3aeba93601ab07cb4d1f21b2aab1ae4880d9577919309aa4","0xa74c8847d513feba22a0f0cb38d53081abf97562cdb293926ba243689e7c41ca"],null,null,"0x5356282e07af4c4642b489e27d452b2ddfe24056963341ed85712149065ac126"]}
----------------------------------------------------------------------------------------------------
4     ║  | 0          | 10052ms       | 0          | 53ms         {"fromBlock":307382,"toBlock":311827,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
5     ║  | 0          | 8539ms        | 0          | 53ms         {"fromBlock":714351,"toBlock":718624,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0xe642E847b16cdb13312F762a92eD4d5371A1B2F3"],"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000e642e847b16cdb13312f762a92ed4d5371a1b2f3"],["0x000000000000000000000000e642e847b16cdb13312f762a92ed4d5371a1b2f3","0x0000000000000000000000000000000000000000000000000000000000000030"]]}
----------------------------------------------------------------------------------------------------
6     ║  | 0          | 4010ms        | 0          | 4602ms       {"fromBlock":1327015,"toBlock":1331396}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
7     ║  | 0          | 8687ms        | 0          | 50ms         {"fromBlock":3410216,"toBlock":3413418,"address":["0xD36aac0c9676e984D72823Fb662ce94D3Ab5E551","0x2e1d90501C3173367ecC6a409Fb1b588Bf3C16A5"],"topics":[["0x91632a5a137b662a3150dceee2472d910950953afbf5ac128f87be7aaef5c952","0x6c3543130e60b15dd1f1f886e61a65ec337da34b356e078fcfe585b5578f39ef","0x147f1c3f003f534adab0058fb3df9bd88c05c9b2830c725bba5c699ac2042446","0x6ab61a3b8c08b4c30ed9c9f7498d6d9760df2118cd386ebd2de31033791a4331","0x66cbca4f3c64fecf1dcb9ce094abcf7f68c3450a1d4e3a8e917dd621edb4ebe0"],"0x000000000000000000000000372bd88bd7fc79f6ba11c553107a75417110f5ad","0x000000000000000000000000000000000000000000000000000000000000a869"]}
----------------------------------------------------------------------------------------------------
8     ║  | 0          | 10051ms       | 0          | 53ms         {"fromBlock":884404,"toBlock":888655,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0x3F3440A7ac383C0cbFa088A2A7D2D945f8331De1"],"topics":[["0x2caecd17d02f56fa897705dcc740da2d237c373f70686f4e0d9bd3bf0400ea7a","0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x4c209b5fc8ad50758f13e2e1088ba56a560dff690a1c6fef26394f4c03821c4f"],["0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d","0x0000000000000000000000003f3440a7ac383c0cbfa088a2a7d2d945f8331de1"],["0x0000000000000000000000003f3440a7ac383c0cbfa088a2a7d2d945f8331de1","0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d"]]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
9     ║  | 0          | 2068ms        | 0          | 3503ms       {"fromBlock":914482,"toBlock":919251}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
10    ║  | 0          | 1865ms        | 0          | 3806ms       {"fromBlock":815908,"toBlock":820890}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
11    ║  | 0          | 10053ms       | 629        | 1328ms       {"fromBlock":64620,"toBlock":67939}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
12    ║  | 0          | 9545ms        | 0          | 54ms         {"fromBlock":3058958,"toBlock":3061483,"address":["0x303F9330093d0E09f5e16e9af5cDdcfb416623CA","0xdEc1bc71bf91431D60eF2742f412DCd1c5A204B8","0xEb2e970cfCb087e7826862880A1F45FB3020bD87"],"topics":["0xa7fc99ed7617309ee23f63ae90196a1e490d362e6f6a547a59bc809ee2291782"]}
----------------------------------------------------------------------------------------------------
13    ║  | 4          | 6864ms        | 4          | 963ms        {"fromBlock":963071,"toBlock":966361,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000f7571bb707b44c11e7225dd6f84b34aa96ea118d"],["0x000000000000000000000000662e19154aacddd6dd08605f69c7f5a7627062d5","0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d"]]}
----------------------------------------------------------------------------------------------------
14    ║  | 0          | 10070ms       | 0          | 8013ms       {"fromBlock":2268388,"toBlock":2273200}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
15    ║  | 0          | 8921ms        | 0          | 51ms         {"fromBlock":618271,"toBlock":622930,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xF516bDBd32A0B4CB07D21F26B9185B336502027f"]}
----------------------------------------------------------------------------------------------------
5001 to 10000 blocks range request
----------------------------------------------------------------------------------------------------
      ║ RocksDB                            SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║  | 0          | 10065ms       | 0          | 6951ms       {"fromBlock":2212482,"toBlock":2220814}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
2     ║  | 0          | 10049ms       | 0          | 54ms         {"fromBlock":1184158,"toBlock":1189949,"address":["0x0115dbCaA30f0350Ec49977874f6f4ef4297D377","0x57c94e147D43f96ecCEB7a978702e71A047c3Bdf","0x7e5d75a58Da31efdBeA997244Bd99e85ff871798"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
3     ║  | 0          | 10047ms       | 0          | 55ms         {"fromBlock":1446562,"toBlock":1452288,"address":["0x4063e6FBB413AA3cEFC6963DbfF937889D6C79F6","0x820F510F1B3E468024761247aDf6e3Be270f3475","0x74775d2A767DBDf062b5C587F87C54396634D11c"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
4     ║  | 0          | 3606ms        | 0          | 3155ms       {"fromBlock":893753,"toBlock":898763}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
5     ║  | 0          | 10058ms       | 0          | 58ms         {"fromBlock":2609330,"toBlock":2619041,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0x5FdCf221e987966b296714953fb6952AfC24aeFE","0xD05526a73bf45DaDf7f9A99dCCeAC23C2D43C6C7"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
6     ║  | 0          | 10049ms       | 0          | 52ms         {"fromBlock":673708,"toBlock":680240,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xF516bDBd32A0B4CB07D21F26B9185B336502027f"],"topics":["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
7     ║  | 0          | 10058ms       | 0          | 10085ms      {"fromBlock":1763681,"toBlock":1770114}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
8     ║  | 0          | 10056ms       | 7239       | 7057ms       {"fromBlock":2746236,"toBlock":2752570}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
9     ║  | 0          | 3206ms        | 0          | 4589ms       {"fromBlock":1975341,"toBlock":1983732}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
10    ║  | 0          | 10061ms       | 0          | 54ms         {"fromBlock":1315877,"toBlock":1322494,"address":["0x7e5d75a58Da31efdBeA997244Bd99e85ff871798","0x570214c493D124A67AE34079961367758cBE0450","0x47353EbfB9078894601677b6E2613B59Ba9Ab049"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
11    ║  | 0          | 10050ms       | 0          | 51ms         {"fromBlock":924213,"toBlock":931249,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xa8D30a70eFe49D9d459f0e25725d7F84D93C976e"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
12    ║  | 0          | 10052ms       | 0          | 10082ms      {"fromBlock":1016191,"toBlock":1024431}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
13    ║  | 0          | 3471ms        | 0          | 4709ms       {"fromBlock":2338869,"toBlock":2347064}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
14    ║  | 0          | 10053ms       | 0          | 52ms         {"fromBlock":181687,"toBlock":188812,"address":["0x9755F1eA7a6f6058dFc02F95Bc932C3c261c1f31","0x42B15a11cc295BE6f97Aa4518E850B064b64FB11"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
15    ║  | 0          | 10059ms       | 0          | 55ms         {"fromBlock":731617,"toBlock":738527,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0x6FFA2707eA1fe81c8DAfA755E74f4961333B75fA"],"topics":[["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0","0x0109fc6f55cf40689f02fbaad7af7fe7bbac8a3d2186600afc7d3e10cac60271","0x92e98423f8adac6e64d0608e519fd1cefb861498385c6dee70d58fc926ddc68c","0x0559884fd3a460db3073b7fc896cc77986f16e378210ded43186175bf646fc5f","0xfe25c73e3b9089fac37d55c4c7efcba6f04af04cebd2fc4d6d7dbb07e1e5234f"],["0x0000000000000000000000000000000000000000000000000000000000000d34","0x0000000000000000000000000000000000000000000000000000000789159345","0x000000000000000000000000000000000000000000000002b5e3af169d628780"],["0x0000000000000000000000006ffa2707ea1fe81c8dafa755e74f4961333b75fa","0x0000000000000000000000000000000000000000000000000000000000000d34"],"0x0000000000000000000000006ffa2707ea1fe81c8dafa755e74f4961333b75fa"]}

}

/// Create the Sqlite database if it does not already exist.
async fn create_database_if_not_exists(pool: &SqlitePool) -> Result<SqliteQueryResult, Error> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's your plan for database migrations in the future?

If it's planned to do it manually, then please add a metadata table that will store the version information.

Or you can consider using a framework, please just take a look making sure it's compatible. Some of the migration tools will store all migrations applied in a migration table (and just check missing ones to apply). In that case we can leave this until the first migration is written.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we deferred it until the first migration needs to occur. In that case the absence of the versions table will serve as version 0 and any subsequent migrations can be built on top of it.

Copy link
Member

@sorpaas sorpaas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than issues with future db migrations.

@@ -1,38 +1,53 @@
[package]
name = "fc-db"
version = "2.0.0-dev"
authors = ["Parity Technologies <admin@parity.io>"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be replaced with workspace setting, like other crates.

_topics: Vec<Vec<Option<H256>>>,
) -> Result<Vec<crate::FilteredLog>, String> {
// KeyValue db does not index logs
unimplemented!();
Copy link
Collaborator

@boundless-forest boundless-forest Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this cause potential panic for the kv backend client? I noticed that this is used in the filter_range_logs_indexed

Suggest returning empty vector.

Copy link
Contributor

@nbaztec nbaztec Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already checked by whether the backend is indexed or not here so will not cause a panic. This ensures that the indexed logs are only returned for supported backends while the unsupported ones use the older filter_range_logs function instead.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not return error instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That can be a possibility to return Err("not supported")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think that would be preferred than unimplemented.

@nbaztec
Copy link
Contributor

nbaztec commented Apr 20, 2023

Changes applied @sorpaas @boundless-forest

@boundless-forest
Copy link
Collaborator

Well done! So many changes, hard to review. I spend the whole afternoon reviewing them. Overall, LGTM with so many test cases are added. I am worried a bit about the migration solution in the future.

@tgmichel
Copy link
Contributor Author

I am worried a bit about the migration solution in the future

Can you explain with more detail what is your concern?

@boundless-forest
Copy link
Collaborator

I am worried a bit about the migration solution in the future

Can you explain with more detail what is your concern?

Similar to the oponion, mainly about the data migration or fix solution if we adjust the data schema later when doing bug fix. This issue also existed when using kv db backend. Let leave it alone now.

@nbaztec
Copy link
Contributor

nbaztec commented Apr 21, 2023

Similar to the oponion, mainly about the data migration or fix solution if we adjust the data schema later when doing bug fix. This issue also existed when using kv db backend. Let leave it alone now.

That's a valid concern, I replied to it earlier on the prospective plans in #883 (comment)

Hope that suffices.

@tgmichel
Copy link
Contributor Author

tgmichel commented May 5, 2023

@sorpaas is there something preventing this to be merged?

@sorpaas
Copy link
Member

sorpaas commented May 16, 2023

Please pull master.

@sorpaas
Copy link
Member

sorpaas commented May 16, 2023

Something seems wrong with the CI. Please push an empty commit!

@sorpaas sorpaas merged commit aa57359 into polkadot-evm:master May 31, 2023
@nazar-pc
Copy link

Maybe consider making this feature optional? It pulls way too many dependencies, including OpenSSL.

@tgmichel
Copy link
Contributor Author

@nazar-pc please create an issue referencing this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants