sql backend w/ receipt log indexing #883

tgmichel · 2022-10-28T09:02:08Z

Co-authored by @nbaztec

This PR adds an optional relational backend type for Frontier database. It can be enabled using the cli argument

--frontier-backend-type sql

When enabled it completely replaces RocksDB as Frontier DB meaning if you want to use it you will need to sync down to genesis.

The only engine currently supported is Sqlite, because is an embeded replacement for Rocks, although the crate of choice - sqlx - should allow us to add support for additional backends in the future (e.g. Postgresql).

The main motivation for this PR is to add proper, performant support for Ethereum Log indexing and querying. At Moonbeam we've been succesfully testing this for a while now although not over live traffic, so it is considered experimental.

Some comparison (Rocks VS Sqlite) tables for response times gathered using a random Log request generation over Moonbase Alpha database (+330M transactions, +16M logs) (click to expand):

100 to 1000 blocks range request

----------------------------------------------------------------------------------------------------
      ║ RocksDB                             ║ SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║ ✔    | 0          | 2379ms          ║ ✔    | 0          | 53ms            ║ {"fromBlock":2284210,"toBlock":2284649,"address":["0x7161ACAEF4059Dd805Df35a4C57D30F2dc31741E","0x7161ACAEF4059Dd805Df35a4C57D30F2dc31741E"],"topics":[null,"0x0000000000000000000000007161acaef4059dd805df35a4c57d30f2dc31741e","0x00000000000000000000000090a70aae360e5e69c3cb466880f025985810f2c8"]}
----------------------------------------------------------------------------------------------------
2     ║ ✔    | 0          | 1942ms          ║ ✔    | 0          | 53ms            ║ {"fromBlock":1888556,"toBlock":1889080,"address":"0x79371FC247ea3780e3d53e10855D114877C8ea99"}
----------------------------------------------------------------------------------------------------
3     ║ ✔    | 0          | 3071ms          ║ ✔    | 0          | 461ms           ║ {"fromBlock":3474,"toBlock":4390}
----------------------------------------------------------------------------------------------------
4     ║ ✔    | 0          | 2598ms          ║ ✔    | 0          | 57ms            ║ {"fromBlock":791633,"toBlock":792520,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xF0322775DC5a0019d88ba301578770C637eBbBDb"],"topics":[["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0","0x4dec04e750ca11537cabcd8a9eab06494de08da3735bc8871cd41250e190bc04","0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x13ed6866d4e1ee6da46f845c46d7e54120883d75c5ea9a2dacc1c4ca8984ab80"],null,"0x000000000000000000000000f0322775dc5a0019d88ba301578770c637ebbbdb"]}
----------------------------------------------------------------------------------------------------
5     ║ ✔    | 0          | 1351ms          ║ ✔    | 0          | 55ms            ║ {"fromBlock":1404551,"toBlock":1404893,"address":["0x94C32B7BCB917036006659CE7D9358BFC0C60201","0x27D4fE01C690317F5584696dE2f815ab3FE87037"]}
----------------------------------------------------------------------------------------------------
6     ║ ✔    | 0          | 293ms           ║ ✔    | 0          | 26ms            ║ {"fromBlock":3584046,"toBlock":3584313,"address":"0xdEc1bc71bf91431D60eF2742f412DCd1c5A204B8"}
----------------------------------------------------------------------------------------------------
7     ║ ✔    | 0          | 465ms           ║ ✔    | 0          | 26ms            ║ {"fromBlock":1579515,"toBlock":1579627,"address":"0xcFCD4A09e5EB3A1b48cc399484990e9660e1F7A6"}
----------------------------------------------------------------------------------------------------
8     ║ ✔    | 984        | 1788ms          ║ ✔    | 984        | 886ms           ║ {"fromBlock":1221398,"toBlock":1221858}
----------------------------------------------------------------------------------------------------
9     ║ ✔    | 754        | 1041ms          ║ ✔    | 754        | 944ms           ║ {"fromBlock":3479005,"toBlock":3479295}
----------------------------------------------------------------------------------------------------
10    ║ ✔    | 392        | 3411ms          ║ ✔    | 392        | 1981ms          ║ {"fromBlock":677686,"toBlock":678417,"topics":["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0"]}
----------------------------------------------------------------------------------------------------
11    ║ ✔    | 0          | 1238ms          ║ ✔    | 0          | 52ms            ║ {"fromBlock":1780013,"toBlock":1780375,"address":["0xCc2C8b69d540Cb0AB3E02561a588436Df9c01910","0x4382DDE5C3DFbAD96087B87cA592436f95e94b54","0x820F510F1B3E468024761247aDf6e3Be270f3475"],"topics":["0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925",null,"0x0000000000000000000000008d49ea9cb3ad3b710f68be6f6514f41ca462b392"]}
----------------------------------------------------------------------------------------------------
12    ║ ✔    | 1573       | 3486ms          ║ ✔    | 1573       | 2658ms          ║ {"fromBlock":2352092,"toBlock":2352726}
----------------------------------------------------------------------------------------------------
13    ║ ✔    | 0          | 194ms           ║ ✔    | 0          | 27ms            ║ {"fromBlock":3652982,"toBlock":3653297,"address":["0x3AeEbD0Fa9701f4549753C4C8498291c1c8AF8d7","0xCD7A0D098E3A750126b0fec54BE401476812cfc0","0xCD7A0D098E3A750126b0fec54BE401476812cfc0"],"topics":[["0x0fe05e323c51ec64db29d65e4471f535fbed63586f534f452455b30e98542483","0xf05ac779af1ec75a7b2fbe9415b33a67c00294a121786f7ce2eb3f92e4a6424a","0x5b5af0622001a9b735a56357ddc1abd65e6a640126498674daf9d2fb05160725","0x29486b9e2ae569b440933a9b1b421467306fa21f3dcad439c262910a634963a9"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000000000000000000000000000000000000000000000000001","0x0000000000000000000000000000000000000000000000000000000000000009","0x0000000000000000000000000000000000000000000000000000000000001d67"],["0x0000000000000000000000000000000000000000000000000000000000fb3e6b","0x0000000000000000000000000000000000000000000000000000000000000131","0x000000000000000000000000d8b6645541aae0ed7ab2c182ab22b6505c5b18d5"],["0x78621d5d09d277d1793101e89a44ce538432318615bb50eb413dbfd54528923b","0x000000000000000000000000000000000000000000000000000000000000000a"]]}
----------------------------------------------------------------------------------------------------
14    ║ ✔    | 0          | 1958ms          ║ ✔    | 0          | 25ms            ║ {"fromBlock":707340,"toBlock":708089,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0x42B15a11cc295BE6f97Aa4518E850B064b64FB11"]}
----------------------------------------------------------------------------------------------------
15    ║ ✔    | 4539       | 3034ms          ║ ✔    | 4539       | 3171ms          ║ {"fromBlock":1277548,"toBlock":1278188}
----------------------------------------------------------------------------------------------------

1001 to 2500 blocks range request

----------------------------------------------------------------------------------------------------
      ║ RocksDB                             ║ SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║ ✔    | 0          | 5271ms          ║ ✔    | 0          | 71ms            ║ {"fromBlock":2424496,"toBlock":2425513,"address":["0x2A330B171AA85b99810AC30E8DE16cf22b69b426","0x2A330B171AA85b99810AC30E8DE16cf22b69b426"]}
----------------------------------------------------------------------------------------------------
2     ║ ✔    | 3496       | 9818ms          ║ ✔    | 3496       | 6124ms          ║ {"fromBlock":3249247,"toBlock":3251175}
----------------------------------------------------------------------------------------------------
3     ║ ✔    | 790        | 4755ms          ║ ✔    | 790        | 5291ms          ║ {"fromBlock":1809259,"toBlock":1811157,"topics":["0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925","0x000000000000000000000000bd8d8650cf1fd991e8e7696d95536a6f7479c49c","0x000000000000000000000000faa8a4d6ac08e8e470d5f4ed771d645d5caf5957"]}
----------------------------------------------------------------------------------------------------
4     ║ ✔    | 0          | 3007ms          ║ ✔    | 0          | 27ms            ║ {"fromBlock":2665851,"toBlock":2667814,"address":"0x04a72797360815e33Fc6676Bf0b02F7D3E901c78"}
----------------------------------------------------------------------------------------------------
5     ║ ✔    | 4823       | 7263ms          ║ ✔    | 4823       | 8344ms          ║ {"fromBlock":3059915,"toBlock":3061926}
----------------------------------------------------------------------------------------------------
6     ║ ✖    | 0          | 10096ms         ║ ✖    | 0          | 5968ms          ║ {"fromBlock":2026368,"toBlock":2027935}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
7     ║ ✔    | 1          | 7963ms          ║ ✔    | 1          | 919ms           ║ {"fromBlock":531763,"toBlock":533514,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000a3dc12efa6477debc01d118f79451f471d7d3c20"],["0x000000000000000000000000a3dc12efa6477debc01d118f79451f471d7d3c20","0x0000000000000000000000000000000000000000000000000000000000000007"],"0x0000000000000000000000000000000000000000000000000000000000007200"]}
----------------------------------------------------------------------------------------------------
8     ║ ✔    | 1          | 908ms           ║ ✔    | 1          | 660ms           ║ {"fromBlock":3582211,"toBlock":3583548,"topics":[["0x127186556e7be68c7e31263195225b4de02820707889540969f62c05cf73525e","0x602f1aeac0ca2e7a13e281a9ef0ad7838542712ce16780fa2ecffd351f05f899"],"0x0000000000000000000000000000000000000000000000000000000000006331"]}
----------------------------------------------------------------------------------------------------
9     ║ ✔    | 9458       | 2877ms          ║ ✔    | 9458       | 6761ms          ║ {"fromBlock":3395715,"toBlock":3397528}
----------------------------------------------------------------------------------------------------
10    ║ ✔    | 1          | 3878ms          ║ ✔    | 1          | 660ms           ║ {"fromBlock":3484267,"toBlock":3485555,"topics":[["0x91632a5a137b662a3150dceee2472d910950953afbf5ac128f87be7aaef5c952","0x6c3543130e60b15dd1f1f886e61a65ec337da34b356e078fcfe585b5578f39ef","0x147f1c3f003f534adab0058fb3df9bd88c05c9b2830c725bba5c699ac2042446","0x6ab61a3b8c08b4c30ed9c9f7498d6d9760df2118cd386ebd2de31033791a4331","0x66cbca4f3c64fecf1dcb9ce094abcf7f68c3450a1d4e3a8e917dd621edb4ebe0","0x3ffdfc472f6c61636e8d2af87c313062b1e6b246a45d16a63553b26089281dc0"],["0x0000000000000000000000006cf00ae50def3fd2767ff1b63b4dfa1a006f41e9","0x0000000000000000000000000000000000000000000000000000000000000001"],["0x0000000000000000000000000000000000000000000000000000000000000fa2","0x000000000000000000000000576cfcf9be1daed9b7066c455d6c7ae17e070a91"],"0x00000000000000000000000006a19e6e274fdb241649ae0f4ae8e9d939ad6193"]}
----------------------------------------------------------------------------------------------------
11    ║ ✔    | 0          | 2898ms          ║ ✔    | 0          | 28ms            ║ {"fromBlock":3060142,"toBlock":3061891,"address":["0x366a64fa544034F6d538B413068E89709cd90cd6","0x2A330B171AA85b99810AC30E8DE16cf22b69b426","0x86dcB387fdab45f2f415c13aA72A61122e49d73E"]}
----------------------------------------------------------------------------------------------------
12    ║ ✔    | 9          | 6652ms          ║ ✔    | 9          | 1247ms          ║ {"fromBlock":3108209,"toBlock":3110354,"topics":["0x6d47968b2a13605f2dbe5e2e9f6513ad9c81fb14696d94ae9299a452ed62e0e6","0x0000000000000000000000000325fb902fd1b7aad06f9467f41b32913d316b2d"]}
----------------------------------------------------------------------------------------------------
13    ║ ✔    | 0          | 3702ms          ║ ✔    | 0          | 26ms            ║ {"fromBlock":3353489,"toBlock":3354934,"address":["0x2e1d90501C3173367ecC6a409Fb1b588Bf3C16A5","0x29893eEFF38C5D5A1B2F693e2d918e618CCFfdD8"]}
----------------------------------------------------------------------------------------------------
14    ║ ✔    | 1635       | 4770ms          ║ ✔    | 1635       | 2293ms          ║ {"fromBlock":3440054,"toBlock":3441087}
----------------------------------------------------------------------------------------------------
15    ║ ✔    | 2          | 5746ms          ║ ✔    | 2          | 1047ms          ║ {"fromBlock":501853,"toBlock":504113,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b","0x003da85d6058444eca57e3b88be14c734f4f239affe28b7629966e318de2a90e"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x00000000000000000000000016b9ae658e5c4bc4d58a7841b3b4050524b6265f","0x000000000000000000000000f8bad7d45f905c2e59a58840dddd73fbc55c23e3"],["0x00000000000000000000000016b9ae658e5c4bc4d58a7841b3b4050524b6265f","0x0000000000000000000000000000000000000000000000000000000000000009","0x000000000000000000000000f8bad7d45f905c2e59a58840dddd73fbc55c23e3","0x000000000000000000000000952b02ebada16f84f64e388a2030e230334f41ed","0x000000000000000000000000000000000000000000000000000000000000000d"],["0x00000000000000000000000000000000000000000000000000000000000035ff","0x000000000000000000000000000000000000000000000000000000000000377a"]]}
----------------------------------------------------------------------------------------------------

2501 to 5000 blocks range request

----------------------------------------------------------------------------------------------------
      ║ RocksDB                             ║ SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║ ✖    | 0          | 10060ms         ║ ✔    | 0          | 68ms            ║ {"fromBlock":1752572,"toBlock":1755935,"address":"0x4bc0760e50eF8Fc2DC48052D1B11cc62578E2F6c"}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
2     ║ ✖    | 0          | 10048ms         ║ ✔    | 0          | 50ms            ║ {"fromBlock":2825227,"toBlock":2828956,"address":["0xB308eDAe61436B3a00EC6adD22a1F738064E8AED","0xd54C1b20Ffb06E9d229395abb45a064026FedEa8","0x33467aaEb8BF5Ad18984C50b9107801F6F12d07B"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
3     ║ ✔    | 1          | 8521ms          ║ ✔    | 1          | 946ms           ║ {"fromBlock":3011544,"toBlock":3014256,"topics":[["0x44e4f8f6bd682c5a3aeba93601ab07cb4d1f21b2aab1ae4880d9577919309aa4","0xa74c8847d513feba22a0f0cb38d53081abf97562cdb293926ba243689e7c41ca"],null,null,"0x5356282e07af4c4642b489e27d452b2ddfe24056963341ed85712149065ac126"]}
----------------------------------------------------------------------------------------------------
4     ║ ✖    | 0          | 10052ms         ║ ✔    | 0          | 53ms            ║ {"fromBlock":307382,"toBlock":311827,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
5     ║ ✔    | 0          | 8539ms          ║ ✔    | 0          | 53ms            ║ {"fromBlock":714351,"toBlock":718624,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0xe642E847b16cdb13312F762a92eD4d5371A1B2F3"],"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x36e996ba214404b0a557ef87433cf92f99d6f4e07c2ca5b7cf72a95d7ac7f09f","0x9110cd00ebf5075b22eead1dbb6c4f4c5b7bd329bc646333c004dfb9e8c21f0b"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000e642e847b16cdb13312f762a92ed4d5371a1b2f3"],["0x000000000000000000000000e642e847b16cdb13312f762a92ed4d5371a1b2f3","0x0000000000000000000000000000000000000000000000000000000000000030"]]}
----------------------------------------------------------------------------------------------------
6     ║ ✖    | 0          | 4010ms          ║ ✖    | 0          | 4602ms          ║ {"fromBlock":1327015,"toBlock":1331396}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
7     ║ ✔    | 0          | 8687ms          ║ ✔    | 0          | 50ms            ║ {"fromBlock":3410216,"toBlock":3413418,"address":["0xD36aac0c9676e984D72823Fb662ce94D3Ab5E551","0x2e1d90501C3173367ecC6a409Fb1b588Bf3C16A5"],"topics":[["0x91632a5a137b662a3150dceee2472d910950953afbf5ac128f87be7aaef5c952","0x6c3543130e60b15dd1f1f886e61a65ec337da34b356e078fcfe585b5578f39ef","0x147f1c3f003f534adab0058fb3df9bd88c05c9b2830c725bba5c699ac2042446","0x6ab61a3b8c08b4c30ed9c9f7498d6d9760df2118cd386ebd2de31033791a4331","0x66cbca4f3c64fecf1dcb9ce094abcf7f68c3450a1d4e3a8e917dd621edb4ebe0"],"0x000000000000000000000000372bd88bd7fc79f6ba11c553107a75417110f5ad","0x000000000000000000000000000000000000000000000000000000000000a869"]}
----------------------------------------------------------------------------------------------------
8     ║ ✖    | 0          | 10051ms         ║ ✔    | 0          | 53ms            ║ {"fromBlock":884404,"toBlock":888655,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0x3F3440A7ac383C0cbFa088A2A7D2D945f8331De1"],"topics":[["0x2caecd17d02f56fa897705dcc740da2d237c373f70686f4e0d9bd3bf0400ea7a","0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x4c209b5fc8ad50758f13e2e1088ba56a560dff690a1c6fef26394f4c03821c4f"],["0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d","0x0000000000000000000000003f3440a7ac383c0cbfa088a2a7d2d945f8331de1"],["0x0000000000000000000000003f3440a7ac383c0cbfa088a2a7d2d945f8331de1","0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d"]]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
9     ║ ✖    | 0          | 2068ms          ║ ✖    | 0          | 3503ms          ║ {"fromBlock":914482,"toBlock":919251}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
10    ║ ✖    | 0          | 1865ms          ║ ✖    | 0          | 3806ms          ║ {"fromBlock":815908,"toBlock":820890}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
11    ║ ✖    | 0          | 10053ms         ║ ✔    | 629        | 1328ms          ║ {"fromBlock":64620,"toBlock":67939}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
12    ║ ✔    | 0          | 9545ms          ║ ✔    | 0          | 54ms            ║ {"fromBlock":3058958,"toBlock":3061483,"address":["0x303F9330093d0E09f5e16e9af5cDdcfb416623CA","0xdEc1bc71bf91431D60eF2742f412DCd1c5A204B8","0xEb2e970cfCb087e7826862880A1F45FB3020bD87"],"topics":["0xa7fc99ed7617309ee23f63ae90196a1e490d362e6f6a547a59bc809ee2291782"]}
----------------------------------------------------------------------------------------------------
13    ║ ✔    | 4          | 6864ms          ║ ✔    | 4          | 963ms           ║ {"fromBlock":963071,"toBlock":966361,"topics":[["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"],["0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000f7571bb707b44c11e7225dd6f84b34aa96ea118d"],["0x000000000000000000000000662e19154aacddd6dd08605f69c7f5a7627062d5","0x00000000000000000000000081e42baeee09de2880d4a8842edb1911755ac48d"]]}
----------------------------------------------------------------------------------------------------
14    ║ ✖    | 0          | 10070ms         ║ ✖    | 0          | 8013ms          ║ {"fromBlock":2268388,"toBlock":2273200}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
15    ║ ✔    | 0          | 8921ms          ║ ✔    | 0          | 51ms            ║ {"fromBlock":618271,"toBlock":622930,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xF516bDBd32A0B4CB07D21F26B9185B336502027f"]}
----------------------------------------------------------------------------------------------------

5001 to 10000 blocks range request

----------------------------------------------------------------------------------------------------
      ║ RocksDB                             ║ SqlDB                               ║
----------------------------------------------------------------------------------------------------
#     ║ OK   | Results    | Time            ║ OK   | Results    | Time            ║ Query
----------------------------------------------------------------------------------------------------
1     ║ ✖    | 0          | 10065ms         ║ ✖    | 0          | 6951ms          ║ {"fromBlock":2212482,"toBlock":2220814}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
2     ║ ✖    | 0          | 10049ms         ║ ✔    | 0          | 54ms            ║ {"fromBlock":1184158,"toBlock":1189949,"address":["0x0115dbCaA30f0350Ec49977874f6f4ef4297D377","0x57c94e147D43f96ecCEB7a978702e71A047c3Bdf","0x7e5d75a58Da31efdBeA997244Bd99e85ff871798"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
3     ║ ✖    | 0          | 10047ms         ║ ✔    | 0          | 55ms            ║ {"fromBlock":1446562,"toBlock":1452288,"address":["0x4063e6FBB413AA3cEFC6963DbfF937889D6C79F6","0x820F510F1B3E468024761247aDf6e3Be270f3475","0x74775d2A767DBDf062b5C587F87C54396634D11c"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
4     ║ ✖    | 0          | 3606ms          ║ ✖    | 0          | 3155ms          ║ {"fromBlock":893753,"toBlock":898763}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
5     ║ ✖    | 0          | 10058ms         ║ ✔    | 0          | 58ms            ║ {"fromBlock":2609330,"toBlock":2619041,"address":["0xf02d804b19b0665690f6B312691B2eb8F80Cd3b8","0x5FdCf221e987966b296714953fb6952AfC24aeFE","0xD05526a73bf45DaDf7f9A99dCCeAC23C2D43C6C7"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
6     ║ ✖    | 0          | 10049ms         ║ ✔    | 0          | 52ms            ║ {"fromBlock":673708,"toBlock":680240,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xF516bDBd32A0B4CB07D21F26B9185B336502027f"],"topics":["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
7     ║ ✖    | 0          | 10058ms         ║ ✖    | 0          | 10085ms         ║ {"fromBlock":1763681,"toBlock":1770114}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
8     ║ ✖    | 0          | 10056ms         ║ ✔    | 7239       | 7057ms          ║ {"fromBlock":2746236,"toBlock":2752570}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
9     ║ ✖    | 0          | 3206ms          ║ ✖    | 0          | 4589ms          ║ {"fromBlock":1975341,"toBlock":1983732}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
10    ║ ✖    | 0          | 10061ms         ║ ✔    | 0          | 54ms            ║ {"fromBlock":1315877,"toBlock":1322494,"address":["0x7e5d75a58Da31efdBeA997244Bd99e85ff871798","0x570214c493D124A67AE34079961367758cBE0450","0x47353EbfB9078894601677b6E2613B59Ba9Ab049"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
11    ║ ✖    | 0          | 10050ms         ║ ✔    | 0          | 51ms            ║ {"fromBlock":924213,"toBlock":931249,"address":["0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0xa8D30a70eFe49D9d459f0e25725d7F84D93C976e"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
12    ║ ✖    | 0          | 10052ms         ║ ✖    | 0          | 10082ms         ║ {"fromBlock":1016191,"toBlock":1024431}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
Error - SQL    : Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
13    ║ ✖    | 0          | 3471ms          ║ ✖    | 0          | 4709ms          ║ {"fromBlock":2338869,"toBlock":2347064}
Error - RocksDB: Returned error: query returned more than 10000 results
Error - SQL    : Returned error: query returned more than 10000 results
----------------------------------------------------------------------------------------------------
14    ║ ✖    | 0          | 10053ms         ║ ✔    | 0          | 52ms            ║ {"fromBlock":181687,"toBlock":188812,"address":["0x9755F1eA7a6f6058dFc02F95Bc932C3c261c1f31","0x42B15a11cc295BE6f97Aa4518E850B064b64FB11"]}
Error - RocksDB: Returned error: query timeout of 10 seconds exceeded
----------------------------------------------------------------------------------------------------
15    ║ ✖    | 0          | 10059ms         ║ ✔    | 0          | 55ms            ║ {"fromBlock":731617,"toBlock":738527,"address":["0x42B15a11cc295BE6f97Aa4518E850B064b64FB11","0xF516bDBd32A0B4CB07D21F26B9185B336502027f","0x6FFA2707eA1fe81c8DAfA755E74f4961333B75fA"],"topics":[["0x8d468b5f823f8d38322e9c4433d184adf453fd3eaa28cef280056aa0664981f0","0x0109fc6f55cf40689f02fbaad7af7fe7bbac8a3d2186600afc7d3e10cac60271","0x92e98423f8adac6e64d0608e519fd1cefb861498385c6dee70d58fc926ddc68c","0x0559884fd3a460db3073b7fc896cc77986f16e378210ded43186175bf646fc5f","0xfe25c73e3b9089fac37d55c4c7efcba6f04af04cebd2fc4d6d7dbb07e1e5234f"],["0x0000000000000000000000000000000000000000000000000000000000000d34","0x0000000000000000000000000000000000000000000000000000000789159345","0x000000000000000000000000000000000000000000000002b5e3af169d628780"],["0x0000000000000000000000006ffa2707ea1fe81c8dafa755e74f4961333b75fa","0x0000000000000000000000000000000000000000000000000000000000000d34"],"0x0000000000000000000000006ffa2707ea1fe81c8dafa755e74f4961333b75fa"]}

# Conflicts: # Cargo.lock # client/db/Cargo.toml # client/db/src/kv/utils.rs # client/db/src/lib.rs # primitives/storage/Cargo.toml # template/node/src/cli.rs # template/node/src/command.rs # template/node/src/service.rs

sorpaas · 2023-04-19T19:06:53Z

client/db/src/sql/mod.rs

+	}
+
+	/// Create the Sqlite database if it does not already exist.
+	async fn create_database_if_not_exists(pool: &SqlitePool) -> Result<SqliteQueryResult, Error> {


What's your plan for database migrations in the future?

If it's planned to do it manually, then please add a metadata table that will store the version information.

Or you can consider using a framework, please just take a look making sure it's compatible. Some of the migration tools will store all migrations applied in a migration table (and just check missing ones to apply). In that case we can leave this until the first migration is written.

Yeah we deferred it until the first migration needs to occur. In that case the absence of the versions table will serve as version 0 and any subsequent migrations can be built on top of it.

sorpaas

LGTM other than issues with future db migrations.

boundless-forest · 2023-04-20T07:52:25Z

client/db/Cargo.toml

@@ -1,38 +1,53 @@
 [package]
 name = "fc-db"
 version = "2.0.0-dev"
+authors = ["Parity Technologies <admin@parity.io>"]


This can be replaced with workspace setting, like other crates.

boundless-forest · 2023-04-20T08:06:10Z

client/db/src/kv/mod.rs

+		_topics: Vec<Vec<Option<H256>>>,
+	) -> Result<Vec<crate::FilteredLog>, String> {
+		// KeyValue db does not index logs
+		unimplemented!();


Will this cause potential panic for the kv backend client? I noticed that this is used in the filter_range_logs_indexed

Suggest returning empty vector.

This is already checked by whether the backend is indexed or not here so will not cause a panic. This ensures that the indexed logs are only returned for supported backends while the unsupported ones use the older filter_range_logs function instead.

Why not return error instead?

That can be a possibility to return Err("not supported")

Yeah I think that would be preferred than unimplemented.

nbaztec · 2023-04-20T12:47:01Z

Changes applied @sorpaas @boundless-forest

boundless-forest · 2023-04-20T13:33:57Z

Well done! So many changes, hard to review. I spend the whole afternoon reviewing them. Overall, LGTM with so many test cases are added. I am worried a bit about the migration solution in the future.

tgmichel · 2023-04-20T13:46:02Z

I am worried a bit about the migration solution in the future

Can you explain with more detail what is your concern?

boundless-forest · 2023-04-21T06:52:23Z

I am worried a bit about the migration solution in the future

Can you explain with more detail what is your concern?

Similar to the oponion, mainly about the data migration or fix solution if we adjust the data schema later when doing bug fix. This issue also existed when using kv db backend. Let leave it alone now.

nbaztec · 2023-04-21T08:23:26Z

Similar to the oponion, mainly about the data migration or fix solution if we adjust the data schema later when doing bug fix. This issue also existed when using kv db backend. Let leave it alone now.

That's a valid concern, I replied to it earlier on the prospective plans in #883 (comment)

Hope that suffices.

tgmichel · 2023-05-05T08:50:48Z

@sorpaas is there something preventing this to be merged?

sorpaas · 2023-05-16T10:21:41Z

Please pull master.

sorpaas · 2023-05-16T12:50:25Z

Something seems wrong with the CI. Please push an empty commit!

nazar-pc · 2023-06-22T10:58:57Z

Maybe consider making this feature optional? It pulls way too many dependencies, including OpenSSL.

tgmichel · 2023-06-22T11:48:47Z

@nazar-pc please create an issue referencing this PR

tgmichel added 30 commits October 21, 2022 17:01

Receipt log sql indexer

34e4f54

Wip unify worker and backend

8e56f99

Wip lay out service

2a23149

Remove old crate

50a510e

mapping sync works (wip genesis)

8248984

Genesis block indexing

fa0bb5c

General rpc update to async

baf725c

WIP filter logs sql

2cd155d

Fix mapping-sync tests

487642d

WIP filter logs sql

d269e21

Test filter_logs

d41542c

Fix tests

dcea8dd

Make it build and pass tests again

25a7992

Use sql backend in rpc

b4461fe

Run ts tests with sql + ci

fcbfd4b

Merge branch 'master' into tgm-fc-log-indexer

8b9f408

# Conflicts: # Cargo.lock # client/db/Cargo.toml # client/db/src/kv/utils.rs # client/db/src/lib.rs # primitives/storage/Cargo.toml # template/node/src/cli.rs # template/node/src/command.rs # template/node/src/service.rs

Fix key-value

22a10ff

editorconfig

9d7045e

taplo

f25de3d

clippy

bd2f945

Always fire interval first

f00932c

Merge branch 'master' into tgm-fc-log-indexer

f1b9104

Try move indexing

8f67251

Always fire interval first

e85646e

await on logs task

9a0931c

wal journal mode

63d4f70

Additional logging

c50f35f

Add pragmas + optimize vec insert

a4c23e2

More logs

fd5596e

More logs

f4c2d26

sorpaas reviewed Apr 19, 2023

View reviewed changes

boundless-forest reviewed Apr 20, 2023

View reviewed changes

apply code suggestions

a9df993

sorpaas approved these changes Apr 21, 2023

View reviewed changes

boundless-forest approved these changes Apr 21, 2023

View reviewed changes

boundless-forest mentioned this pull request Apr 26, 2023

eth_getLogs performance issue darwinia-network/darwinia#1015

Closed

Merge branch 'master' into tgm-fc-log-indexer

811c9cb

nbaztec and others added 5 commits May 16, 2023 16:38

Merge branch 'master' into tgm-fc-log-indexer

7c0f200

rebase

a153293

disallow SyncStrategy::Normal for the time being

63aeb8c

emit notifications from sql backend

b79534f

update tests

67ce0f8

sorpaas approved these changes May 31, 2023

View reviewed changes

sorpaas merged commit aa57359 into polkadot-evm:master May 31, 2023

nazar-pc mentioned this pull request Jun 16, 2023

Upgrade Substrate (with core-evm-relay removed) autonomys/subspace#1549

Merged

1 task

nazar-pc mentioned this pull request Jun 22, 2023

Make sql backend optional #1086

Closed

Dinonard mentioned this pull request Jul 19, 2023

Feat/polkadot v0.9.43 uplift AstarNetwork/Astar#978

Merged

16 tasks

dmitrylavrenov mentioned this pull request Oct 16, 2023

Bump to polkadot-sdk v1.1.0 as a base for substrate related deps humanode-network/humanode#803

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql backend w/ receipt log indexing #883

sql backend w/ receipt log indexing #883

tgmichel commented Oct 28, 2022 •

edited

Loading

sorpaas Apr 19, 2023

nbaztec Apr 20, 2023

sorpaas left a comment

boundless-forest Apr 20, 2023

boundless-forest Apr 20, 2023 •

edited

Loading

nbaztec Apr 20, 2023 •

edited

Loading

boundless-forest Apr 20, 2023

sorpaas Apr 20, 2023

nbaztec Apr 20, 2023

sorpaas Apr 21, 2023

nbaztec commented Apr 20, 2023

boundless-forest commented Apr 20, 2023

tgmichel commented Apr 20, 2023

boundless-forest commented Apr 21, 2023

nbaztec commented Apr 21, 2023

tgmichel commented May 5, 2023

sorpaas commented May 16, 2023

sorpaas commented May 16, 2023

nazar-pc commented Jun 22, 2023

tgmichel commented Jun 22, 2023

sql backend w/ receipt log indexing #883

sql backend w/ receipt log indexing #883

Conversation

tgmichel commented Oct 28, 2022 • edited Loading

sorpaas Apr 19, 2023

Choose a reason for hiding this comment

nbaztec Apr 20, 2023

Choose a reason for hiding this comment

sorpaas left a comment

Choose a reason for hiding this comment

boundless-forest Apr 20, 2023

Choose a reason for hiding this comment

boundless-forest Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

nbaztec Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

boundless-forest Apr 20, 2023

Choose a reason for hiding this comment

sorpaas Apr 20, 2023

Choose a reason for hiding this comment

nbaztec Apr 20, 2023

Choose a reason for hiding this comment

sorpaas Apr 21, 2023

Choose a reason for hiding this comment

nbaztec commented Apr 20, 2023

boundless-forest commented Apr 20, 2023

tgmichel commented Apr 20, 2023

boundless-forest commented Apr 21, 2023

nbaztec commented Apr 21, 2023

tgmichel commented May 5, 2023

sorpaas commented May 16, 2023

sorpaas commented May 16, 2023

nazar-pc commented Jun 22, 2023

tgmichel commented Jun 22, 2023

tgmichel commented Oct 28, 2022 •

edited

Loading

boundless-forest Apr 20, 2023 •

edited

Loading

nbaztec Apr 20, 2023 •

edited

Loading