[Nakamoto] 5 Second Blocks; Requirements and Limitations

[AshtonStephens]

This is a discussion to provide research pertaining to transaction limitations for 5 second blocks and the core requirements for the blocks themselves. During a call on 11/10/23 we determined that we need to iron out the transaction count and properties within the 5s blocks and address any limitations not related to block latency. In the end, it is possible that I/O will be a bottleneck.

The existing research around I/O suggests that we can improve I/O significantly with not much work: stacks-network/stacks-core#3777

The questions to answer:

• What are the core transaction expectations for users of stacks for a single block
• What are the other transaction bottlenecks

[AshtonStephens]

Some info from @obycode during the SIP review session: "Read count is almost always the budget that is hit first. Recently, the runtime budget is close to being hit due to expensive oracle transactions in every block. Clarity-Wasm will definitely help with the runtime budget. I’m unclear on impact to read/write budgets."

[philiphacks]

Just wanted to add here that this is also a current concern when analysing blocks & the mempool. From a technical point of view, I understand that the read count can get hit, but from a UX perspective it is not clear how transactions are prioritised or sorted, and the fees returned by Leather do not always include TXs within even ~5 blocks

Something to address imho - not sure how it would relate to fast blocks

[AshtonStephens]

A note about the importance of low latency on any Stacks API by @rafaelcr: If miners are producing blocks every 5 seconds and everything is fine with the nodes, but there's a problem on the API that we stop ingesting blocks for any reason and then it startsfalling behind on following the chain, then the impression would be that stacks is actually halted or slow when it isn't.

[AshtonStephens]

Two main improvements to enable 5s blocks:

• Need to improve MARF - increases read budget
• Need to implement ClarityWASM - Increases compute budget

[AshtonStephens]

MARF: We need to optimize the way we store data on the SQLite database. @cylewitruk Will have the best context on this.

[cylewitruk]

Re: SQLite, there are a few issues I created over the last days which can be included in this, pasting them here for ease:

• [Clarity VM] Metadata keys stored in metadata_table are extremely verbose stacks-network/stacks-core#4034 (Clarity SQLite db)
• [Clarity VM] Contract src is stored uncompressed in the Clarity DB stacks-network/stacks-core#4035 (Clarity SQLite db)
• [Clarity VM] Contract AST is stored as plain-text JSON stacks-network/stacks-core#4036 (Clarity SQLite db)
• [Database: General] All hashes in node databases are stored as hex strings instead of bytes stacks-network/stacks-core#4037 (All SQLite db's)
• [SQLite] Explore SQLite options which may help to improve performance stacks-network/stacks-core#4052 (All SQLite db's)

Prior to embarking on any of these improvements it's important to have benchmarking in place so that we can measure improvements (storage, CPU cycles, etc.).