docs(book): start documenting ExExes

book/SUMMARY.md

@@ -73,4 +73,7 @@
     - [`reth recover`](./cli/reth/recover.md)
       - [`reth recover storage-tries`](./cli/reth/recover/storage-tries.md)
 - [Developers](./developers/developers.md) <!-- CLI_REFERENCE END -->
+   - [Execution Extensions](./developers/exex/exex.md)
+      - [How do ExExes work?](./developers/exex/how-it-works.md)
+      - [Hello World](./developers/exex/hello-world.md)
    - [Contribute](./developers/contribute.md)

book/developers/developers.md

@@ -1,3 +1,3 @@
 # Developers
 
-Reth is composed of several crates that can be used in standalone projects. If you are interested in using one or more of the crates, you can get an overview of them in the [developer docs](https://github.com/paradigmxyz/reth/tree/main/docs), or take a look at the [crate docs](https://paradigmxyz.github.io/reth/docs).
+Reth is composed of several crates that can be used in standalone projects. If you are interested in using one or more of the crates, you can get an overview of them in the [developer docs](https://github.com/paradigmxyz/reth/tree/main/docs), or take a look at the [crate docs](https://paradigmxyz.github.io/reth/docs).

book/developers/exex/exex.md

@@ -0,0 +1,19 @@
+# Execution Extensions (ExEx)
+
+## What are Execution Extensions?
+
+Execution Extensions allow developers to build their own infrastructure that relies on Reth
+as a base for driving the chain (be it [Ethereum](../../run/mainnet.md) or [OP Stack](../../run/optimism.md)) forward.
+
+An Execution Extension is a task that derives its state from changes in Reth's state.
+Some examples of such state derivations are rollups, bridges, and indexers.
+
+Read more about things you can build with Execution Extensions in the [Paradigm blog](https://www.paradigm.xyz/2024/05/reth-exex).
+
+## How do I build an Execution Extension?
+
+Let's dive into how to build our own ExEx (short for Execution Extension) from scratch, add tests for it,
+and run it on the Holesky testnet.
+
+1. [How do ExExes work?](./how-it-works.md)
+1. [Hello World](./hello-world.md)

book/developers/exex/hello-world.md

@@ -0,0 +1,162 @@
+# Hello World
+
+Let's write a simple "Hello World" ExEx that emits a log every time a new chain of blocks is committed, reverted, or reorged.
+
+### Create a project
+
+First, let's create a new project for our ExEx
+
+```console
+cargo new --bin my-exex
+cd my-exex
+```
+
+And add Reth as a dependency in `Cargo.toml`
+
+```toml
+[package]
+name = "my-exex"
+version = "0.1.0"
+edition = "2021"
+
+[dependencies]
+reth = { git = "https://github.com/paradigmxyz/reth.git" } # Reth
+reth-exex = { git = "https://github.com/paradigmxyz/reth.git" } # Execution Extensions
+reth-node-ethereum = { git = "https://github.com/paradigmxyz/reth.git" } # Ethereum Node implementation
+reth-tracing = { git = "https://github.com/paradigmxyz/reth.git" } # Logging
+eyre = "0.6" # Easy error handling
+```
+
+### Default Reth node
+
+Now, let's jump to our `main.rs` and start by initializing and launching a default Reth node
+
+```rust,norun,noplayground,ignore
+use reth_node_ethereum::EthereumNode;
+
+fn main() -> eyre::Result<()> {
+    reth::cli::Cli::parse_args().run(|builder, _| async move {
+        let handle = builder.node(EthereumNode::default()).launch().await?;
+
+        handle.wait_for_node_exit().await
+    })
+}
+```
+
+You can already test that it works by running the binary and initializing the Holesky node in a custom datadir
+(to not interfere with any instances of Reth you already have on your machine):
+
+```console
+$ cargo run -- init --chain holesky --datadir data
+
+2024-06-12T16:48:06.420296Z  INFO reth init starting
+2024-06-12T16:48:06.422380Z  INFO Opening storage db_path="data/db" sf_path="data/static_files"
+2024-06-12T16:48:06.432939Z  INFO Verifying storage consistency.
+2024-06-12T16:48:06.577673Z  INFO Genesis block written hash=0xb5f7f912443c940f21fd611f12828d75b53
+4364ed9e95ca4e307729a4661bde4
+```
+
+### Simplest ExEx
+
+The simplest ExEx is just an async function that never returns. We need to install it into our node
+
+```rust,norun,noplayground,ignore
+use reth::api::FullNodeComponents;
+use reth_exex::{ExExContext, ExExEvent, ExExNotification};
+use reth_node_ethereum::EthereumNode;
+use reth_tracing::tracing::info;
+
+async fn my_exex<Node: FullNodeComponents>(mut ctx: ExExContext<Node>) -> eyre::Result<()> {
+    loop {}
+}
+
+fn main() -> eyre::Result<()> {
+    reth::cli::Cli::parse_args().run(|builder, _| async move {
+        let handle = builder
+            .node(EthereumNode::default())
+            .install_exex("my-exex", |ctx| async move { Ok(my_exex(ctx)) })
+            .launch()
+            .await?;
+
+        handle.wait_for_node_exit().await
+    })
+}
+```
+
+See that unused `_ctx`? That's the context that we'll use to listen to new notifications coming from the main node,
+and send events back to it. It also contains all components that the node exposes to the ExEx.
+
+Currently, our ExEx does absolutely nothing by running an infinite loop in an async function that never returns.
+
+<div class="warning">
+
+It's important that the future returned by the ExEx (`my_exex`) never resolves.
+
+If you try running a node with an ExEx that exits, the node will exit as well.
+
+</div>
+
+### Hello World ExEx
+
+Now, let's extend our simplest ExEx and start actually listening to new notifications, log them, and send events back to the main node
+
+```rust,norun,noplayground,ignore
+use reth::api::FullNodeComponents;
+use reth_exex::{ExExContext, ExExEvent, ExExNotification};
+use reth_node_ethereum::EthereumNode;
+use reth_tracing::tracing::info;
+
+async fn my_exex<Node: FullNodeComponents>(mut ctx: ExExContext<Node>) -> eyre::Result<()> {
+    while let Some(notification) = ctx.notifications.recv().await {
+        match &notification {
+            ExExNotification::ChainCommitted { new } => {
+                info!(committed_chain = ?new.range(), "Received commit");
+            }
+            ExExNotification::ChainReorged { old, new } => {
+                info!(from_chain = ?old.range(), to_chain = ?new.range(), "Received reorg");
+            }
+            ExExNotification::ChainReverted { old } => {
+                info!(reverted_chain = ?old.range(), "Received revert");
+            }
+        };
+
+        if let Some(committed_chain) = notification.committed_chain() {
+            ctx.events
+                .send(ExExEvent::FinishedHeight(committed_chain.tip().number))?;
+        }
+    }
+
+    Ok(())
+}
+
+fn main() -> eyre::Result<()> {
+    reth::cli::Cli::parse_args().run(|builder, _| async move {
+        let handle = builder
+            .node(EthereumNode::default())
+            .install_exex("my-exex", |ctx| async move { Ok(my_exex(ctx)) })
+            .launch()
+            .await?;
+
+        handle.wait_for_node_exit().await
+    })
+}
+```
+
+Woah, there's a lot of new stuff here! Let's go through it step by step:
+
+- First, we've added a `while let Some(notification) = ctx.notifications.recv().await` loop that waits for new notifications to come in.
+   - The main node is responsible for sending notifications to the ExEx, so we're waiting for them to come in.
+- Next, we've added a `match &notification { ... }` block that matches on the type of the notification.
+   - In each case, we're logging the notification and the corresponding block range, be it a chain commit, revert, or reorg.
+- Finally, we're checking if the notification contains a committed chain, and if it does, we're sending a `ExExEvent::FinishedHeight` event back to the main node using the `ctx.events.send` method.
+
+<div class="warning">
+
+Sending an `ExExEvent::FinishedHeight` event is a very important part of every ExEx.
+
+It's the only way to communicate to the main node that the ExEx has finished processing the specified height
+and it's safe to prune the associated data.
+
+</div>
+
+What we've arrived at is the [minimal ExEx example](https://github.com/paradigmxyz/reth/blob/b8cd7be6c92a71aea5341cdeba685f124c6de540/examples/exex/minimal/src/main.rs) that we provide in the Reth repository.

book/developers/exex/how-it-works.md

@@ -0,0 +1,26 @@
+# How do ExExes work?
+
+ExExes are just [Futures](https://doc.rust-lang.org/std/future/trait.Future.html) that run indefinitely alongside Reth
+– as simple as that.
+
+An ExEx is usually driven by and acts on new notifications about chain commits, reverts, and reorgs, but it can span beyond that.
+
+They are installed into the node by using the [node builder](https://reth.rs/docs/reth/builder/struct.NodeBuilder.html).
+Reth manages the lifecycle of all ExExes, including:
+- Polling ExEx futures
+- Sending [notifications](https://reth.rs/docs/reth_exex/enum.ExExNotification.html) about new chain, reverts,
+  and reorgs from historical and live sync
+- Processing [events](https://reth.rs/docs/reth_exex/enum.ExExEvent.html) emitted by ExExes
+- Pruning (in case of a full or pruned node) only the data that have been processed by all ExExes
+- Shutting ExExes down when the node is shut down
+
+## Pruning
+
+Pruning deserves a special mention here.
+
+ExExes **SHOULD** emit an [`ExExEvent::FinishedHeight`](https://reth.rs/docs/reth_exex/enum.ExExEvent.html#variant.FinishedHeight)
+event to signify what blocks have been processed. This event is used by Reth to determine what state can be pruned.
+
+An ExEx will only receive notifications for block numbers greater than the block in the most recently emitted `FinishedHeight` event.
+
+To clarify: if an ExEx emits `ExExEvent::FinishedHeight(0)` it will receive notifications for any `block_number > 0`.