Skip to content

Commit

Permalink
chore: add external-node docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@itsacoyote
itsacoyote committed Apr 19, 2024
1 parent c543156 commit dee585e
Show file tree
Hide file tree
Showing 8 changed files with 644 additions and 2 deletions.
162 changes: 160 additions & 2 deletions content/20.external-node/00.index.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,163 @@
---
title: zkSync Era External Node
title: Introduction
description:
---

Welcome to the external node pages.
::callout{icon="i-heroicons-information-circle" color="blue"}
For local testing, we recommend setting up an in-memory node and forking mainnet.
::

This documentation explains the basics of the zkSync Era External Node.

## Disclaimers

- The external node is in the alpha phase, and should be used with caution.
- While in alpha, the EN is dependent on a DB snapshot in order to run that is not yet publicly available.
- The EN is a read-only replica of the main node. We are currently working on decentralizing our infrastructure by
creating a consensus node. The EN is not going to be the consensus node.

## What is the External Node?

The external node (herein EN) is a read-replica of the main (centralized) node that can be run by external parties. It
functions by fetching data from the zkSync API and re-applying transactions locally, starting from the genesis block.
The EN shares most of its codebase with the main node. Consequently, when it re-applies transactions, it does so exactly
as the main node did in the past.

In Ethereum terms, the current state of the EN represents an archive node, providing access to the entire history of the
blockchain.

## High-level Overview

At a high level, the EN can be seen as an application that has the following modules:

- API server that provides the publicly available Web3 interface.
- Synchronization layer that interacts with the main node and retrieves transactions and blocks to re-execute.
- Sequencer component that actually executes and persists transactions received from the synchronization layer.
- Several checker modules that ensure the consistency of the EN state.

With the EN, you are able to:

- Locally recreate and verify the zkSync Era mainnet/testnet state.
- Interact with the recreated state in a trustless way (in a sense that the validity is locally verified, and you should
not rely on a third-party API zkSync Era provides).
- Use the Web3 API without having to query the main node.
- Send L2 transactions (that will be proxied to the main node).

With the EN, you _can not_:

- Create L2 blocks or L1 batches on your own.
- Generate proofs.
- Submit data to L1.

A more detailed overview of the EN's components is provided in the components section.

## API Overview

API exposed by the EN strives to be Web3-compliant.
If some method is exposed but behaves differently compared to
Ethereum, it should be considered a bug.
Please [report](https://zksync.io/contact) such cases.

### `eth_` Namespace

Data getters in this namespace operate in the L2 space: require/return L2 block numbers, check balances in L2, etc.

Available methods:

| Method | Notes |
| ----------------------------------------- | ------------------------------------------------------------------------- |
| `eth_blockNumber` | |
| `eth_chainId` | |
| `eth_call` | |
| `eth_estimateGas` | |
| `eth_gasPrice` | |
| `eth_newFilter` | Maximum amount of installed filters is configurable |
| `eth_newBlockFilter` | Same as above |
| `eth_newPendingTransactionsFilter` | Same as above |
| `eth_uninstallFilter` | |
| `eth_getLogs` | Maximum amount of returned entities can be configured |
| `eth_getFilterLogs` | Same as above |
| `eth_getFilterChanges` | Same as above |
| `eth_getBalance` | |
| `eth_getBlockByNumber` | |
| `eth_getBlockByHash` | |
| `eth_getBlockTransactionCountByNumber` | |
| `eth_getBlockTransactionCountByHash` | |
| `eth_getCode` | |
| `eth_getStorageAt` | |
| `eth_getTransactionCount` | |
| `eth_getTransactionByHash` | |
| `eth_getTransactionByBlockHashAndIndex` | |
| `eth_getTransactionByBlockNumberAndIndex` | |
| `eth_getTransactionReceipt` | |
| `eth_protocolVersion` | |
| `eth_sendRawTransaction` | |
| `eth_syncing` | EN is considered synced if it's less than 11 blocks behind the main node. |
| `eth_coinbase` | Always returns a zero address |
| `eth_accounts` | Always returns an empty list |
| `eth_getCompilers` | Always returns an empty list |
| `eth_hashrate` | Always returns zero |
| `eth_getUncleCountByBlockHash` | Always returns zero |
| `eth_getUncleCountByBlockNumber` | Always returns zero |
| `eth_mining` | Always returns false |

### PubSub

Only available on the WebSocket servers.

Available methods:

| Method | Notes |
| ------------------ | ----------------------------------------------- |
| `eth_subscribe` | Maximum amount of subscriptions is configurable |
| `eth_subscription` | |

### `net_` Namespace

Available methods:

| Method | Notes |
| ---------------- | -------------------- |
| `net_version` | |
| `net_peer_count` | Always returns 0 |
| `net_listening` | Always returns false |

### `web3_` Namespace

Available methods:

| Method | Notes |
| -------------------- | ----- |
| `web3_clientVersion` | |

### `debug` namespace

The `debug` namespace gives access to several non-standard RPC methods, which will allow developers to inspect and debug
calls and transactions.

This namespace is disabled by default and can be configured via setting `EN_API_NAMESPACES` as described in the example config.

Available methods:

| Method | Notes |
| -------------------------- | ----- |
| `debug_traceBlockByNumber` | |
| `debug_traceBlockByHash` | |
| `debug_traceCall` | |
| `debug_traceTransaction` | |

### `zks` namespace

This namespace contains rollup-specific extensions to the Web3 API.
Note that _only methods_ specified in the documentation are considered public.
There may be other methods exposed in this namespace, but undocumented
methods come without any kind of stability guarantees and can be changed or removed without notice.

Always refer to the documentation linked above to see the list of stabilized methods in this namespace.

[zks_docs](/build/api-reference)

### `en` namespace

This namespace contains methods that external nodes call on the main node while syncing. If this namespace is enabled
other ENs can sync from this node.
88 changes: 88 additions & 0 deletions content/20.external-node/10.component-breakdown.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: EN Components
description:
---

This section contains an overview of the EN's main components.

## API

The EN can serve both the HTTP and the WS Web3 API, as well as PubSub.
Whenever possible, it provides data based on the local state, with a few exceptions:

- Submitting transactions: Since it is a read replica,
submitted transactions are proxied to the main node,
and the response is returned from the main node.
- Querying transactions: The EN is not aware of the main node's mempool,
and it does not sync rejected transactions.
Therefore, if a local lookup for a transaction or its receipt fails,
the EN will attempt the same query on the main node.

Apart from these cases, the API does not depend on the main node.
Even if the main node is temporarily unavailable, the EN can continue to serve the state it has locally.

## Fetcher

The Fetcher component is responsible for maintaining synchronization between the EN and the main node.
Its primary task is to fetch new blocks in order to update the local chain state.
However, its responsibilities extend beyond that.
For instance, the Fetcher is also responsible for keeping track of L1 batch statuses.
This involves monitoring whether locally applied batches have been committed, proven, or executed on L1.

It is worth noting that in addition to fetching the _state_, the EN also retrieves the L1 gas price from the main node
for the purpose of estimating fees for L2 transactions (since this also happens based on the local state).
This information is necessary to ensure that gas estimations are performed in the exact same manner as the main node,
thereby reducing the chances of a transaction not being included in a block.

## State Keeper / VM

The State Keeper component serves as the "sequencer" part of the node.
It shares most of its functionality with the main node, with one key distinction.
The main node retrieves transactions from the mempool and has the authority to decide when a specific L2 block or L1 batch should be sealed.
On the other hand, the EN retrieves transactions from the queue populated by the Fetcher and seals the corresponding blocks/batches
based on the data obtained from the Fetcher queue.

The actual execution of batches takes place within the VM, which is identical in both the Main and External nodes.

## Reorg Detector

In zkSync Era, it is theoretically possible for L1 batches to be reverted before the corresponding "execute" operation
is applied on L1, that is before the block is [final](https://docs.zksync.io/zk-stack/concepts/finality.html).
Such situations are highly uncommon and typically occur due to significant issues:
e.g. a bug in the sequencer implementation preventing L1 batch commitment.
Prior to batch finality, the zkSync operator can perform a rollback,
reverting one or more batches and restoring the blockchain state to a previous point.
Finalized batches cannot be reverted at all.

However, even though such situations are rare, the EN must handle them correctly.

To address this, the EN incorporates a Reorg Detector component.
This module keeps track of all L1 batches that have not yet been finalized.
It compares the locally obtained state root hashes with those provided by the main node's API.
If the root hashes for the latest available L1 batch do not match,
the Reorg Detector searches for the specific L1 batch responsible for the divergence.
Subsequently, it rolls back the local state and restarts the node.
Upon restart, the EN resumes normal operation.

## Consistency Checker

The main node API serves as the primary source of information for the EN.
However, relying solely on the API may not provide sufficient security since the API data could potentially be incorrect due to various reasons.
The primary source of truth for the rollup system is the L1 smart contract.
Therefore, to enhance the security of the EN, each L1 batch undergoes cross-checking against
the L1 smart contract by a component called the Consistency Checker.

When the Consistency Checker detects that a particular batch has been sent to L1,
it recalculates a portion of the input known as the "block commitment" for the L1 transaction.
The block commitment contains crucial data such as the state root and batch number,
and is the same commitment that is used for generating a proof for the batch.
The Consistency Checker then compares the locally obtained commitment with the actual commitment sent to L1.
If the data does not match, it indicates a potential bug in either the main node
or external node implementation or that the main node API has provided incorrect data.
In either case, the state of the EN cannot be trusted, and the EN enters a crash loop until the issue is resolved.

## Health check server

The EN also exposes an additional server that returns HTTP 200 response when the EN is operating normally,
and HTTP 503 response when some of the health checks don't pass (e.g. when the EN is not fully initialized yet).
This server can be used, for example, to implement the readiness probe in an orchestration solution you use.
77 changes: 77 additions & 0 deletions content/20.external-node/20.configuration.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
title: Configuration
description:
---

This document outlines various configuration options for the EN.
Currently, the EN requires the definition of numerous environment variables.
To streamline this process, we provide prepared configs for the zkSync Era - for both mainnet and testnet.
You can use these files as a starting point and modify only the necessary sections.

## Database

The EN uses two databases: PostgreSQL and RocksDB.

PostgreSQL serves as the main source of truth in the EN, so all the API requests fetch the state from there.
The PostgreSQL connection is configured by the `DATABASE_URL`.
Additionally, the `DATABASE_POOL_SIZE` variable defines the size of the connection pool.

RocksDB is used in components where IO is a bottleneck, such as the State Keeper and the Merkle tree.
If possible, it is recommended to use an NVME SSD for RocksDB.
RocksDB requires two variables to be set: `EN_STATE_CACHE_PATH` and `EN_MERKLE_TREE_PATH`, which must point to different directories.

## L1 Web3 client

EN requires a connection to an Ethereum node.
The corresponding env variable is `EN_ETH_CLIENT_URL`.
Make sure to set the URL corresponding to the correct L1 network (L1 mainnet for L2 mainnet and L1 sepolia for L2 testnet).

Note: Currently, the EN makes 2 requests to the L1 per L1 batch,
so the Web3 client usage for a synced node should not be high.
However, during the synchronization phase the new batches would be persisted on the EN quickly,
so make sure that the L1 client won't exceed any limits (e.g. in case you use Infura).

## Exposed ports

The dockerized version of the server exposes the following ports:

- HTTP JSON-RPC: `3060`
- WebSocket JSON-RPC: `3061`
- Prometheus listener: `3322`
- Healthcheck server: `3081`

While the configuration variables for them exist, you are not expected to change them unless you want to use the EN
outside of provided docker environment (not supported at the time of writing).

::callout{icon="i-heroicons-information-circle" color="blue"}
**NOTE**: if the Prometheus port is configured, it must be [scraped](https://prometheus.io/docs/introduction/overview/)
periodically to avoid a memory leak due to a
[bug in an external metrics library](https://github.com/metrics-rs/metrics/issues/245).
If you are not intending to use the metrics, leave this port not configured, and the metrics won't be collected.
::

## API limits

There are variables that allow you to fine-tune the limits of the RPC servers, such as limits on the number of returned
entries or the limit for the accepted transaction size. Provided files contain sane defaults that are recommended for
use, but these can be edited, e.g. to make the EN more/less restrictive.

## JSON-RPC API namespaces

There are 7 total supported API namespaces: `eth`, `net`, `web3`, `debug` - standard ones; `zks` - rollup-specific one;
`pubsub` - a.k.a. `eth_subscribe`; `en` - used by external nodes while syncing. You can configure what namespaces you
want to enable using `EN_API_NAMESPACES` and specifying namespace names in a comma-separated list. By default, all but
the `debug` namespace are enabled.

## Logging and observability

`MISC_LOG_FORMAT` defines the format in which logs are shown: `plain` corresponds to the human-readable format, while
the other option is `json` (recommended for deployments).

`RUST_LOG` variable allows you to set up the logs granularity (e.g. make the EN emit fewer logs). You can read about the
format [here](https://docs.rs/env_logger/0.10.0/env_logger/#enabling-logging).

`MISC_SENTRY_URL` and `MISC_OTLP_URL` variables can be configured to set up Sentry and OpenTelemetry exporters.

If Sentry is configured, you also have to set `EN_SENTRY_ENVIRONMENT` variable to configure the environment in events
reported to sentry.
Loading

0 comments on commit dee585e

Please sign in to comment.