Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Multichain indexing #42

Merged
merged 13 commits into from
Aug 29, 2024
Merged

feat: Multichain indexing #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
e39eea7
chore: Update packages to its latest versions.
brunomenezes Aug 9, 2024
eeb24ef
refactor: Change processor config to use new methods instead of depre…
brunomenezes Aug 9, 2024
6122a05
chore: Add new coverage exclusion mappings and change vitest config e…
brunomenezes Aug 9, 2024
f9b3b6e
feat: Add env var loader for chains and default on its absence
brunomenezes Aug 15, 2024
b07cbfb
feat: Add Chain entity and comment to id schema. Change Fields type g…
brunomenezes Aug 16, 2024
ce86481
feat: Add option to rate-limiting per chain.
brunomenezes Aug 16, 2024
fd2701a
doc: Add information about the environment variables and supported ch…
brunomenezes Aug 16, 2024
3288757
feat: Add chain logic to app-creation and ownership-transferred handl…
brunomenezes Aug 16, 2024
2d4e27c
feat: Update InputAdded handler to have Chain logic. Refactored and a…
brunomenezes Aug 20, 2024
681f0fb
feat: Update logic to support multiple processors running. Also, adde…
brunomenezes Aug 22, 2024
9d2ff80
chore: Removed deprecated property from docker-compose file.
brunomenezes Aug 22, 2024
c736f7c
feat: Added new migration for the new indexes, entity, constraints an…
brunomenezes Aug 22, 2024
86c7efc
feat: Change Typeorm isolationLevel to deal with reads/writes and cle…
brunomenezes Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 4 additions & 9 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,11 @@
DB_NAME=squid
DB_PORT=23798
GQL_PORT=4350
# Check the README file for more information about the available environment variables.
CHAIN_IDS="1, 11155111"
# JSON-RPC node endpoint, both wss and https endpoints are accepted
CHAIN_ID="11155111"
# possible configurations
# RPC_URL_1=
# RPC_URL_11155111=
# RPC_URL_31337=
# RPC_URL_10=
# RPC_URL_11155420=
# RPC_URL_8453=
# RPC_URL_84532=
# RPC_RATE_LIMIT_1=
# NORMALLY FOR USE WITH LOCAL/DEV
# GENESIS_BLOCK_31337=22
# BLOCK_CONFIRMATIONS_31337=1
# BLOCK_CONFIRMATIONS_31337=0
46 changes: 36 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,45 @@
## Rollups Explorer Backend

[![Coverage Status](https://coveralls.io/repos/github/cartesi/rollups-explorer-api/badge.svg?branch=main)](https://coveralls.io/github/cartesi/rollups-explorer-api?branch=main)

Experimentation with subsquid as a new indexing protocol.
This is a Subsquid Indexing protocol API for Cartesi Rollups Contracts and Applications.

That project requires [docker](https://docker.com) to be installed so the backend can be used locally.

### Project structure

the following folders belongs to the `backend`

- abi/
- assets/
- db/
- lib/ (generated when building)
- src/
### Configuration (Environment Variables)

> Supported Chains: 1, 11155111, 10, 11155420, 8453, 84532, 31337

| Variables | Default | Description |
| :-------------------------: | :--------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| `CHAIN_IDS` | `11155111` | Comma separated supported chain ids to be indexed |
| `RPC_URL_1` | `https://rpc.ankr.com/eth` | Endpoint for Mainnet RPC node |
| `RPC_URL_11155111` | `https://rpc.ankr.com/eth_sepolia` | Endpoint for Sepolia RPC node |
| `RPC_URL_31337` | `http://127.0.0.1:8545` | Endpoint for local node |
| `RPC_URL_10` | `https://mainnet.optimism.io` | Endpoint for Optimism Mainnet RPC node |
| `RPC_URL_11155420` | `https://sepolia.optimism.io` | Endpoint for Optimism Sepolia RPC node |
| `RPC_URL_8453` | `https://mainnet.base.org` | Endpoint for Base Mainnet RPC node |
| `RPC_URL_84532` | `https://sepolia.base.org` | Endpoint for Base Sepolia RPC node |
| `RPC_RATE_LIMIT_{CHAIN_ID}` | `undefined` | Option to fine tune concurrent requests by rate limiting the requests to RPC node providers. Replace {CHAIN_ID} with a supported chain id and set a Number e.g. 15. That is good to avoid [429](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429) status code when using free tiers. The default is Full speed |

For use with local devnet for fine tunning and deal with any unexpected changes e.g. Foundry nightly versions.

| Variables | Default | Description |
| :-------------------------: | :-----: | :----------------------------------------------------------------------------------: |
| `GENESIS_BLOCK_31337` | `22` | Genesis block for indexer to start |
| `BLOCK_CONFIRMATIONS_31337` | `1` | Distance from the head block behind which all blocks are considered to be finalized. |

Subsquid Related variables on Development mode. Check the docker-compose and commands.json files.

| Variables | Default | Description |
| :--------: | :---------: | :------------------------: |
| `DB_NAME` | `squid` | Database name |
| `DB_PORT` | `5432` | Database port to be used |
| `DB_SSL` | `true` | Config for SSL connections |
| `DB_PASS` | `postgres` | Database password |
| `DB_HOST` | `localhost` | Database host |
| `DB_USER` | `postgres` | Database user |
| `GQL_PORT` | `4350` | GraphQL API port |

### Development

Expand Down
85 changes: 85 additions & 0 deletions db/migrations/1724312923354-Data.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
module.exports = class Data1724312923354 {
name = 'Data1724312923354'

async up(db) {
await db.query(`CREATE TABLE "chain" ("id" character varying NOT NULL, CONSTRAINT "PK_8e273aafae283b886672c952ecd" PRIMARY KEY ("id"))`)
await db.query(`ALTER TABLE "token" ADD "address" text NOT NULL`)
await db.query(`ALTER TABLE "token" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "erc20_deposit" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "nft" ADD "address" text NOT NULL`)
await db.query(`ALTER TABLE "nft" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "erc721_deposit" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "multi_token" ADD "address" text NOT NULL`)
await db.query(`ALTER TABLE "multi_token" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "erc1155_deposit" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "input" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "application" ADD "address" text NOT NULL`)
await db.query(`ALTER TABLE "application" ADD "chain_id" character varying`)
await db.query(`ALTER TABLE "application_factory" ADD "address" text NOT NULL`)
await db.query(`ALTER TABLE "application_factory" ADD "chain_id" character varying`)
await db.query(`CREATE INDEX "IDX_de7e0d7203642a7985ad1e5bf9" ON "token" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_98e0e62905bf3179bdbf443735" ON "token" ("address", "chain_id") `)
await db.query(`CREATE INDEX "IDX_8f6b794ba2a2b727b1929bb557" ON "erc20_deposit" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_1bcdd2b8e813324a2804544eb1" ON "nft" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_bfc419f78ddfbf14a8daef4abd" ON "nft" ("address", "chain_id") `)
await db.query(`CREATE INDEX "IDX_01a73626515aef160a3049a788" ON "erc721_deposit" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_a0b30e82e31660302a4af8cbe2" ON "multi_token" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_3d12b347a7a916b6975b8b6737" ON "multi_token" ("address", "chain_id") `)
await db.query(`CREATE INDEX "IDX_732c4624c6d3a7de141edf6f73" ON "erc1155_deposit" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_cf5fb31b5cf6719dc28adbb7d1" ON "input" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_29002c32f1d054e8f0ec4ea9cd" ON "input" ("id", "chain_id") `)
await db.query(`CREATE INDEX "IDX_13e23630dd6edeb8dd7062483f" ON "application" ("chain_id") `)
await db.query(`CREATE INDEX "IDX_6b4911d2e20363e9d2b4b24178" ON "application" ("address", "chain_id") `)
await db.query(`CREATE INDEX "IDX_402986e1692e825c3c773a56d6" ON "application_factory" ("chain_id") `)
await db.query(`ALTER TABLE "token" ADD CONSTRAINT "FK_de7e0d7203642a7985ad1e5bf97" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "erc20_deposit" ADD CONSTRAINT "FK_8f6b794ba2a2b727b1929bb5571" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "nft" ADD CONSTRAINT "FK_1bcdd2b8e813324a2804544eb1b" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "erc721_deposit" ADD CONSTRAINT "FK_01a73626515aef160a3049a7884" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "multi_token" ADD CONSTRAINT "FK_a0b30e82e31660302a4af8cbe2f" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "erc1155_deposit" ADD CONSTRAINT "FK_732c4624c6d3a7de141edf6f732" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "input" ADD CONSTRAINT "FK_cf5fb31b5cf6719dc28adbb7d1b" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "application" ADD CONSTRAINT "FK_13e23630dd6edeb8dd7062483f5" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
await db.query(`ALTER TABLE "application_factory" ADD CONSTRAINT "FK_402986e1692e825c3c773a56d62" FOREIGN KEY ("chain_id") REFERENCES "chain"("id") ON DELETE NO ACTION ON UPDATE NO ACTION`)
}

async down(db) {
await db.query(`DROP TABLE "chain"`)
await db.query(`ALTER TABLE "token" DROP COLUMN "address"`)
await db.query(`ALTER TABLE "token" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "erc20_deposit" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "nft" DROP COLUMN "address"`)
await db.query(`ALTER TABLE "nft" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "erc721_deposit" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "multi_token" DROP COLUMN "address"`)
await db.query(`ALTER TABLE "multi_token" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "erc1155_deposit" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "input" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "application" DROP COLUMN "address"`)
await db.query(`ALTER TABLE "application" DROP COLUMN "chain_id"`)
await db.query(`ALTER TABLE "application_factory" DROP COLUMN "address"`)
await db.query(`ALTER TABLE "application_factory" DROP COLUMN "chain_id"`)
await db.query(`DROP INDEX "public"."IDX_de7e0d7203642a7985ad1e5bf9"`)
await db.query(`DROP INDEX "public"."IDX_98e0e62905bf3179bdbf443735"`)
await db.query(`DROP INDEX "public"."IDX_8f6b794ba2a2b727b1929bb557"`)
await db.query(`DROP INDEX "public"."IDX_1bcdd2b8e813324a2804544eb1"`)
await db.query(`DROP INDEX "public"."IDX_bfc419f78ddfbf14a8daef4abd"`)
await db.query(`DROP INDEX "public"."IDX_01a73626515aef160a3049a788"`)
await db.query(`DROP INDEX "public"."IDX_a0b30e82e31660302a4af8cbe2"`)
await db.query(`DROP INDEX "public"."IDX_3d12b347a7a916b6975b8b6737"`)
await db.query(`DROP INDEX "public"."IDX_732c4624c6d3a7de141edf6f73"`)
await db.query(`DROP INDEX "public"."IDX_cf5fb31b5cf6719dc28adbb7d1"`)
await db.query(`DROP INDEX "public"."IDX_29002c32f1d054e8f0ec4ea9cd"`)
await db.query(`DROP INDEX "public"."IDX_13e23630dd6edeb8dd7062483f"`)
await db.query(`DROP INDEX "public"."IDX_6b4911d2e20363e9d2b4b24178"`)
await db.query(`DROP INDEX "public"."IDX_402986e1692e825c3c773a56d6"`)
await db.query(`ALTER TABLE "token" DROP CONSTRAINT "FK_de7e0d7203642a7985ad1e5bf97"`)
await db.query(`ALTER TABLE "erc20_deposit" DROP CONSTRAINT "FK_8f6b794ba2a2b727b1929bb5571"`)
await db.query(`ALTER TABLE "nft" DROP CONSTRAINT "FK_1bcdd2b8e813324a2804544eb1b"`)
await db.query(`ALTER TABLE "erc721_deposit" DROP CONSTRAINT "FK_01a73626515aef160a3049a7884"`)
await db.query(`ALTER TABLE "multi_token" DROP CONSTRAINT "FK_a0b30e82e31660302a4af8cbe2f"`)
await db.query(`ALTER TABLE "erc1155_deposit" DROP CONSTRAINT "FK_732c4624c6d3a7de141edf6f732"`)
await db.query(`ALTER TABLE "input" DROP CONSTRAINT "FK_cf5fb31b5cf6719dc28adbb7d1b"`)
await db.query(`ALTER TABLE "application" DROP CONSTRAINT "FK_13e23630dd6edeb8dd7062483f5"`)
await db.query(`ALTER TABLE "application_factory" DROP CONSTRAINT "FK_402986e1692e825c3c773a56d62"`)
}
}
20 changes: 9 additions & 11 deletions docker-compose-db.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
version: "3"

services:
db:
image: postgres:15
environment:
POSTGRES_DB: squid
POSTGRES_PASSWORD: postgres
ports:
- "${DB_PORT}:5432"
# command: ["postgres", "-c", "log_statement=all"]
shm_size: 1gb
db:
image: postgres:15
environment:
POSTGRES_DB: squid
POSTGRES_PASSWORD: postgres
ports:
- '${DB_PORT}:5432'
# command: ["postgres", "-c", "log_statement=all"]
shm_size: 1gb
Loading
Loading