DelegationZ - Get latest and historical delegations events on Tezos Chain

Introduction

A Golang service to fetch delegations events from TZkt.io api, save and present them.

Specifications

LINK HERE

Architecture

As per the specifications above, we will need at least three distinct parts :

• Importer / Watcher * : First brick to run in order to sync historical datas from reference tzkt.io api. Once sync is completed, a "realtime" or "cron like" job is kept running to keep it.

• Api *: A "RESTish" API to expose gathered data.

* The whole parts are combined in one binary or could be built and deployed separately according to requirements.

Implementation

Data fetching :

To first fetch reference api data's, the need to develop an isolated, tested and reusable tzkt api client wrapper comes pretty quickly* (regarding the requirements only Delegations is actually implemented).

* As of 28/06/28 DipDup Go SDK open source project with the same goals seems to be a promising start but currently not used in this project.

Persistence and communication :

The three parts detailed above should be able to communicate together or at least referring themselves to a single source of truth. In a will to be "simple", this implementation is using Postgres to store and retrieve delegations datas. For realtime need or added observability, a "pubsub style" messaging queue could be later introduced for example to share sync progress or to receive filtered notifications for a particular "baker" or else...

API :

Path	Params	Response
/*	NA	HTML
/xtz/delegations	QueryParams: year : YYYY limit : <= 100 page : int	JSON 200 : [{timestamp: string, amount: string, delegator: string, block: string}, ...] 500 : {message: string} 400 : [{timestamp: string, amount: string, delegator: string, block: string}, ...] 500 : {message: string}
/xtz/sync	NA	JSON 200 : {id: number, timestamp: string, amount: number, delegator: string, block_hash: string, block_level: number} 500 : {message: string}
/health	NA	HTTP NO CONTENT CODE 200

Limits

• Service is highly dependent to tzkt api for its synchronization speed and overall functionning.

• Relation with the data provider could go wrong and lead to corrupted or no datas.

• Maybe delegation events could be acquired from a node rpc source directly ?

• This implementation lacks a proper caching strategy, a request rate limit had been put in place to avoid database load but it's far from being sufficient if usage increase.

Setup

Tldr: With your favorite (UN*X) distribution on hand, fire up your shell and run according to your package manager :

apt-get install golang nvm jq docker
nvm install 20
nvm use 20
make testdb DOCKER=docker
go mod download
go run cmd/api/main.go &
curl "http://localhost:8080/xtz/delegations" | jq
## 2 latest delegation events order by latest first

Go

Install and setup a modern (>=1.20) golang env

Sqlboiler

To fetch delegations datas stored in postgres sql without having to worry about golang models drifting or incompactibility with the database scheme, runnning make gen will let sqlboiler tool introspect the running database scheme and generate a repository / orm* style object to use on api resolvers. To use it run :

go install github.com/volatiletech/sqlboiler/v4@latest
go install github.com/volatiletech/sqlboiler/v4/drivers/sqlboiler-psql@latest
export PATH=$PATH:/$HOME/go/bin

* Of course, such an orm is only used for the "READ" database operations, the data import processes use "custom" sql queries to avoid too much overhead and optimise the sensible "WRITE" operations.

Node , Npm , Npx ...

Install nvm and use latest nodejs version

Docker

To help during the development, a postgres db is used, install docker or podman and use

make testdb

to test the success of your installation.

Development

Database scheme and migrations

In order to ease, control and track the database scheme(s) definitions, the prisma tool is used. To udpate the db scheme, update schema.prisma and create a new mutation with cd migrations && npx --yes prisma@latest migrate dev.

Unit Testing

To illustrate a robust and maintenable project setup, some unit tests have been written for tzkt client and automatically generated for the database repository. Run them with make test

Git

Conventions

If not already mandatory in your institution, conventional commit is a clear and concise way to write your commit messages and easily understand them later.

<type>(scope): <description>

[optional body]

[optional footer(s)]

Eg :

chore(repo): init repo
...
feat(api): pagination filters for /tzc/delegations endpoint
...

Deployment

Regarding the deployment strategy, it should be carefully planned based on required business and engineering requirements such as availability, latency , usage volume / related costs, existing pipelines and infrastucture ...

This repository project is not actually deployed somewhere, sorry about that. UPDATE : It is! thanks to Neon for postgres DB and Fly.io for serverless app hosting ! 🚀🚀🚀🚀

Misc

• During the importer development, a strange reference api behavior appear, reproduction and developement is summarised in tzkt_pagination_test.sh script. (Tss .. those developers who can't read a documentation ...)

• In the specifications requirement , the block field asked for the api response to looks like a stringified bigint (as the block level) ("block": "2338084") but the reference api is returning the hash like : "block": "BLwRUPupdhP8TyWp9J6TbjLSCxPPW6tyhVPF2KmNAbLPt7thjPw", . Current implementation will return the level as the block response field. It could been suggested that each api features a more descriptive name for their fields as block_level or block_hash.

• In the specifiction requirements, the delegator field has been mapped from "newDelegate": { "address":"..."} with no confidence in the fact that newDelegate is the asked delegator. (Tss .. those developers who can't read the specifications...)

• Regarding the delegator field, it could sometimes be empty .. 🤷 ...

• Should have spent some more time reading tzkt blog and more precisely this subject