Tracking issue: Improve parachain liveliness observability

[sandreim]

The Problem

We currently rely only on node side metrics to observe the liveliness of the network. Mainly, we look at things like parachain block times, approval checking, disputes and finality lag which we are able to determine only if we have access to metrics scraped from validators and collators. This works really nice for test networks where we manage both the validators and collators. We want to build some tooling to enable additional observability via RPC . This would not obsolete any of the node metrics, as we would still rely on those when debugging.

Plan

We need to implement tracking of all parachains in the parachain commander which is currently limited to only one parachain. This full tracking should be the deafult mode for running the tool in Prometheus mode while CLI remains unchanged to tracking one parachain.

Metrics

The following metrics will be computed from the parchain inherent data from each relay chain block:

• parachain block times (measure in relay chain blocks)
• relay chain block times (via inherent timestamps)
• availability health (bitfield count and 1 bits when core occupied)
• backing
• DMP/UMP/HRMP throughput
• dispute initiation and conclusion times
• dispute initiation per validator (via address label)

Deployment

We want to deploy this for both prod and test networks: Kusama, Polkadot, Westend, Rococo and Versi.
Additionally to the implementation we need to create Grafana dashboards that will be available in the polkadot-introspector repo.

Milestones

• all metrics implemented
• dashboards created and published
• deployed on Polkadot/Kusama/Westend/Rococo
• alerting and paging configured

Project tracking board

https://github.com/orgs/paritytech/projects/70/views/1