Horizon Lite - Productionization / Optimization Epic

[jcx120]

Epic for Productionalization and Optimization of Horizon Lite. Continuation from #4317

Backlog

These are in order of importance:

• Horizon Lite: Improve the performance and functionality of the batch-based indexer. #4566
• exp/lighthorizon: Support order parameter in endpoints. #4503
• exp/lighthorizon: Make s3 bucket of indexer backend configurable #4504
• exp/horizonlight: Make indexing data versionable #4506
• exp/horizonlight: indexing backend should use the support/storage package #4505

R&D Tasks

This entails exploring use-cases for Horizon Lite besides the main MVP described above.

• exp/services/ledgerexporter: Research Spike - Alternative storage methods #4473
• services/horizon: Research Spike - Using unpacked txmeta for reingestion #4474
• Horizon Lite: Come up with better index and meta compression scheme #4497
• exp/lighthorizon: Research Spike - Smart Contracts - Are txmeta archives usable by other tooling? #4459

Archived Tasks

These are past work items that have been completed, kept around for visibility and historical tracking.

Phase 1: Feature Work

• Spike: Light Horizon ingesting historical data from archives #4212
• Ingest via txmeta instead of from history archives
• exp/lighthorizon: Use XDR for serialization rather than a custom binary format #4372
• Investigate off-the-shelf data structures (e.g. RoaringBitmap. Conclusions: off-the-shelfs...
  • are slower for bitmaps, particularly the NextActive() equivalent
  • don't support all of the operations we need (e.g. insert vs. upserts, which may be important for non-txhash index use cases)
• Index (re)builder needs updating to build based on txmeta files, instead of history archives:
  • single-process (./index/cmd/single)
  • lighthorizon: Enable parallel creation of indices via map-reduce #4403
• Complete FileBackend.ReadAccounts()
• exp/services/ledgerexporter: Extend tool to work on ledger ranges in parallel. #4426
• Support real-time ingestion:
  • Extend the ledgerexporter tool to allow unbounded ledger ranges
  • lighthorizon: Extend the indexing tool to allow unbounded ledger ranges #4406
  • Integrate the above into request serving to ensure access to new ledger data as quickly as possible
  • Add a version byte to the written txmeta files, in case we need to update the format in future.
• exp/lighthorizon: Correctly paginate responses to respect query limits #4430
• exp/lighthorizon: add endpoints for paged list of entities(tx/op) by account ID #4433
• The request parsing should mirror Horizon so that we can effectively do comparison testing in Phase 2. For example, right now we have GET /operations?account=<id> whereas in production we have GET /account/<id>/operations.
• starter on API docs with OAPI

Phase 2: Correctness

Is the work we've done actually correct, i.e. Does it match what Horizon serves today?

There are two main questions here:

1. Are we generating indices correctly?
2. Are we using indices correctly?

• Export pubnet ledger txmeta
• Correctness: end-to-end tests for indexing jobs
• exp/lighthorizon: Create pubnet indices for the MVP endpoints. #4475

Phase 3: Going Live

In this phase, we'll actually be deploying the feature to real traffic and seeing how it performs.

Metrics

We want to easily be able to easily answer the question "How long does stuff take?"
At minimum, we should include:

• Add metrics to Horizon Lite #4451
• Minimum Dashboard for Testing / Verification

• Dashboards or some other form of at-a-glance analysis and comparison against existing infrastructure

Deployment

Work with Ops to build out infrastructure and deploy the branch. Questions we need to answer beforehand and tasks we need to complete:

• What kind of architecture should it have?
• Do we want to mirror traffic from testnet or production?
• Do we want to do ALB replays?
• A build/deployment pipeline for easily updating the infra with changes
• Approximate the $ cost of a production-ready architecture and compare it to the current one
• Dev/Demo deployment on infra - Dev deployment of Horizon Light  #4493

MVP Testing

Understand viability of Lite design for real world conditions using our Steel thread impl.

• Understand request latency characteristics (can it support web scale response)
  • Request profiling: history age vs variation of data
  • Hotspots (indices lookup > txmeta download > request processing etc)
• Load testing (is there a path to horizontal scaling)
• Confirm HW footprint (does it confirm our “solid state, constant” assumptions)
  • IO, RAM, CPU, Disk

Performance Optimization

• exp/lighthorizon: Fetch ledgers in parallel whenever possible #4468
• Understand tradeoff between file-per-ledger and file-per-checkpoint
• Identify optimizations as needed to minimize high-latency operations (TBD based on the above, but e.g. minizing network round-trips)

Phase 4: Everything Else

The remaining tasks are contingent on the success of Phase 3: if we deem that Horizon Lite is actually worth doing, there's quite a bit of work to getting everything else ready for production and public release.

• Indices
  • Indices Updating Lifecycle POC
    • Note: currently indexes point to checkpoints. might need updating to point to ledgers? might be fine? tbd.
  • Complete buildout of indices (excluding Derived Data, eg: Trade Agg, Fee stats)
• Complete buildout of endpoints:
  • Accounts, transactions, operations
  • Other endpoints
  • Streaming: ensure streaming works for Ingestion Lite design pattern
• Smart Contract Integration: Identify the minimum requirements we need for lighthorizon to ship it for smart contracts  #4358
• Public, distributed infrastructure for remote TX Meta and Indices
  • Design minimal distributed infrastructure for generating, hosting files - Light Horizon: deployment of ledgerexporter as service #4483
  • Implement 1 SDF node for TX Meta and Indices
  • Develop plan to have other T1 operators generate/host
• Code refactors:
  • Should the leaf of a trie index be a bitmap? This is a clean way to combine the two index types.
  • Optimize workers to work on entire checkpoint ranges rather than on individual ledgers (one flush per checkpoint)
  • Unify filesystem abstractions, lots of variations around the codebase
  • exp/lighthorizon: Refactor codebase to avoid using HistoryArchiveBackend as a filesystem interface #4460
  • Horizon Lite: refactor to its own source repo #4437