Ingestion Filtering by Asset

[sreuland]

What problem does your feature solve?

Horizon ingestion of full ledger space from core can take up massive amounts of disk space on db.

What would you like to see?

As a service provider with my own Horizon installation, I want to track transactions and operations related to assets I issue. I want to perform reconciliation against my Horizon DB for assets I issue.

I want to configure which assets trigger related transaction and operational info to be ingested by my Horizon installation and persisted to my horizon historical database.

I don't want my horizon historical database to contain data related to other assets, so, I expect to see my horizon db size on disk remain relatively small with a slower growth rate compared to no-filtering(i.e. data for entire ledger was saved)

Acceptance Criteria:

• Horizon provides an Asset Filter rule configuration API endpoint on the secure Horizon Admin port to perfrom CRUD on the list of whitelisted canonical asset codes of <asset_code>:<asset_isssuer>
• The Asset filter rule configuration interface accepts a list of canonical asset codes <asset_code>:<asset_isssuer> to add or remove to whitelist.
• All Horizon ingestion processes(current-forward/reingest) run filtration and apply the whitelist asset rules and result in only persisting historical data that passes the filter.
• The asset filter effectively applies latest config rules to current and ongoing ingestion activity which occurs after a filter rule is updated.
• The Horizon Asset Filter feature can be configured as a toggle/switch, enabled or disabled.
• The Horizon Asset Filter feature should be available as early/alpha drop builds to some willing early-adopter partners for trial usage.
• Need to choose feature branch or mainline development(i.e. merged to master often).

What alternatives are there?

[sreuland]

dev complete on asset filtering on the feature branch including unit tests:

master...filtering-feature

moving this to review phase.