services/horizon/internal/ingest: Remove unnecessary use of ChangeCompactor to reduce memory bloat during ingestion

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Part of #5258

Horizon users have reported that state rebuilds have caused Horizon's memory usage to exceed 32 GB. After profiling Horizon we discovered that most memory is being retained in the ChangeCompactor:

The ChangeCompactor is described in detail here https://github.com/stellar/go/blob/master/ingest/change_compactor.go#L49 . It turns out that there is no need to use the ChangeCompactorwhen ingesting from the history archives because all ledger entries changes obtained from the history archives are of the created type which means that we will never compact any ledger entries.

By removing the ChangeCompactor from all the state processors we avoid creating additional copies of ledger entries obtained from the history archives. This has resulted in a substantial reduction in memory usage. Previously we used ~32 GB of RAM to do a state rebuild, now, we are able to do a state rebuild in ~5G:

The only place where it makes sense to use a ChangeCompactor is when processing ledger entry changes extracted from a LedgerCloseMeta payload. It is possible for a ledger entry to be created and then modified or removed within the same ledger.

Known limitations

[N/A]

[ire-and-curses]

The only place where it makes sense to use a ChangeCompactor is when processing ledger entry changes extracted from a LedgerCloseMeta payload. It is possible for a ledger entry to be created and then modified or removed within the same ledger.

It's probably worth adding a comment to ChangeCompactor saying something like this, to prevent incorrect usage in future.

[sreuland]

lgtm, great write-up as well on problem/solution in pr description, left some minor comments for consideration.