Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exp/lighthorizon: Refactor single-process index builder. #4410

Merged
merged 10 commits into from
May 31, 2022
Merged

exp/lighthorizon: Refactor single-process index builder. #4410

merged 10 commits into from
May 31, 2022

Commits on May 25, 2022

  1. Refactor index builder: 

     - allow worker count to be a command line parameter
 - split work by checkpoints rather than ledgers
 - move actual index insertion work to helpers
 - move progress bar into helpers
 - simplify participants code, payments vs. all
    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    90d8ffa View commit details
    Browse the repository at this point in the history

  2. Properly work on a checkpoint range at a time: 

     - previously, it was just arbitrary 64-ledger chunks which is not as helpful
    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    183ec35 View commit details
    Browse the repository at this point in the history

  3. Move index building work into helper function

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    a6a33e4 View commit details
    Browse the repository at this point in the history

  4. Define a generic module processing function

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    a9731bc View commit details
    Browse the repository at this point in the history

  5. Use existing range structure over custom

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    c3ea13e View commit details
    Browse the repository at this point in the history

  6. Move index building into a separate object

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    ca8cf1e View commit details
    Browse the repository at this point in the history

  7. Move index builder into separate file

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    2bc682a View commit details
    Browse the repository at this point in the history

  8. More accurate progress string lol

    @Shaptic
    Shaptic committed May 25, 2022
    Configuration menu
    Copy the full SHA
    ed91d7a View commit details
    Browse the repository at this point in the history

Commits on May 31, 2022

  1. Rename function for clarity

    @Shaptic
    Shaptic committed May 31, 2022
    Configuration menu
    Copy the full SHA
    2023843 View commit details
    Browse the repository at this point in the history

  2. Fix off-by-one error in checkpoint index builder: 

      - Keeping this as-is would mean that the first chunk of ledgers
    will be "Checkpoint 0" which doesn't make sense in the bitmap
  - Calling index.setActive(0) is essentially a no-op, because no
    bit will ever be set.
  - In the case of an empty index in which the only active account
    checkpoint is the first one, this is indistinguishable from an
    index with no activity.
    @Shaptic
    Shaptic committed May 31, 2022
    Configuration menu
    Copy the full SHA
    cde0ba1 View commit details
    Browse the repository at this point in the history