Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preloader architecture, memory, and other improvements #39

Merged
merged 28 commits into from
Jul 28, 2023

Conversation

xadahiya
Copy link
Member

FEAT: Modular preloader architecture with memory and other improvements

Checklist

  • My branch is up-to-date with upstream/develop branch.
  • Everything works and tested for Python 3.8.0 and above.
  • I ran pre-commit checks against my changes.

Change logs

  • Cleaner and Resilient Preloader Architecture to support generic preloading for projects
  • Refactoring and project structure improvements
  • Memory Optimizations
  • 7d aggregate calculation fixes
  • Other minor performance and documentation improvements

Preloader Config

{
    "preloaders": [
        {
          "task_type": "block_transactions",
          "module": "snapshotter.utils.preloaders.tx_receipts.preloader",
          "class_name": "TxPreloadWorker"
        },
        {
          "task_type": "block_details",
          "module": "snapshotter.utils.preloaders.block_details.preloader",
          "class_name": "BlockDetailsPreloader"
        },
        {
          "task_type": "eth_price",
          "module": "snapshotter.utils.preloaders.eth_price.preloader",
          "class_name": "EthPricePreloader"
        }
    ],
    "delegate_tasks": [
        {
          "task_type": "txreceipt",
          "module": "snapshotter.utils.preloaders.tx_receipts.delegated_worker.tx_receipts",
          "class_name": "TxReceiptProcessor"
        }
    ],
    "timeout": 60
}

Projects Config changes for each project type

       "preload_tasks":[
           "block_transactions"
      ],

xadahiya and others added 28 commits June 11, 2023 15:30
… source_chain_id in processor_distributor for cleaner startup
@xadahiya xadahiya requested review from anomit and removed request for anomit July 28, 2023 08:43
@xadahiya xadahiya requested review from SwaroopH and anomit July 28, 2023 08:43
Copy link
Member

@SwaroopH SwaroopH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Tested on 2 different machines.

@SwaroopH SwaroopH merged commit 56c3dd7 into main Jul 28, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants