Enhanced Pooler Efficiency for Smaller Epoch Sizes

[xadahiya]

Fixes #42

Checklist

• Synchronized my branch with the upstream/develop branch.
• Thoroughly tested changes across Python 3.8.0 and higher.
• Successfully passed pre-commit checks for code quality.

Current Behavior

The current behavior of the pooler, as outlined in issue #42, demonstrates peak efficiency when employed with larger epoch sizes. However, the performance falters significantly when attempting to utilize the pooler with an epoch size set to 1. This issue initiates a cascading sequence of problems, beginning with the failure of the 24-hour trade volume aggregate computation. This failure subsequently disrupts the entire workflow, rendering it non-operational.

Proposed Changes and Solutions

Drawing from the insights provided in the issue description, the root cause of this issue stems from the intricate nature of the 24-hour trade volume aggregate calculation. The current implementation relies heavily on prior finalized epoch data to execute efficient computations, referencing data from the previous epoch. Unfortunately, the brief time window of 12 seconds (as observed with ETH) allotted for the finalization of the last epoch is inadequate. Consequently, a full recalculation is triggered for each epoch, leading to the retrieval of extensive data from either the RPC or the local Cache.

This excessive data retrieval results in a surge of RPC requests and subsequent Rate Limit errors, creating a domino effect of failures that incapacitate the entire system.

To address this challenge, I propose a comprehensive redesign of the 24-hour trade volume aggregate calculation. The suggested approach revolves around eliminating the dependency on finalized calculations and instead capitalizing on locally stored base snapshots submitted by the snapshotter for the latest unfinalized epochs. This strategic shift in methodology will significantly optimize the system's overall efficiency, paving the way for seamless support of an epoch size equal to 1.

This enhancement holds the potential to surmount the existing challenges and to elevate the pooler's functionality significantly, spanning a wide range of epoch sizes.

Impact of Changes

• Redesigned the 24-hour trade volume aggregate calculation process for enhanced efficiency.

Deployment Instructions

To integrate these transformative changes, follow these steps:

1. Pull the latest changes from the repository.
2. Restart the system to implement the updates.

Updated 24h Trade Volume Aggregation Workflow

[anomit]

Approving since changes has been observed running stably in staging instances