-
Notifications
You must be signed in to change notification settings - Fork 503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
exp/lighthorizon: Create pubnet indices for the MVP endpoints. #4475
Comments
I'm helping take over last step here of running a range on cloud to confirm indexes are built out. Initially will build for 07/01 - 07/31 to confirm. |
@Shaptic , looking into alternative k8s job for batch processing in lieu of AWS batch in short term. needs some adjustment on cluster to support JOB_COMPLETION_INDEX per https://github.com/stellar/ops/issues/1790 |
status update, running map/reduce jobs on k8s, identified some issues with slow performance of reduce jobs when running against s3 index. next step, adding logging into reduce, capture time spent in sections with s3 calls to identify i/o rates, |
Parent epic: #4317
Our target endpoint is:
So we need an index for each account that stores information about which ledgers the account was active in. Before we actually do this, there's an open question to answer: What kind of indices do we make?
The index builder is ready for parallel construction, but it might not be perfectly ready for AWS Batch. There is some preliminary work to get both steps (map and reduce) "dockerized" and running on Batch.
The text was updated successfully, but these errors were encountered: