Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record processed epochs / tipsets in events index #11640

Closed
2 tasks
rvagg opened this issue Feb 22, 2024 · 5 comments · Fixed by #12080
Closed
2 tasks

Record processed epochs / tipsets in events index #11640

rvagg opened this issue Feb 22, 2024 · 5 comments · Fixed by #12080

Comments

@rvagg
Copy link
Member

rvagg commented Feb 22, 2024

Arising out of #11618 - Lotus currently has no way to know whether the earliest events it has are the earliest there are or the earliest that that have been recorded. Ideally requesting "events from height X" should error if we can say for sure "I don't have events for that height", but that's not possible today because we don't know.

  • Add a minor migration to the event index database with a new table to record "height" once it's been fully processed, the migration could either walk back through indexed events and mark those as done, or to be more comprehensive, walk back through the message receipts and mark them off?
  • Error on the GetActorEvents and SubscribeActorEvents APIs if you request a height that's earlier than we have.
@rjan90 rjan90 added this to FilOz Feb 22, 2024
@rjan90 rjan90 moved this to 🐱Todo in FilOz Feb 22, 2024
@rjan90 rjan90 added this to the v1.27.0 milestone Feb 22, 2024
@Stebalien
Copy link
Member

Specifically, we need to record every tipset (tipset key) we've processed, and the tipset's height (really, we just need the tipset key, but having the height is likely useful for querying).

When querying for events, we'd be able to distinguish between "no events" and "haven't processed this tipset" by checking if the tipset key is in the database.

This will also let us fix #11007 by:

  1. Looking at the tipset we get from Lotus when we start "observing" tipset changes.
  2. Walking back from that tipset until we find a tipset present in the events database.

@Stebalien
Copy link
Member

In terms of the migration... I believe that checking if we have any events at a tipset is relatively inexpensive, and we only need to do that for tipsets that don't have recorded events.

@jennijuju
Copy link
Member

@Stebalien iirc we want this for the message index too, is that right (when we were discussing tracing api)?

@Stebalien
Copy link
Member

Yes.

@masih masih self-assigned this Mar 6, 2024
@masih
Copy link
Member

masih commented Mar 6, 2024

@rvagg please feel free to assign me other issues that relate to this, considering the need for migration. That should reduce the need for coordination.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: ☑️ Done (Archive)
6 participants