Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot module #164

Merged
merged 9 commits into from
Dec 10, 2020
Merged

Snapshot module #164

merged 9 commits into from
Dec 10, 2020

Conversation

ethanfrey
Copy link
Member

@ethanfrey ethanfrey commented Dec 9, 2020

Closes #161

Make a SnapshotMap in storage-plus that handles this automatically for us.

  • Implement base
  • Implement snapshot every block
  • Implement snapshot selected blocks
  • Tests

@ethanfrey ethanfrey requested a review from maurolacy December 10, 2020 00:54
Copy link
Contributor

@maurolacy maurolacy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Will take a detailed look at the tests during the afternoon.

Next step / variant: timestamp based snapshots.

packages/storage-plus/src/snapshot.rs Outdated Show resolved Hide resolved
packages/storage-plus/src/snapshot.rs Outdated Show resolved Hide resolved
r.map(|(_, v)| v.old)
} else {
// otherwise, return current value
self.may_load(store, k)
Copy link
Contributor

@maurolacy maurolacy Dec 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about signalling / exposing this difference? Particularly for Selected, it can be useful. At least for debugging.
Maybe by providing an alternative method, is_correct() or so; not to spoil the return value here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the data is not available, I can either return possibly incorrect data, or return an error.

I think you are correct that an error is better.

That applies to any historical query for Selected::Never

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I addressed this, it down returns StdError::NotFound if there is no relevant checkpoint

Copy link
Contributor

@maurolacy maurolacy Dec 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is my understanding that it may apply to Selected as well: If you're using Selected, and didn't create / request a checkpoint for that specific height, you cannot tell for sure if the returned value is correct. It may be correct, but it may also be an updated value, which didn't get snapshotted at that particular height but at a later one. So, it may have been updated (and stored) at a later height (and we don't know what may have happened in between). Or, it's the current value, and we also cannot tell if it's still correct for that height.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm proposing is returning the value nevertheless, but return also a flag (or, maybe better, provide another method) to indicate if the value is guaranteed to be correct for that height, or not.

Copy link
Contributor

@maurolacy maurolacy Dec 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen you have implemented returning an error when the value is not guaranteed to be correct.

I prefer that, as it's less error prone: if you are using Selected and didn't get the guaranteed values for your heights, you're probably using it wrong. Same if you're not using a map with snapshots enabled, and pretend to get some historical values.

packages/storage-plus/src/snapshot.rs Outdated Show resolved Hide resolved
@maurolacy
Copy link
Contributor

Next step / variant: timestamp based snapshots.

If the sub-key is the same type / size, this is already working, by the way.

@ethanfrey
Copy link
Member Author

Thanks for the feedback, I will do some requested cleanup

Next step / variant: timestamp based snapshots.

Yeah, you have to store both separately, and I find height a much more natural (and guaranteed unique) measure than timestamp. There may be 3 blocks in the same second for example.

I will leave that as an excercise for the reader.

@ethanfrey
Copy link
Member Author

Fixed up change requests. I will merge when CI is happy.

#165 builds on top and is another chance to review the module in practice.

@ethanfrey ethanfrey merged commit 9336a4a into master Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor snapshotting into reusable module
2 participants