Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot item #409

Merged
merged 8 commits into from
Sep 13, 2021
Merged

Snapshot item #409

merged 8 commits into from
Sep 13, 2021

Conversation

maurolacy
Copy link
Contributor

@maurolacy maurolacy commented Sep 9, 2021

Closes #193.

Straightforward impl adapting from SnapshotMap. Created a Snapshot abstr, and used it in SnapshotItem and SnaphotMap.

Still need to review / validate tests. (done in #418)

@maurolacy maurolacy requested a review from ethanfrey September 9, 2021 15:40
@maurolacy maurolacy self-assigned this Sep 9, 2021
Copy link
Member

@ethanfrey ethanfrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, looks good.

I would either just hardcode "always checkpoint" to simplify the SnapshotItem (preferred) or figure out a way to reuse logic between both, as those concepts are non-trivial to code and test.

packages/storage-plus/src/snapshot_item.rs Show resolved Hide resolved
@ethanfrey
Copy link
Member

I'm doing a deeper dig through the code now. The diffs don't help, I need to check it out.

The first comment is with snapshot.rs, snapshot_map.rs and snapshot_item.rs, maybe they deserve a submodule. This package is growing and could use some structure. Like snapshot and below that mod.rs map.rs and item.rs. And then re-exported on the top level.

Maybe index code could also be grouped like this as well, but that is a different PR.

Copy link
Member

@ethanfrey ethanfrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice design and well done pulling out the Snapshot.

I propose 2 more methods to put into Snapshot.
And it would be nice to have a few basic unit tests in snapshot.rs (I trust it as it works in higher level, but to demo the API and hit a few edge cases, it would be nice).

Other than that, good to merge IMO

packages/storage-plus/src/keys.rs Show resolved Hide resolved
packages/storage-plus/src/snapshot_item.rs Show resolved Hide resolved
packages/storage-plus/src/snapshot_item.rs Outdated Show resolved Hide resolved
/// load old value and store changelog
fn write_change(&self, store: &mut dyn Storage, height: u64) -> StdResult<()> {
// if there is already data in the changelog for this block, do not write more
if self
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise, a helper here, like:

self.snapshots.has_changelog(store, (), height) would be nice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And maybe expose write_change on self.snapshots, as this same basic logic is used for Map

packages/storage-plus/src/snapshot_item.rs Show resolved Hide resolved
packages/storage-plus/src/snapshot_map.rs Show resolved Hide resolved
@maurolacy
Copy link
Contributor Author

maurolacy commented Sep 13, 2021

OK, will refactor.

Yes, diffs don't help anymore. What you can do is read / review snapshot.rs entirely, and then check for usage in snapshot_item and snapshot_map. Also, manually obtaining and then diffing snapshot_map.rs against main's snapshot.rs could help.

Finally, I can separate these into two / three PRs for ease of reviewing:

  • Rename snapshot.rs to snapshot_map.rs.
  • Refactor Snapshot out of SnapshotMap/Item.
  • Rename to add a snapshot/ folder.

Hadn't seen your latest comments. Let's put this last one into a follow-up.

@ethanfrey
Copy link
Member

I just checked out the code and read it. It looked good. No need to break it up.

Just saw these 2 non-trivial methods I figured could move into snapshot, then they all look great.

Moving the files and even adding more unit tests could be done in a separate PR. Happy to have this.

@maurolacy
Copy link
Contributor Author

Please take a look. I would merge this as it is, and then do a follow-up with the paths refactoring, and adding some Snapshot tests.

Copy link
Member

@ethanfrey ethanfrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. Thanks for this cleanup. Will merge

packages/storage-plus/src/snapshot_item.rs Show resolved Hide resolved
packages/storage-plus/src/snapshot_item.rs Show resolved Hide resolved
// if we found a match, return this last one
r.map(|(_, v)| Some(v.old))
} else {
Ok(None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice decision here

r.map(|(_, v)| v.old)
let snapshot = self.snapshots.may_load_at_height(store, (), height)?;

if let Some(r) = snapshot {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this looks nice here, keeping the primary load in this function, and checking an option to switch.

@ethanfrey ethanfrey merged commit 5fc8ae9 into main Sep 13, 2021
@ethanfrey ethanfrey deleted the snapshot-item branch September 13, 2021 17:50
@maurolacy maurolacy mentioned this pull request Sep 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add SnapshotItem to storage-plus
2 participants