chore: Add check for uneven stores' height

[catShaark]

Description

There's a problem in many chains that doesn't handle adding new module store correctly. That problem is described here. In short, these chains have trouble prunning and snapshoting because the new module store height is off from the actual height. @chillyvee has made some fixs in iavl and cosmos-sdk for snapshoting and prunning for this problem, I think we also need to add sanity check to prevent this from happening.

Changes

This PR add sanity check in loadVersion() so that it errors when the lastest version of any module stores mismatch root store version, except for the case when we add a new module store with up StoreUpgrades.Added

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• included the correct type prefix in the PR title
• added ! to the type prefix if API or client breaking change
• targeted the correct branch (see PR Targeting)
• provided a link to the relevant issue or specification
• followed the guidelines for building modules
• included the necessary unit and integration tests
• added a changelog entry to CHANGELOG.md
• included comments for documenting Go code
• updated the relevant documentation or specification
• reviewed "Files changed" and left comments if necessary
• confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

• confirmed the correct type prefix in the PR title
• confirmed ! in the type prefix if API or client breaking change
• confirmed all author checklist items have been addressed
• reviewed state machine logic
• reviewed API design and naming
• reviewed documentation is accurate
• reviewed tests and test coverage
• manually tested (if applicable)

[chillyvee]

The panic on load is interesting.

Should we warn (not panic) and let the chain run since the problem is already committed to disk?

Another idea: Prevent commit to disk if height mismatches (easier recovery with new binary).

[catShaark]

Should we warn (not panic) and let the chain run since the problem is already committed to disk?

I think error on loadVersion() will prevents commit, that's why I put it there.

[tac0turtle]

is it true to say there is a different problem with adding modules that may not be addressed here? If so, I would say lets fix the problem of adding modules at the root instead of adding another check

[chillyvee]

Adding modules without a keeper(?) migration results in mismatched heights. If stores would default to the current height rather than 0, then the issue could be prevented.

This would not fix any current issues that chains are encountering but could reduce future incidents.

[catShaark]

is it true to say there is a different problem with adding modules that may not be addressed here? If so, I would say lets fix the problem of adding modules at the root instead of adding another check

I don't know the problem you're mentioning, can you elaborate

[tac0turtle]

I don't know the problem you're mentioning, can you elaborate

seeing the issue teams are running into it seems there is a fundamental issue that we should address. This is a quick fix, but not the fix that will prevent further issues as chilly eluded to.

[catShaark]

I'm fixing the tests

[tac0turtle]

@Mergifyio backport release/v0.45.x

[mergify]

backport release/v0.45.x

✅ Backports have been created

• #15114 chore: Add check for uneven stores' height (backport #14410) has been created for branch release/v0.45.x

[tac0turtle]

@Mergifyio backport release/v0.46.x

[mergify]

backport release/v0.46.x

✅ Backports have been created

• #15115 chore: Add check for uneven stores' height (backport #14410) has been created for branch release/v0.46.x