feat: Fraud Proof generation and partial verification using IAVL store #1

Manav-Aggarwal · 2022-10-14T04:05:41Z

Description

Consolidates code across PRs from celestiaorg#245 along with replacing the usage of the SMT store with the IAVL store.

Covers Fraud Proof Generation and the first two levels of verification.

The third level of verification needs the functionality to set up a BaseApp from a Fraud Proof. This requires the ability to create IAVL Deep Subtree. This PR includes placeholders for these IAVL Deep Subtrees which can be added once rollkit/iavl#1 is wrapped up.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

MSevey · 2022-10-17T14:56:30Z

baseapp/abci.go

+
+	appHash, err := cms.GetAppHash()
+	if err != nil {
+		panic(err)


Is calling panic a standard pulled from the cosmos sdk?

Yes, for ABCI methods, if something goes wrong, there's no way to gracefully throw an error other than panic. If you look at the BeginBlock implementation, you'll see panic calls.

tzdybal

💪

As discussed Yesterday, small issues should be addressed now, bigger ones can be extracted as followup-issues and implemented later.

baseapp/abci.go

tzdybal · 2022-10-17T13:18:10Z

baseapp/abci.go

+	cms := app.cms.(*rootmulti.Store)
+	err := cms.LoadLastVersion()
+	if err != nil {
+		panic(err)


Is there any situation when this can happen? Because of pruning settings, genesis block?

Happens when there is no last state to load form, added a inline comment mentioning it

tzdybal · 2022-10-17T13:25:08Z

baseapp/abci.go

+}
+
+func (app *BaseApp) executeNonFraudulentTransactions(req abci.RequestGenerateFraudProof) {
+	nonFraudulentRequests := req.DeliverTxRequests[:len(req.DeliverTxRequests)-1]


What if EndBlock is fraudulent? Then all TXs are not fraudulent, but the last one will be skipped.

Fixed this case.

tzdybal · 2022-10-18T07:53:08Z

baseapp/baseapp_test.go

+		4. Generate a fraudproof which should ignore the uncommitted set of transactions, and export S1 into a fraudProof data structure (minimal snapshot)
+		5. Verify the fraudproof and check verification passes (done in light client)
+		6. Load a fresh baseapp, B2, with the contents of fraud proof data structure from (4) so it can begin from state S1.
+		7. Check if the SMT root hashes of the new app with saved SMT root hash


Suggested change

7. Check if the SMT root hashes of the new app with saved SMT root hash

7. Check if the root hashes of the new app with saved root hash

tzdybal · 2022-10-18T07:54:24Z

baseapp/baseapp_test.go

+		6. Load a fresh baseapp, B2, with the contents of fraud proof data structure from (4) so it can begin from state S1.
+		7. Check if the SMT root hashes of the new app with saved SMT root hash
+
+		Tests to write in future:


👍 👍 👍
I think we should create tracking issue for those tests and later implement them one by one.

Created a tracking issue: #3

Tests can be child issues of this issue later.

tzdybal · 2022-10-18T08:04:29Z

baseapp/fraudproof.go

+}
+
+// Returns true only if only one of the three pointers is nil
+func (fraudProof *FraudProof) checkFraudulentStateTransition() bool {


Please add unit test covering all possible cases.

tzdybal · 2022-10-18T08:08:52Z

baseapp/fraudproof.go

+
+func (fraudProof *FraudProof) verifyFraudProof() (bool, error) {
+	if !fraudProof.checkFraudulentStateTransition() {
+		return false, fmt.Errorf("fraudProof has more than one type of fradulent state transitions marked nil")


errors.New should be used instead of fmt.Errorf. This error should be extracted as package level sentinel error.

tzdybal · 2022-10-18T08:21:07Z

store/tracekv/store.go

@@ -17,6 +20,10 @@ const (
 	iterValueOp operation = "iterValue"
 )

+var (
+	ErrBufferEmpty = fmt.Errorf("provided buffer is empty")


Suggested change

ErrBufferEmpty = fmt.Errorf("provided buffer is empty")

ErrBufferEmpty = errors.New("provided buffer is empty")

baseapp/fraudproof.go

+	for k := range fraudProof.stateWitness {
+		keys = append(keys, k)
+	}


baseapp/fraudproof.go

+	for storeKey, stateWitness := range fraudProof.stateWitness {
+		proofOp := stateWitness.Proof
+		proof, err := types.CommitmentOpDecoder(proofOp)
+		if err != nil {
+			return false, err
+		}
+		if !bytes.Equal(proof.GetKey(), []byte(storeKey)) {
+			return false, fmt.Errorf("got storeKey: %s, expected: %s", string(proof.GetKey()), storeKey)
+		}
+		appHash, err := proof.Run([][]byte{stateWitness.RootHash})
+		if err != nil {
+			return false, err
+		}
+		if !bytes.Equal(appHash[0], fraudProof.appHash) {
+			return false, fmt.Errorf("got appHash: %s, expected: %s", string(fraudProof.appHash), string(fraudProof.appHash))
+		}
+		// Fraudproof verification on a substore level
+		for _, witness := range stateWitness.WitnessData {
+			proofOp, key, value := witness.Proof, witness.Key, witness.Value
+			if err != nil {
+				return false, err
+			}
+			if !bytes.Equal(key, proofOp.GetKey()) {
+				return false, fmt.Errorf("got key: %s, expected: %s for storeKey: %s", string(key), string(proof.GetKey()), storeKey)
+			}
+			proof, err := types.CommitmentOpDecoder(proofOp)
+			if err != nil {
+				return false, err
+			}
+			rootHash, err := proof.Run([][]byte{value})
+			if err != nil {
+				return false, err
+			}
+			if !bytes.Equal(rootHash[0], stateWitness.RootHash) {
+				return false, fmt.Errorf("got rootHash: %s, expected: %s for storeKey: %s", string(rootHash[0]), string(stateWitness.RootHash), storeKey)
+			}
+		}
+	}


baseapp/fraudproof.go

+	for storeKey, stateWitness := range fraudProof.stateWitness {
+		abciWitnessData := make([]*abci.WitnessData, 0, len(stateWitness.WitnessData))
+		for _, witnessData := range stateWitness.WitnessData {
+			abciWitness := abci.WitnessData{
+				Key:   witnessData.Key,
+				Value: witnessData.Value,
+				Proof: &witnessData.Proof,
+			}
+			abciWitnessData = append(abciWitnessData, &abciWitness)
+		}
+		proof := stateWitness.Proof
+		abciStateWitness[storeKey] = &abci.StateWitness{
+			Proof:       &proof,
+			RootHash:    stateWitness.RootHash,
+			WitnessData: abciWitnessData,
+		}
+	}


baseapp/fraudproof.go

+	for storeKey, abciStateWitness := range abciFraudProof.StateWitness {
+		witnessData := make([]WitnessData, 0, len(abciStateWitness.WitnessData))
+		for _, abciWitnessData := range abciStateWitness.WitnessData {
+			witness := WitnessData{
+				Key:   abciWitnessData.Key,
+				Value: abciWitnessData.Value,
+				Proof: *abciWitnessData.Proof,
+			}
+			witnessData = append(witnessData, witness)
+		}
+		stateWitness[storeKey] = StateWitness{
+			Proof:       *abciStateWitness.Proof,
+			RootHash:    abciStateWitness.RootHash,
+			WitnessData: witnessData,
+		}
+	}


store/rootmulti/store.go

+	for _, sk := range s.keysByName {
+		storeKeys = append(storeKeys, sk)
+	}


store/rootmulti/store.go

+	for _, storeParams := range rs.storesParams {
+		if storeParams.traceWriter != nil {
+			buf, ok := storeParams.traceWriter.(*bytes.Buffer)
+			if ok {
+				buf.Reset()
+			}
+		}
+	}


store/rootmulti/store.go

+	for key := range stores {
+		name := key.Name()
+		iavlStore, err := rs.GetIAVLStore(name)
+		if err != nil {
+			return nil, err
+		}
+		m[name], err = iavlStore.Root()
+		if err != nil {
+			return nil, err
+		}
+	}


baseapp/fraudproof.go

+	for storeKey, stateWitness := range fraudProof.stateWitness {
+		rootHash := stateWitness.RootHash
+		// TODO(manav): Replace with IAVL Deep Subtrees once implementation is done
+		substoreDeepSMT := smtlib.NewDeepSparseMerkleSubTree(smtlib.NewSimpleMap(), smtlib.NewSimpleMap(), sha256.New(), rootHash)
+		for _, witnessData := range stateWitness.WitnessData {
+			proofOp, key, val := witnessData.Proof, witnessData.Key, witnessData.Value
+			proof, err := types.CommitmentOpDecoder(proofOp)
+			if err != nil {
+				return nil, err
+			}
+			// TODO(manav): Replace with an IAVL proof instead of SMT here
+			smtProof := proof.(*smt.ProofOp).GetProof()
+			substoreDeepSMT.AddBranch(smtProof, key, val)
+		}
+		// TODO(manav): Replace with actual iavl stores
+		storeKeyToIAVLTree[storeKey] = &iavltree.MutableTree{}
+	}


…EndBlock is fraudulent

Manav-Aggarwal force-pushed the manav/fraudproofs_iavl branch from 4e5b2c6 to d6b7a07 Compare October 15, 2022 15:16

Manav-Aggarwal self-assigned this Oct 15, 2022

Manav-Aggarwal requested a review from tzdybal October 15, 2022 21:47

Manav-Aggarwal added the fraudproofs State Fraud Proofs label Oct 15, 2022

Manav-Aggarwal changed the title ~~Fraud Proof prototype using IAVL store~~ feat: Fraud Proof prototype using IAVL store Oct 16, 2022

Manav-Aggarwal marked this pull request as ready for review October 16, 2022 14:11

Manav-Aggarwal changed the title ~~feat: Fraud Proof prototype using IAVL store~~ feat: Create Fraud Proof prototype using IAVL store Oct 16, 2022

Manav-Aggarwal changed the title ~~feat: Create Fraud Proof prototype using IAVL store~~ feat: Create Fraud Proof prototype using IAVL store: No Deep Subtree Oct 16, 2022

Manav-Aggarwal changed the title ~~feat: Create Fraud Proof prototype using IAVL store: No Deep Subtree~~ feat: Fraud Proof generation and partial verification using IAVL store: No Deep Subtree Oct 16, 2022

Manav-Aggarwal changed the title ~~feat: Fraud Proof generation and partial verification using IAVL store: No Deep Subtree~~ feat: Fraud Proof generation and partial verification using IAVL store Oct 16, 2022

Manav-Aggarwal requested review from adlerjohn, jbowen93 and liamsi October 17, 2022 07:54

MSevey reviewed Oct 17, 2022

View reviewed changes

Manav-Aggarwal linked an issue Oct 18, 2022 that may be closed by this pull request

Fraud Proof generation and partial verification using IAVL store #2

Closed

tzdybal suggested changes Oct 18, 2022

View reviewed changes

github-advanced-security bot found potential problems Oct 31, 2022

View reviewed changes

Manav-Aggarwal added 13 commits November 1, 2022 15:21

Replace tendermint with celestiaorg tendermint version

51d06e8

Add new ABCI method skeleton: build works now

9396aab

Add fraudproof data structure

1975e89

Add helper methods to baseapp test for fraudproofs

8168e32

Add TODOs in baseapp test method

20e50ff

Add SetupBaseParams with SMT, WIP to IAVL

9801e3e

Add tracekv readOperations and getAllKeysUsedInTrace

b3737ae

Cleanup errors

a841e31

Add loadLastVersion to IAVL rootmulti store

afda438

Remove enableFraudProofGenerationMode

e9e96ef

Add substore tracing to rootmulti store

255115b

Add ability to reset trace writer buffers after each commit

3730b3c

Able to get the trace made by fraudulent state transition now

f61fc8d

Manav-Aggarwal added 11 commits November 1, 2022 15:28

Start setup BaseApp from IAVL store WIP

d3279d0

Add option to SetInitialHeight

3f51f21

Initialize baseapp from IAVL trees

2ee763e

Use deepIAVLTree in loadVersion

d3bba65

Add SetTracerFor to mock store

48b968c

Add gdocs to new ABCI methods

f52e1e2

Refactor from fmt.errorf into errors.New

536d58b

Update error packages used

6a6c843

Update docs to remove SMT references

d3e0a32

Add unit test for checkFraudulentStateTransition

19edc8e

modify executeNonFraudulentStateTransactions to accomodate case when …

da75457

…EndBlock is fraudulent

Manav-Aggarwal force-pushed the manav/fraudproofs_iavl branch from 2d8866b to da75457 Compare November 1, 2022 14:31

Refactor code

7db9131

tzdybal approved these changes Nov 8, 2022

View reviewed changes

Manav-Aggarwal merged commit 55d350f into manav/fraudproof_iavl_prototype Nov 8, 2022

Manav-Aggarwal deleted the manav/fraudproofs_iavl branch December 1, 2022 05:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Fraud Proof generation and partial verification using IAVL store #1

feat: Fraud Proof generation and partial verification using IAVL store #1

Manav-Aggarwal commented Oct 14, 2022 •

edited

Loading

MSevey Oct 17, 2022

Manav-Aggarwal Oct 18, 2022

tzdybal left a comment

tzdybal Oct 17, 2022

Manav-Aggarwal Oct 20, 2022

tzdybal Oct 17, 2022

Manav-Aggarwal Oct 20, 2022

tzdybal Oct 18, 2022

tzdybal Oct 18, 2022

Manav-Aggarwal Oct 20, 2022

tzdybal Oct 18, 2022

Manav-Aggarwal Oct 20, 2022 •

edited

Loading

tzdybal Oct 18, 2022

tzdybal Oct 18, 2022

	7. Check if the SMT root hashes of the new app with saved SMT root hash
	7. Check if the root hashes of the new app with saved root hash

	ErrBufferEmpty = fmt.Errorf("provided buffer is empty")
	ErrBufferEmpty = errors.New("provided buffer is empty")

feat: Fraud Proof generation and partial verification using IAVL store #1

feat: Fraud Proof generation and partial verification using IAVL store #1

Conversation

Manav-Aggarwal commented Oct 14, 2022 • edited Loading

Description

Author Checklist

Reviewers Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzdybal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Manav-Aggarwal Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Manav-Aggarwal commented Oct 14, 2022 •

edited

Loading

Manav-Aggarwal Oct 20, 2022 •

edited

Loading