refactor!: use big endian and simplify transcript logic

[alxiong]

Part of #1739
Closes #1797

This PR:

• Point to the latest release of jf-plonk which modify the PlonkTranscript logic
• Remove endian reversal for field elements
• For G1Point, instead of using compressed form via g1Serialize, simply concatenate two base fields to the transcript byte array
• Update _computeChallenges() to reflect the same change

Gas benchmark

@@ -1,2 +1,2 @@
-LightClient_newFinalizedState_Test:testCorrectUpdateBench() (gas: 594565)
-PlonkVerifier_verify_Test:test_verify_succeeds() (gas: 507806)
\ No newline at end of file
+LightClient_newFinalizedState_Test:testCorrectUpdateBench() (gas: 661721)
+PlonkVerifier_verify_Test:test_verify_succeeds() (gas: 496676)

I'm not 100% sure why the cost of PlonkVerifier goes down but the LightClient cost goes up.
(see if you have any idea @philippecamacho ?)

[alxiong]

⚠️ WIP

I'm still debugging a mysterious failure. Currently computeChallenges test is failing (thus most of proof related tests are also failing). But all tests regarding Transcripts are passing now, which includes all the logic that _computeChallenges uses.

Diving even deeper, the challenge beta and gamma is computed correctly, but alpha, and all subsequent ones are wrong. I still unable to comprehend what could be the source of this inconsistency. Will continue to debug tmr.

---

Oh wow, this is such an unexpected behavior, somehow adding a console.log line pass all the test

        transcript.transcript = abi.encodePacked(
            transcript.transcript, proof.wire3.x, proof.wire3.y, proof.wire4.x, proof.wire4.y
        );

        // UNCOMMENT THIS LINE, THEN ALL FAILED TEST PASSED
        // console2.logBytes(transcript.transcript);
        res.beta = transcript.getAndAppendChallenge();
        res.gamma = transcript.getAndAppendChallenge();

🤯 investigating why....

[alxiong]

Bug (that I introduce in this PR)

Let me document a minimally reproducible bug (of my own code) that blocks me for days:

import "forge-std/Test.sol";
import { Transcript as T } from "../src/libraries/Transcript.sol";

library Transcript {
    struct TranscriptData {
        bytes transcript;
    }
    function getAndAppendChallenge(TranscriptData memory self) internal pure returns (uint256) {
        bytes memory transcript = self.transcript;
        uint256 ret = uint256(keccak256(transcript)) % BN254.R_MOD;

        assembly {
            let len := mload(transcript)
            let newLen := add(len, 32)
            let dataPtr := add(transcript, 0x20)

            mstore(transcript, newLen)
            mstore(add(dataPtr, len), ret)
        }
        return ret;
    }
}

contract Whatever is Test {
    // run with `forge test --mt whatever -vv`
    function test_whatever() external {
        T.TranscriptData memory transcript;
        transcript.transcript = abi.encodePacked(
            transcript.transcript,
            uint256(0x1234)
        );

        // HERE !! try to comment and uncomment
        console2.log("hi");

        uint256 chal = T.getAndAppendChallenge(transcript);
        console2.log("chal: %x", chal);
        console2.logBytes(transcript.transcript);
        revert(); // deliberately revert to print logs
    }
}

so comment out the console2.log before the getAndAppendChallenge(): you would get:

  chal: 0x21909f356f36b5c053a1b997dd03e86a313c86db7ca957c0840832cca43b62c1
  0x0000000000000000000000000000000000000000000000000000000000001234
    0000000000000000000000000000000000000000000000000000000000000008

uncomment it, you would get the correct/expected transcript bytes:

  chal: 0x21909f356f36b5c053a1b997dd03e86a313c86db7ca957c0840832cca43b62c1
0x0000000000000000000000000000000000000000000000000000000000001234
  21909f356f36b5c053a1b997dd03e86a313c86db7ca957c0840832cca43b62c1

Diagonsis

I couldn't wrap my head around this behavior, and only after went into the debugger and look at stack and memory content and stepping through instruction by instruction do I realize the problem comes from:

my inline assembly code in getAndAppendChallenge() only updated the transcript's length but not the free memory pointer stored at 0x40 memory address. Thus later function call would overwrite the first 32 bytes the outdated free memory points to.
🤦

Reason why injecting a console.log between transcript.transcript = abi.encodePacked(transcript.transcript) and transcript.getAndAppendChallenge() makes things back to correct behavior is:

that console.log() call re-adjust the free memory pointer to point to somewhere much deeper in the memory, thus leaving the entire transcript region untact.

Knowing my problem, I find this StackOverflow question that discussed the same issue.

Solution

simply add one line to update the free memory pointer towards the end:

        assembly {
            // ...

            // update free memory pointer since we extend the dynamic array
            // to prevent potential overwrite
            mstore(0x40, add(mload(0x40), 32))
        }

Takeaway

Be careful about memory safety when editing dynamic memory using inline assembly.
Remember to update both the length field in the struct but also the free pointer memory at 0x40

[philippecamacho]

Re benchmarks:

When I comment these three lines the gas consumption is around 170k.

When I do the same in b3ec621 I get 90k. So there is around a 80k gas leakage in this part of the code. Weird.

[philippecamacho]

Re benchmarks:

When I comment these three lines the gas consumption is around 170k.

When I do the same in b3ec621 I get 90k. So there is around a 80k gas leakage in this part of the code. Weird.

@alxiong : #1816 created so that we can come back to it later.

[alxiong]

Also found this on Solidity doc about safe memory management.

Apparently I didn't update myself with this warning. Lessons learned.