feat: parser state machine

[Autoparallel]

• Switch template
• Parser state machine that parses the input data and lays down a list of "instructions" that label what state transitions should be taken by the machine at a given byte index of that data. E.g., if we hit an open brace when we are not inside of a value or inside of a key, we should increase tree depth.

---

Alright the above two tasks are pretty much done for simpler JSON. Where I am at with this now is I have created a few circuits. They are in circuits.json which circomkit uses to do some automations with them! I found this quite helpful.

What I was working with in particular was test_extract which digests the json_examples/test.json. To try this out, run:

npx circomkit compile test_extract
npx circomkit witness test_extract input

and you should see some logs appear (note, follow README instructions to get circomkit up and running). These logs track the state of the Parser component you'll find in circuits/extract.circom and circuits/parser.circom.

I left notes in circuits/parser.circom that walk through expected states we see given what we have just read at a given byte index. You can validate yourself by clicking through test.json.

Nonetheless, you should get code that verifies this witness meaning that, in some sense, it is valid JSON! Also, we have marked where the keys are by enabling the inside_key bit and marked where values are by enabling the inside_value bit.

Output would look like this 😄:

❯ npx circomkit witness test_extract input
State[ 1 ].tree_depth =  1
State[ 1 ].parsing_to_key =  1
State[ 1 ].inside_key =  0
State[ 1 ].parsing_to_value =  0
State[ 1 ].inside_value =  0
State[ 1 ].end_of_kv =  0
---
State[ 2 ].tree_depth =  1
State[ 2 ].parsing_to_key =  1
State[ 2 ].inside_key =  0
State[ 2 ].parsing_to_value =  0
State[ 2 ].inside_value =  0
State[ 2 ].end_of_kv =  0
---
State[ 3 ].tree_depth =  1
State[ 3 ].parsing_to_key =  1
State[ 3 ].inside_key =  0
State[ 3 ].parsing_to_value =  0
State[ 3 ].inside_value =  0
State[ 3 ].end_of_kv =  0
---
State[ 4 ].tree_depth =  1
State[ 4 ].parsing_to_key =  1
State[ 4 ].inside_key =  0
State[ 4 ].parsing_to_value =  0
State[ 4 ].inside_value =  0
State[ 4 ].end_of_kv =  0
---
State[ 5 ].tree_depth =  1
State[ 5 ].parsing_to_key =  1
State[ 5 ].inside_key =  0
State[ 5 ].parsing_to_value =  0
State[ 5 ].inside_value =  0
State[ 5 ].end_of_kv =  0
---
State[ 6 ].tree_depth =  1
State[ 6 ].parsing_to_key =  1
State[ 6 ].inside_key =  0
State[ 6 ].parsing_to_value =  0
State[ 6 ].inside_value =  0
State[ 6 ].end_of_kv =  0
---
State[ 7 ].tree_depth =  1
State[ 7 ].parsing_to_key =  0
State[ 7 ].inside_key =  1
State[ 7 ].parsing_to_value =  0
State[ 7 ].inside_value =  0
State[ 7 ].end_of_kv =  0
---
State[ 8 ].tree_depth =  1
State[ 8 ].parsing_to_key =  0
State[ 8 ].inside_key =  1
State[ 8 ].parsing_to_value =  0
State[ 8 ].inside_value =  0
State[ 8 ].end_of_kv =  0
---
State[ 9 ].tree_depth =  1
State[ 9 ].parsing_to_key =  0
State[ 9 ].inside_key =  1
State[ 9 ].parsing_to_value =  0
State[ 9 ].inside_value =  0
State[ 9 ].end_of_kv =  0
---
State[ 10 ].tree_depth =  1
State[ 10 ].parsing_to_key =  0
State[ 10 ].inside_key =  1
State[ 10 ].parsing_to_value =  0
State[ 10 ].inside_value =  0
State[ 10 ].end_of_kv =  0
---
State[ 11 ].tree_depth =  1
State[ 11 ].parsing_to_key =  0
State[ 11 ].inside_key =  1
State[ 11 ].parsing_to_value =  0
State[ 11 ].inside_value =  0
State[ 11 ].end_of_kv =  0
---
State[ 12 ].tree_depth =  1
State[ 12 ].parsing_to_key =  0
State[ 12 ].inside_key =  0
State[ 12 ].parsing_to_value =  0
State[ 12 ].inside_value =  0
State[ 12 ].end_of_kv =  0
---
State[ 13 ].tree_depth =  1
State[ 13 ].parsing_to_key =  0
State[ 13 ].inside_key =  0
State[ 13 ].parsing_to_value =  1
State[ 13 ].inside_value =  0
State[ 13 ].end_of_kv =  1
---
State[ 14 ].tree_depth =  1
State[ 14 ].parsing_to_key =  0
State[ 14 ].inside_key =  0
State[ 14 ].parsing_to_value =  1
State[ 14 ].inside_value =  0
State[ 14 ].end_of_kv =  0
---
State[ 15 ].tree_depth =  1
State[ 15 ].parsing_to_key =  0
State[ 15 ].inside_key =  0
State[ 15 ].parsing_to_value =  0
State[ 15 ].inside_value =  1
State[ 15 ].end_of_kv =  0
---
State[ 16 ].tree_depth =  1
State[ 16 ].parsing_to_key =  0
State[ 16 ].inside_key =  0
State[ 16 ].parsing_to_value =  0
State[ 16 ].inside_value =  1
State[ 16 ].end_of_kv =  0
---
State[ 17 ].tree_depth =  1
State[ 17 ].parsing_to_key =  0
State[ 17 ].inside_key =  0
State[ 17 ].parsing_to_value =  0
State[ 17 ].inside_value =  1
State[ 17 ].end_of_kv =  0
---
State[ 18 ].tree_depth =  1
State[ 18 ].parsing_to_key =  0
State[ 18 ].inside_key =  0
State[ 18 ].parsing_to_value =  0
State[ 18 ].inside_value =  1
State[ 18 ].end_of_kv =  0
---
State[ 19 ].tree_depth =  1
State[ 19 ].parsing_to_key =  0
State[ 19 ].inside_key =  0
State[ 19 ].parsing_to_value =  0
State[ 19 ].inside_value =  0
State[ 19 ].end_of_kv =  0
---
State[ 20 ].tree_depth =  1
State[ 20 ].parsing_to_key =  0
State[ 20 ].inside_key =  0
State[ 20 ].parsing_to_value =  0
State[ 20 ].inside_value =  0
State[ 20 ].end_of_kv =  1
---
Witness created: ./build/test_extract/input/witness.wtns

---

There are next steps. So I have other JSON there to test the limits of this. Also, I should write some tests on valid state transitions for the parser. I did not do this yet. Please hold me accountable and make sure I make issues for these things.

Signing off for now. Gn.

[thor314]

appreciate the comments, looks like a good testin flow

[thor314]

nice, looks like a good test flow

[thor314]

good notes

[Autoparallel]

@thor314 thank ser