feat: value extractor (#37)

* add: `reddit_response.json` * refactor tests + add failing case * easier fix * test: parse to key * tests: key parsing * bug: `next_end_of_kv` on read `:` * fix: `end_of_kv` bug * test: find value * tests: `inside_value` and `inside_value_to_exit` * test: parse to NEXT key * parses JSON with two string keys * WIP: value inside value * comment * refactor (#10) * wip: start with bitmask * WIP: time to start testing * tests: `ArrayAdd` and `ArrayMul` * tests passing * update comments * feat: 2 key depth 1 json * 2 kv json and all tests passing * nested json works!!! * reduce constraints * cleanup * rename variables * more cleaning * more cleanup * make comments clean * WAYLON NITPICKING ME LOL * feat: improved CLI for witness * gitignore input.json * Update main.rs * feat: update rust * feat: parse with array as value * feat; `InRange` template * WIP: number parsing * good stopping point * compiling again * stack hard * save progress * save wip * getting through tests * add optimised substring search * resolve random input bug using poseidon hash * import circomlib and remove circuits * add array and hash circuits * add search * add js poseidon * update tests * add search tests * wip: stack * big ints killing me * use strings for tests * reduce and fix some tests! * refactor tests * reduce and fix some tests! * moar tests * cosmetics * continuing with tests * save state * renumbered stack entries * wip: handle end of arr/obj * maybe progress * save * almost! * tests passing * cleaning * wip: big example * bug: fails to parse example.json * add basic switch template again * feat: `GetTopOfStack` Gotta work with this to make life easier. We can make easier conditionals for this function and perhaps not even need to store a pointer and just store the stack! Other notes: The issue is that parsing commas in arrays will return back a -3 value to overwrite a stack position, but this will fail when inside of an array. So, if we know that the top of stack is a 2 (for inside of an array) we can make conditionals on this. We could return even something like -4 for the comma so we have to transform it depending on what the top of stack is * IT"S LIVING * refactor: use `circomlib` directly * refactor: JSONs * redo all in `circuits.json` * changeable stack height * save state * reorganize tests * small cleanup * continuing test cleanup * remove antiquated tests * cleanup * refactor: `language.circom` * todo note * good save state * good state! * Update notes.md * 2d stack * basic array tracking * tests passing but still not properly indexing in arrays * small cleanup * almost there! * arrays! * satisfying! * another example * Update notes.md * save notes * Squashed commit of the following: commit ed2c440 Author: Colin Roberts <colin@autoparallel.xyz> Date: Thu Aug 15 16:32:44 2024 -0600 tests/refactor: state update and improved JSON parsing (#11) * add: `reddit_response.json` * refactor tests + add failing case * easier fix * test: parse to key * tests: key parsing * bug: `next_end_of_kv` on read `:` * fix: `end_of_kv` bug * test: find value * tests: `inside_value` and `inside_value_to_exit` * test: parse to NEXT key * parses JSON with two string keys * WIP: value inside value * comment * refactor (#10) * wip: start with bitmask * WIP: time to start testing * tests: `ArrayAdd` and `ArrayMul` * tests passing * update comments * feat: 2 key depth 1 json * 2 kv json and all tests passing * nested json works!!! * reduce constraints * cleanup * rename variables * more cleaning * more cleanup * make comments clean * WAYLON NITPICKING ME LOL clean up merge commit * WIP: remove `pointer` state * WIP: cleaning further Many stack tests passing. I want to now not push to stack when we read a colon and instead enable use of the second stack position. This will break more things temporarily. * WIP: reduced many variables Lot's of complexity cleared away and many important tests are passing again. Still WIP, but getting close. * all tests pass * updated circuits * formatting * add zkemail * extracting string yayyy!!!!! * add tests for string extractor * parse numbers * convert unicode number array to number * extracting array index * yay!! multi-depth key extraction done. onto more nested structures noww * extract nested array working * add nested structure example circuit * delete utils * move input generation to separate binary * initial codegen * fix: correct hardcoded signals * add codegen example * remove old code and rename to interpreter * codegen changes * add test files * feat(codegen): add output filename * complete tests * chore(tests): remove duplicate * chore(docs): interpreter * tests: add interpreter * fix(test): add tests failing in parallel * fix(codegen): add poseidon hasher * Merge branch 'main' into temp-fetcher * refactor: code reorganisation * experiment with mocha parallel tests * review nits * handle conflicting files --------- Co-authored-by: Colin Roberts <colin@autoparallel.xyz>
pluto · Aug 28, 2024 · 6a48ff5 · 6a48ff5
1 parent 9a44739
commit 6a48ff5
Show file tree

Hide file tree

Showing 30 changed files with 2,612 additions and 101 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -25,11 +25,11 @@ jobs:
 
     - name: Download and install Circom
       run: |
-        CIRCOM_VERSION=2.1.9  
+        CIRCOM_VERSION=2.1.9
         curl -L https://github.com/iden3/circom/releases/download/v$CIRCOM_VERSION/circom-linux-amd64 -o circom
         chmod +x circom
         sudo mv circom /usr/local/bin/
-        circom --version 
+        circom --version
 
     - name: Run tests
-      run: npx mocha
+      run: npm run par-test
diff --git a/circuits/http/parser_request/language.circom b/circuits/http/parser_request/language.circom
@@ -0,0 +1,49 @@
+pragma circom 2.1.9;
+
+// All the possible request methods: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods
+
+template Syntax() {
+    //-Delimeters---------------------------------------------------------------------------------//
+    // - ASCII char `:`
+    signal output COLON     <== 58;
+    // - ASCII char `;`
+    signal output SEMICOLON <== 59;
+    // - ASCII char `,`
+    signal output COMMA     <== 44;
+    // - ASCII char `"`
+    signal output QUOTE     <== 34;
+    //-White_space--------------------------------------------------------------------------------//
+    // - ASCII pair: `\r\n`
+    signal output CLRF   <== [13, 10]; // https://www.rfc-editor.org/rfc/rfc2616#section-2.2
+                                       // https://www.rfc-editor.org/rfc/rfc7230#section-3.5
+    // - ASCII char: ` `
+    signal output SPACE     <== 32;
+    //-Escape-------------------------------------------------------------------------------------//
+    // - ASCII char: `\`
+    signal output ESCAPE    <== 92;
+}
+
+template RequestMethod() {
+    signal output GET[3]  <== [71, 69, 84];
+    // signal output HEAD[4] <== [72, 69, 65, 68];
+    signal output POST[4] <== [80, 79, 83, 84];
+    // signal output PUT     <== 3;
+    // signal output DELETE  <== 4;
+    // signal output CONNECT <== 5;
+    // signal output OPTIONS <== 6;
+    // signal output TRACE   <== 7;
+    // signal output PATCH   <== 8;
+}
+
+// NOTE: Starting at 1 to avoid a false positive with a 0.
+template RequestMethodTag() {
+    signal output GET  <== 1;
+    // signal output HEAD <== 2;
+    signal output POST <== 3;
+    // signal output PUT     <== 4;
+    // signal output DELETE  <== 5;
+    // signal output CONNECT <== 6;
+    // signal output OPTIONS <== 7;
+    // signal output TRACE   <== 8;
+    // signal output PATCH   <== 9;
+}
diff --git a/circuits/http/parser_request/machine.circom b/circuits/http/parser_request/machine.circom
@@ -0,0 +1,28 @@
+pragma circom 2.1.9;
+
+include "language.circom";
+include "../utils/array.circom";
+
+template ParseMethod() {
+    signal input bytes[7];
+    signal output MethodTag;
+
+    component RequestMethod = RequestMethod();
+    component RequestMethodTag = RequestMethodTag();
+
+    component IsGet = IsEqualArray(3);
+    for(var byte_idx = 0; byte_idx < 3; byte_idx++) {
+        IsGet.in[0][byte_idx] <== bytes[byte_idx];
+        IsGet.in[1][byte_idx] <== RequestMethod.GET[byte_idx];
+    }
+    signal TagGet <== IsGet.out * RequestMethodTag.GET;
+
+    component IsPost = IsEqualArray(4);
+    for(var byte_idx = 0; byte_idx < 4; byte_idx++) {
+        IsPost.in[0][byte_idx] <== bytes[byte_idx];
+        IsPost.in[1][byte_idx] <== RequestMethod.POST[byte_idx];
+    }
+    signal TagPost <== IsPost.out * RequestMethodTag.POST;
+
+    MethodTag <== TagGet + TagPost;
+}
diff --git a/circuits/http/parser_request/parser.circom b/circuits/http/parser_request/parser.circom
@@ -0,0 +1,24 @@
+pragma circom 2.1.9;
+
+include "../utils/bytes.circom";
+include "machine.circom";
+
+
+template Parser(DATA_BYTES) {
+    signal input data[DATA_BYTES];
+
+    signal output Method;
+
+    //--------------------------------------------------------------------------------------------//
+    //-CONSTRAINTS--------------------------------------------------------------------------------//
+    //--------------------------------------------------------------------------------------------//
+    component dataASCII = ASCII(DATA_BYTES);
+    dataASCII.in <== data;
+    //--------------------------------------------------------------------------------------------//
+
+    component ParseMethod = ParseMethod();
+    for(var byte_idx = 0; byte_idx < 7; byte_idx++) {
+        ParseMethod.bytes[byte_idx] <== data[byte_idx];
+    }
+    log("MethodTag: ", ParseMethod.MethodTag);
+}