Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: byte slice JSON parser #1415

Merged
merged 80 commits into from
Mar 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
f1308d3
numberKind byte slice parser
notJoon Dec 6, 2023
c30634c
remove unused packages
notJoon Dec 6, 2023
7b0e6c2
add utf16 package
notJoon Dec 7, 2023
cf0655e
add escape handler
notJoon Dec 7, 2023
dbeb921
parse float
notJoon Dec 8, 2023
d39ad52
basic lexer
notJoon Dec 11, 2023
1d2a600
update lexer
notJoon Dec 11, 2023
26f9c70
key position range finder
notJoon Dec 12, 2023
32760c4
find multiple key positions in JSON
notJoon Dec 12, 2023
1de7f0c
parsing Integer and float values
notJoon Dec 12, 2023
e19a06d
number parser refactoring
notJoon Dec 12, 2023
9ba8808
parse primitive types
notJoon Dec 12, 2023
534f0f7
create searchKeys
notJoon Dec 13, 2023
f12375d
get type value
notJoon Dec 13, 2023
6868c10
parse array
notJoon Dec 13, 2023
1a1657f
remove Lexer struct
notJoon Dec 13, 2023
39bfa64
refactor
notJoon Dec 13, 2023
33ccb14
re-refactor
notJoon Dec 13, 2023
789abfd
parse uint value
notJoon Dec 13, 2023
7de39d5
revert
notJoon Dec 14, 2023
cc136d5
JSON PoC
notJoon Dec 15, 2023
7e9ccb8
remove unused errors
notJoon Dec 15, 2023
b69eadb
flatten
notJoon Dec 15, 2023
6e72466
investigate flattening
notJoon Dec 15, 2023
83a58eb
refactor
notJoon Dec 15, 2023
57916bc
type assertion in flatten
notJoon Dec 17, 2023
5a54e9a
save
notJoon Dec 19, 2023
5f138b6
key extract
notJoon Dec 20, 2023
ebef5d5
re-organize the files
notJoon Dec 20, 2023
2b35743
save
notJoon Dec 20, 2023
40faddb
fix array
notJoon Dec 23, 2023
fbc13fc
struct parser
notJoon Jan 4, 2024
7041cfc
struct parser finish
notJoon Jan 4, 2024
9fee201
marshaling
notJoon Jan 8, 2024
16e9cf3
handle type formatting
notJoon Jan 8, 2024
e5964bd
handle type formatting
notJoon Jan 8, 2024
d3d0ff7
fmt
notJoon Jan 8, 2024
fffe9d1
add more types for marshaling
notJoon Jan 10, 2024
188e39c
define dynamic structs and generate JSON
notJoon Jan 11, 2024
cc0c1ed
reduce memory alloc when marshal
notJoon Jan 12, 2024
3135854
CRUD for struct instance
notJoon Jan 16, 2024
4f93bc2
add output
notJoon Jan 16, 2024
e8a8251
Merge branch 'master' into json
notJoon Jan 16, 2024
7a62f93
add basic linter and example
notJoon Jan 17, 2024
807d77d
add lookup table
notJoon Jan 22, 2024
6fd80ce
Merge branch 'master' into json
notJoon Feb 6, 2024
a49efbb
change fmt package into p/demo/ufmt
notJoon Feb 11, 2024
95e41ed
regex as global
notJoon Feb 11, 2024
ee9eec9
fmt
notJoon Feb 11, 2024
db0c8ab
fixup
notJoon Feb 11, 2024
48049c2
create EachKey
notJoon Feb 12, 2024
ecbc853
Merge branch 'master' into json
notJoon Feb 12, 2024
d8c6498
fixup
notJoon Feb 15, 2024
ac77d71
fmt
notJoon Feb 15, 2024
680f152
create error code
notJoon Feb 19, 2024
e8ab3cf
basic buffer
notJoon Feb 19, 2024
2ab0eed
License
notJoon Feb 19, 2024
e3298e3
json path parser
notJoon Feb 19, 2024
ecaab8a
json path token handler
notJoon Feb 21, 2024
2faabe2
add test case
notJoon Feb 22, 2024
f7716c4
save
notJoon Feb 22, 2024
2c6fc6e
Merge branch 'master' into json
notJoon Feb 28, 2024
de817ba
fmt and mv license
notJoon Mar 2, 2024
29224d9
state machine decoder and some node getters
notJoon Mar 5, 2024
42b4a15
rewrite JSON code to avoid struct
notJoon Mar 12, 2024
03a9d17
use ryu algorithm to format float and reorganize the structure
notJoon Mar 12, 2024
cc4cd3e
tidy
notJoon Mar 12, 2024
2218981
fix DeleteIndex
notJoon Mar 13, 2024
faf9890
Improve determining int and float type
notJoon Mar 13, 2024
33ab79a
resolve conflict
notJoon Mar 13, 2024
a688fc5
Merge branch 'master' into json
notJoon Mar 13, 2024
a54cfab
remove utf16 package from json
notJoon Mar 13, 2024
c999121
Merge branch 'master' into json
notJoon Mar 28, 2024
88e93cc
refactor
notJoon Mar 28, 2024
8bffa48
some lint
notJoon Mar 28, 2024
aee3526
save
notJoon Mar 29, 2024
447718a
Merge branch 'master' into json
notJoon Mar 29, 2024
bd14f50
refactor and update README
notJoon Mar 29, 2024
c693ccc
Merge branch 'master' into json
notJoon Mar 29, 2024
209a754
typo
notJoon Mar 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions examples/gno.land/p/demo/json/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# MIT License

Copyright (c) 2019 Pyzhov Stepan

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
170 changes: 170 additions & 0 deletions examples/gno.land/p/demo/json/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# JSON Parser

The JSON parser is a package that provides functionality for parsing and processing JSON strings. This package accepts JSON strings as byte slices.

Currently, gno does not [support the `reflect` package](https://docs.gno.land/concepts/effective-gno#reflection-is-never-clear), so it cannot retrieve type information at runtime. Therefore, it is designed to infer and handle type information when parsing JSON strings using a state machine approach.

After passing through the state machine, JSON strings are represented as the `Node` type. The `Node` type represents nodes for JSON data, including various types such as `ObjectNode`, `ArrayNode`, `StringNode`, `NumberNode`, `BoolNode`, and `NullNode`.

This package provides methods for manipulating, searching, and extracting the Node type.

## State Machine

To parse JSON strings, a [finite state machine](https://en.wikipedia.org/wiki/Finite-state_machine) approach is used. The state machine transitions to the next state based on the current state and the input character while parsing the JSON string. Through this method, type information can be inferred and processed without reflect, and the amount of parser code can be significantly reduced.

The image below shows the state transitions of the state machine according to the states and input characters.

```mermaid
stateDiagram-v2
[*] --> __: Start
__ --> ST: String
__ --> MI: Number
__ --> ZE: Zero
__ --> IN: Integer
__ --> T1: Boolean (true)
__ --> F1: Boolean (false)
__ --> N1: Null
__ --> ec: Empty Object End
__ --> cc: Object End
__ --> bc: Array End
__ --> co: Object Begin
__ --> bo: Array Begin
__ --> cm: Comma
__ --> cl: Colon
__ --> OK: Success/End
ST --> OK: String Complete
MI --> OK: Number Complete
ZE --> OK: Zero Complete
IN --> OK: Integer Complete
T1 --> OK: True Complete
F1 --> OK: False Complete
N1 --> OK: Null Complete
ec --> OK: Empty Object Complete
cc --> OK: Object Complete
bc --> OK: Array Complete
co --> OB: Inside Object
bo --> AR: Inside Array
cm --> KE: Expecting New Key
cm --> VA: Expecting New Value
cl --> VA: Expecting Value
OB --> ST: String in Object (Key)
OB --> ec: Empty Object
OB --> cc: End Object
AR --> ST: String in Array
AR --> bc: End Array
KE --> ST: String as Key
VA --> ST: String as Value
VA --> MI: Number as Value
VA --> T1: True as Value
VA --> F1: False as Value
VA --> N1: Null as Value
OK --> [*]: End
```

## Examples

This package provides parsing functionality along with encoding and decoding functionality. The following examples demonstrate how to use this package.

### Decoding

Decoding (or Unmarshaling) is the functionality that converts an input byte slice JSON string into a `Node` type.

The converted `Node` type allows you to modify the JSON data or search and extract data that meets specific conditions.

```go
package main

import (
"fmt"
"gno.land/p/demo/json"
"gno.land/p/demo/ufmt"
)

func main() {
node, err := json.Unmarshal([]byte(`{"foo": "var"}`))
if err != nil {
ufmt.Errorf("error: %v", err)
}

ufmt.Sprintf("node: %v", node)
}
```

### Encoding

Encoding (or Marshaling) is the functionality that converts JSON data represented as a Node type into a byte slice JSON string.

> ⚠️ Caution: Converting a large `Node` type into a JSON string may _impact performance_. or might be cause _unexpected behavior_.

```go
package main

import (
"fmt"
"gno.land/p/demo/json"
"gno.land/p/demo/ufmt"
)

func main() {
node := ObjectNode("", map[string]*Node{
"foo": StringNode("foo", "bar"),
"baz": NumberNode("baz", 100500),
"qux": NullNode("qux"),
})

b, err := json.Marshal(node)
if err != nil {
ufmt.Errorf("error: %v", err)
}

ufmt.Sprintf("json: %s", string(b))
}
```

### Searching

Once the JSON data converted into a `Node` type, you can **search** and **extract** data that satisfy specific conditions. For example, you can find data with a specific type or data with a specific key.

To use this functionality, you can use methods in the `GetXXX` prefixed methods. The `MustXXX` methods also provide the same functionality as the former methods, but they will **panic** if data doesn't satisfies the condition.

Here is an example of finding data with a specific key. For more examples, please refer to the [node.gno](node.gno) file.

```go
package main

import (
"fmt"
"gno.land/p/demo/json"
"gno.land/p/demo/ufmt"
)

func main() {
root, err := Unmarshal([]byte(`{"foo": true, "bar": null}`))
if err != nil {
ufmt.Errorf("error: %v", err)
}

value, err := root.GetKey("foo")
if err != nil {
ufmt.Errorf("error occurred while getting key, %s", err)
}

if value.MustBool() != true {
ufmt.Errorf("value is not true")
}

value, err = root.GetKey("bar")
if err != nil {
t.Errorf("error occurred while getting key, %s", err)
}

_, err = root.GetKey("baz")
if err == nil {
t.Errorf("key baz is not exist. must be failed")
}
}
```

## Contributing

Please submit any issues or pull requests for this package through the GitHub repository at [gnolang/gno](<https://github.com/gnolang/gno>).
Loading
Loading