Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: byte slice JSON parser (#1415)
## Description I implemented a JSON parser using byte slices ### State Machine For JSON Each state transitions to the next when specific conditions are met, representing the process of parsing a JSON structure sequentially. State transitions are defined based on the conditions and actions that occur during the parsing process. The diagram essentially includes the following states (All states are defined at `internal.gno` file): - It starts in the initial state (`__`), returning an error if an unexpected token is encountered. - States for handling string (`ST`), number (`MI`, `ZE`, `IN`), boolean (`T1`, `F1`), and null (`N1`) values. - States for handling the start and end of objects (`co`) and arrays (`bo`) (`ec`, `cc`, `bc`). - States expecting keys (`KE`) and values (`VA`), and for handling commas (`cm`) and colons (`cl`). Each state deals with various scenarios that can occur during JSON parsing, with transitions to the next state determined by the current token and context. Below is a graph depicting how the states transition: ```mermaid stateDiagram-v2 [*] --> __: Start __ --> ST: String __ --> MI: Number __ --> ZE: Zero __ --> IN: Integer __ --> T1: Boolean (true) __ --> F1: Boolean (false) __ --> N1: Null __ --> ec: Empty Object End __ --> cc: Object End __ --> bc: Array End __ --> co: Object Begin __ --> bo: Array Begin __ --> cm: Comma __ --> cl: Colon __ --> OK: Success/End ST --> OK: String Complete MI --> OK: Number Complete ZE --> OK: Zero Complete IN --> OK: Integer Complete T1 --> OK: True Complete F1 --> OK: False Complete N1 --> OK: Null Complete ec --> OK: Empty Object Complete cc --> OK: Object Complete bc --> OK: Array Complete co --> OB: Inside Object bo --> AR: Inside Array cm --> KE: Expecting New Key cm --> VA: Expecting New Value cl --> VA: Expecting Value OB --> ST: String in Object (Key) OB --> ec: Empty Object OB --> cc: End Object AR --> ST: String in Array AR --> bc: End Array KE --> ST: String as Key VA --> ST: String as Value VA --> MI: Number as Value VA --> T1: True as Value VA --> F1: False as Value VA --> N1: Null as Value OK --> [*]: End ``` ## Walkthrough The JSON Machine Gno is not completely compatible with Go, which means that many functions within the standard library are not fully implemented yet. Therefore, some files are added not directly related to JSON but necessary for functionality implementation. ### Float Value Handler The `strconv` package currently provided by gno has functions injected for parsing basic `int` and `uint` types, but does not have an implementation for parsing floating-point numbers with `ParseFloat`. Therefore, I have brought over the implementation of the `eisel-lemire` algorithm from Go's strconv package (`./p/demo/json/eisel_lemire`).<sup>[1](#footnote_1)</sup> Additionally, since the `FormatFloat` function is also not implemented yet. So, I imported the `ryu64` algorithm <sup>[2](#footnote_2)</sup> to implement this feature (`./p/demo/json/ryu`). Anyway, I plan to add this code to the strconv package if possible, so that the necessary functionality and functions can be completely written in gno. ### Buffer `buffer.gno` manages internal buffer management and interaction with the state machine for JSON parsing. The buffer processes the JSON string sequentially, interpreting the meaning of each character and deciding the next action through the state machine. Here, I'll describe the key functions and how they interact with the state machine. The `/` next to a number is a notation borrowed from Elixir to indicate the number of parameters: - `newBuffer`: This function creates a new buffer instance containing the given data. The initial state is set to `GO`, signifying the start of parsing and preparing for subsequent parsing stages as the state machine's initial state. - `first`: Finds the first meaningful (non-whitespace) character. Although the state machine is not yet activated at this stage, the result of this function plays a crucial role in determining the first step of parsing. - `current`, `next`, `step`: These functions manage the current position within the buffer, reading characters or moving to the next one. `current` returns the character at the current index, `next` returns the next character, and `step` only moves to the next position. These movement functions are necessary to decide what input should be processed when the state machine transitions to the next state. - `getState`: Determines the next state based on the character at the current buffer position. This function evaluates the class (type of character) of the current character and uses a state transition table to decide the next state. This process is central to how the state machine interprets the JSON structure. - `numeric/1`, `string/2`, `word/1`: These functions parse numbers, strings, and specified word tokens, respectively. During parsing, the state machine transitions to the next state based on the current character's type and context, which is essential for accurately interpreting the structure and meaning of JSON data. - `skip`, `skipAny/1`: Functions for skipping characters that meet certain conditions, such as moving the buffer index until a specific character or set of tokens is encountered. These functions are mainly used to manage the current state of the state machine while parsing structural elements (e.g., the end of an object or array). These functions are used to closely interact with the state machine to recognize and interpret the various data types and structures within the JSON string. The current state of the state machine changes based on each character or string the buffer processes, dynamically controlling the parsing process. ### Unescape These functions are designed to process JSON strings, specifically by managing internal buffer interactions and unescaping characters as per JSON standards. This involves translating escape sequences like `\uXXXX` for unicode characters, as well as simpler escapes like `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, and `\t`. Here's some key functions for this file: - `Unescape/2`: This is the primary function that takes an input byte slice (representing a JSON string with escape sequences) and an output byte slice to write the unescaped version of the input. It processes each escape sequence encountered in the input slice and translates it into the corresponding UTF-8 character in the output slice. - `Unquote/2`: This function is designed to remove surrounding quotes from a JSON string and unescape its contents. It's useful for processing JSON string values to their literal representations. ### Node When a JSON string is decoded, the package converts the data into a Node type. ```go type Node struct { prev *Node // prev is the parent node of the current node. next map[string]*Node // next is the child nodes of the current node. key *string // key holds the key of the current node in the parent node. data []byte // byte slice of JSON data value interface{} // value holds the value of the current node. nodeType ValueType // NodeType holds the type of the current node. (Object, Array, String, Number, Boolean, Null) index *int // index holds the index of the current node in the parent array node. borders [2]int // borders stores the start and end index of the current node in the data. modified bool // modified indicates the current node is changed or not. } ``` This node type allows you to fetch and manipulate the specific values from JSON. For example, you can use the `GetKey/1` function to retrieve the value stored at a specific key, and you can use `Delete` to remove the node. By doing so, enabling you to process JSON data. --- <a name="footnote_1">1</a>: The Eisel-Lemire algorithm provides a fast way to parse floating-point numbers from strings. The core idea of this algorithm is to minimize potential errors during the conversion process from strings to numbers, while processing the conversion as quickly as possible. Eisel-Lemire is particularly useful when dealing with large amounts of numerical data, providing much faster and more accurate results than traditional parsing methods. <a name="footnote_2">2</a>: The Ryu algorithm focuses on converting floating-point numbers to strings. Ryu generally converts floating-point numbers to the shortest possible string representation accurately, with excellent performance and precision. A key advantage of the Ryu algorithm is that the converted string maintains the minimum length while precisely representing the original number. This helps save storage space and reduces data transmission times over networks.
- Loading branch information