Skip to content

Commit

Permalink
Reject invalid non-null tokens beginning with n (#93)
Browse files Browse the repository at this point in the history
* Add test cases for invalid n tokens

Tokens that begin with `n` should not be treated as `null`.

* Reject non-null tokens beginning with n

In 0f609dd, a regression was introduced
where the parser would read any token that started with n as null.

The `.type()` method returns `json_type::null` simply if the token
begins with `n`, and it does not check whether the token is actually
valid.

Adding an `is_null()` call here ensures this and returns an error if the
token starts with `n` but is not `null`. Calling `.value()` on the
returned value will raise it as an exception if `is_null()` returned an
error.

Currently with simdjson 3.10.1, there is a bug where calling `is_null`
on a document object lacks this behaviour, so currently this patch only
fixes the problem for non-scalar documents.

* Comment out broken tests for now
  • Loading branch information
tobil4sk authored Sep 24, 2024
1 parent 8e18887 commit 4a65dc5
Show file tree
Hide file tree
Showing 6 changed files with 13 additions and 2 deletions.
1 change: 1 addition & 0 deletions jsonexamples/invalid/nil_token.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[nil]
1 change: 1 addition & 0 deletions jsonexamples/invalid/nil_token_scalar.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
nil
1 change: 1 addition & 0 deletions jsonexamples/invalid/nully_token.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[nully]
1 change: 1 addition & 0 deletions jsonexamples/invalid/nully_token_scalar.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
nully
6 changes: 5 additions & 1 deletion spec/compile_spec.lua
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,11 @@ if tonumber(major) >= 5 and tonumber(minor) >= 3 then
end

local invalid_files = {
"bool_trailing.json"
"bool_trailing.json",
"nil_token.json",
-- "nil_token_scalar.json",
"nully_token.json",
-- "nully_token_scalar.json"
}

describe("Make sure invalid files are not accepted", function()
Expand Down
5 changes: 4 additions & 1 deletion src/luasimdjson.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,10 @@ void convert_ondemand_element_to_table(lua_State *L, T& element) {
break;

case ondemand::json_type::null:
lua_pushlightuserdata(L, NULL);
// calling is_null().value() will trigger an exception if the value is invalid
if (element.is_null().value()) {
lua_pushlightuserdata(L, NULL);
}
break;
}
}
Expand Down

0 comments on commit 4a65dc5

Please sign in to comment.