json start/end position implementation #4517

sushshring · 2024-11-25T23:29:05Z

Abstract

Referring to discussion: 4455, this pull request introduces the implementation to retrieve the start and end positions of nested objects within the JSON during parsing.

Motivation

We have a service implementation with JSON schema where a field within the nested objects contains the hash value for that object. The service verifies the hash value of each of the nested objects before operating on the rest of the data sent.

For example, consider the following JSON:

{
    "name": "foo",
    "data":
    {
        "type": "typeA",
        "value": 1,
        "details": {
            "nested_type": "nested_typeA",
            "nested_value": 2
        }
    },
    "data_hash": "hashA"
}

Here, data_hash contains the hash of the object "details". In order to verify the data hash, we need to be able to retrieve the exact string that parsed out "details" including the spaces and newlines. Currently there is no way to achieve this using nlohmann/json parser.

Changes proposed

Add two fields to basic_json: size_t start_position and size_t end_position.
Add a reference to the lexer in json_sax_parser to retrieve the current position in the input string.
Whenever a BasicJsonType is created by the parser, calculate the start and end positions for that object from the original string and store those values.

Memory considerations

We considered storing substrings in the output JSON objects and sub-objects directly as well, however, considering the memory footprint increase that it would create, we opted for the option where only two size_t fields are stored per basic_json created.

Validation

We have added tests to the class_parser test suite that cover the following cases:

Array inside an object
Objects inside arrays
Doubly nested objects
String fields
Integer and float fields
Float values with insignificant digits
Boolean fields
Null fields

Since the change affects the sax_parser, for each of these test cases we validate scenarios where no callback is passed, a callback is passed that accepts all fields, and a callback is passed that filters specific fields.

Pull request checklist

Read the Contribution Guidelines for detailed information.

Changes are described in the pull request, or an existing issue is referenced.
The test suite compiles and runs without error.
Code coverage is 100%. Test cases can be added by editing the test suite.
The source code is amalgamated; that is, after making changes to the sources in the include/nlohmann directory, run make amalgamate to create the single-header files single_include/nlohmann/json.hpp and single_include/nlohmann/json_fwd.hpp. The whole process is described here.

Please don't

The C++11 support varies between different compilers and versions. Please note the list of supported compilers. Some compilers like GCC 4.7 (and earlier), Clang 3.3 (and earlier), or Microsoft Visual Studio 13.0 and earlier are known not to work due to missing or incomplete C++11 support. Please refrain from proposing changes that work around these compiler's limitations with #ifdefs or other means.
Specifically, I am aware of compilation problems with Microsoft Visual Studio (there even is an issue label for this kind of bug). I understand that even in 2016, complete C++11 support isn't there yet. But please also understand that I do not want to drop features or uglify the code just to make Microsoft's sub-standard compiler happy. The past has shown that there are ways to express the functionality such that the code compiles with the most recent MSVC - unfortunately, this is not the main objective of the project.
Please refrain from proposing changes that would break JSON conformance. If you propose a conformant extension of JSON to be supported by the library, please motivate this extension.
Please do not open pull requests that address multiple issues.

… parse

…elop

coveralls · 2024-11-26T05:56:53Z

coverage: 99.466% (-0.2%) from 99.647%
when pulling b96a5d1 on sushshring:develop
into ee32bfc on nlohmann:develop.

nlohmann · 2024-11-26T06:59:17Z

Thanks for the effort!

However, adding two size_t members is a lot of overhead. When we introduced diagnostics, we hid a single pointer behind a preprocessor macro to avoid every single client to suffer from the overhead. Issues like #4514 show that the memory efficiency is already quite bad.

I am hesitant how to continue here.

gregmarr · 2024-11-26T14:30:02Z

I wonder if this is something that could be done with the data in the custom base class, so that only those that want to opt in to this behavior could enable it. That does make it a custom class, and not nlohmann::json or nlohmann::ordered_json.

sushshring · 2024-11-26T23:38:48Z

Taking that advice, I'm gonna add a new class like below to use as a json custom base class

class json_base_class_with_start_end_markers {
    size_t start_position = std::string::npos;
    size_t end_position = std::string::npos;

public:
    size_t get_start_position() const noexcept
    {
        return start_position;
    }

    size_t get_end_position() const noexcept
    {
        return end_position;
    }

    void set_start_position(size_t start) noexcept
    {
        start_position = start;
    }

    void set_end_position(size_t end) noexcept
    {
        end_position = end;
    }
};

We will use if (std::is_base_of<json_base_class_with_start_end_markers, BasicJsonType>){} whenever the start_position and end_position setters are called within json_sax.hpp.

include/nlohmann/detail/input/json_sax.hpp

tests/src/unit-disabled_exceptions.cpp

tests/src/unit-deserialization.cpp

include/nlohmann/detail/json_base_class_with_start_end_markers.hpp

nlohmann

Some comments from my side. I am honestly not 100% sold to the use case this PR is solving. As I stated in #4455 (comment), I think we preserve every relevant part of the raw input except whitespace. Did I miss anything?

include/nlohmann/detail/input/binary_reader.hpp

include/nlohmann/detail/input/json_sax.hpp

…s PR comments and fix CI failures

sushshring · 2024-12-06T22:42:53Z

Some comments from my side. I am honestly not 100% sold to the use case this PR is solving. As I stated in #4455 (comment), I think we preserve every relevant part of the raw input except whitespace. Did I miss anything?

Whitespace is key to our use case - my colleague will provide a more detailed explanation in #4455 soon.

include/nlohmann/detail/json_base_class_with_start_end_markers.hpp

include/nlohmann/detail/input/json_sax.hpp

tests/src/unit-deserialization.cpp

raphgianotti · 2024-12-09T18:19:21Z

Some comments from my side. I am honestly not 100% sold to the use case this PR is solving. As I stated in #4455 (comment), I think we preserve every relevant part of the raw input except whitespace. Did I miss anything?

The main issue we are trying to solve is when needing to sign the string corresponding to a nested object in a json, example:
{
"object_to_sign": {
"int_value": 23,
"float_value": 1.234,
"string_value": "hello"
},
"signature": ""
}
In this scenario, for us to calculate the signature using a public key to validate the contents of "object_to_sign" are correct depends fully on every thing about the string representing "object_to_sign" in the original payload (including spaces/line breaks). When defining a protocol for this, it's not possible (since the sender could be employing any json library) to simply use the nlohmann::json object generated from parsing to generate the string for "object_to_sign" again, as it could differ slightly, even if its contents are equivalent.

Even thought this is a somewhat niche scenario, it's mostly exposing information nlohmann has access to and, as an optional feature, I think it can add value to the library, as doing it outside it would require some form of parsing the contents of the string again, doubling the complexity and involving using two separate json parsers where this change allows the use of a single one.

gregmarr · 2024-12-09T23:30:57Z

I think it's going to be difficult to use the macro for handle_start_end_pos_for_json_value due to the unused parameters when the base class is used. It may make sense to write those two functions out individually.

I had another thought about these:

#define HANDLE_START_END_POS_DEFINITION(__handlecase, __statement,...)   \
    template <class Q = BasicJsonType>                                                                                                                  \
    typename std::enable_if<std::is_base_of<::nlohmann::detail::json_base_class_with_start_end_markers, Q>::value, void>::type                          \
    handle_start_end_pos_for_##__handlecase(__VA_ARGS__)                                                                                                \
    {                                                                                                                                                   \
        if (m_lexer_ref)                                                                                                                                \
        {                                                                                                                                               \
            __statement                                                                                                                                 \
        }                                                                                                                                               \
    }                                                                                                                                                   \
    \
    template <class Q = BasicJsonType>                                                                                                                  \
    typename std::enable_if<!std::is_base_of<::nlohmann::detail::json_base_class_with_start_end_markers, Q>::value, void>::type                         \
    handle_start_end_pos_for_##__handlecase(__VA_ARGS__){}

    HANDLE_START_END_POS_DEFINITION(start_object,
    {
        ref_stack.back()->start_position = m_lexer_ref->get_position() - 1;
    })

What if we created two set functions, and put them in the base class, with no-ops in the main class if the base class isn't used? Then the function can always be called, and you don't need to protect the one-line callers, but you would want to protect the more involved ones because of the extra code involved.

struct json_base_class_with_start_end_markers {
    size_t start_position = ...;
    size_t end_position = ...;
    // functions for storing the start and end positions when this is the basic_json base class.
    void set_start_position(size_t i) { start_position = i; }
    void set_end_position(size_t i) { end_position = i; }
}

class basic_json {
    // no-op functions for storing the start and end positions when the base class isn't used.
    typename std::enable_if<!std::is_base_of<detail::json_base_class_with_start_end_markers, decltype(*this)>::value, void>::type
    void set_start_position(size_t){}
    typename std::enable_if<!std::is_base_of<detail::json_base_class_with_start_end_markers, decltype(*this)>::value, void>::type
    void set_end_position(size_t){}

};

    void handle_start_end_pos_for_start_object()
    {
        if (m_lexer_ref != nullptr)
        {
            ref_stack.back()->set_start_position(m_lexer_ref->get_position() - 1);
        }
    }

sushshring · 2024-12-10T18:11:31Z

@gregmarr I'd considered that, but decided against it to minimize changes to basic_json. Since it's a no-op I'm guessing the compiler might just optimize those away. I will update with that change

gregmarr · 2024-12-10T19:22:47Z

Yeah, since this is header-only, the call itself should definitely get eliminated, and then any computations that are just related to that will hopefully be eliminated too.

nlohmann · 2024-12-10T19:54:33Z

include/nlohmann/json_fwd.hpp

@@ -70,6 +71,20 @@ struct ordered_map;
 /// @sa https://json.nlohmann.me/api/ordered_json/
 using ordered_json = basic_json<nlohmann::ordered_map>;

+/// @brief a minimal specialization that uses the base class json_base_class_with_start_end_markers
+using json_with_start_end_markers = nlohmann::basic_json <


I hope we can find a better name for this feature...

Open to discussion for names :). I'm terrible at coming up with good names

If I had a better one, I'd have dropped it here. :-)

For the base class, how about json_base_with_positions? or json_base_positional?
Then this could be json_with_positions, or json_positional.

Implemented the macros, so base class is gone. basic_json now exposes two property getters: start_pos() and end_pos(), if the DIAGNOSTIC_POSITIONS macro is set. Otherwise there is no additional memory footprint.

For this PR, changing the name to JSON_DIAGNOSTIC_POSITION would be sufficient. As possible next step, we should also pick up the positions to provide better error message since we now know the positions of each part of the JSON.

include/nlohmann/detail/input/json_sax.hpp

sushshring · 2024-12-11T00:56:53Z

@gregmarr After some prototyping I have realized that to correctly enable SFINAE in the derived class, we need to add the following:

    template<class T = CustomBaseClass>
    typename std::enable_if<!std::is_same<T, detail::json_base_class_with_start_end_markers>::value, void>::type
    set_start_position(size_t){}
    template<class T = CustomBaseClass>
    typename std::enable_if<!std::is_same<T, detail::json_base_class_with_start_end_markers>::value, void>::type
    set_end_position(size_t){}
 
    template<class T = CustomBaseClass>
    typename std::enable_if<std::is_same<T, detail::json_base_class_with_start_end_markers>::value, void>::type
    set_start_position(size_t pos)
    {
        reinterpret_cast<detail::json_base_class_with_start_end_markers*>(this)->set_start_position(pos);
    }
    template<class T = CustomBaseClass>
    typename std::enable_if<std::is_same<T, detail::json_base_class_with_start_end_markers>::value, void>::type
    set_end_position(size_t pos)
    {
        reinterpret_cast<detail::json_base_class_with_start_end_markers*>(this)->set_end_position(pos);
    }

The methods need the alternate implementation in order to work with SFINAE. The already implemented versions in the base class do not count for the SFINAE pattern's error ignore. This results in a rather ugly reinterpret_cast<> within basic_json that will certainly violate https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#c153-prefer-virtual-function-to-casting. The safer alternative is to add virtual methods to both base classes and override those in basic_json, which will likely affect performance slightly.

The macros that I've added in the recent commit don't work with C++11, however, that can be fixed by adding a new version of the macros that takes no arguments and removes the varargs, which makes the CI checks happy. I will implement that in my next commit instead.

nlohmann · 2024-12-11T05:41:03Z

A very wild idea: why not adding another macro like DIAGNOSTIC_POSITIONS and implement this without inheritance?

gregmarr · 2024-12-11T16:53:06Z

The methods need the alternate implementation in order to work with SFINAE. The already implemented versions in the base class do not count for the SFINAE pattern's error ignore.

There is no SFINAE for the simple functions with this method.

    void handle_start_end_pos_for_end_array()
    {
        if (m_lexer_ref != nullptr)
        {
            ref_stack.back()->set_end_position(m_lexer_ref->get_position());
        }
    }

    bool end_array()
    {
        bool keep = true;

        if (ref_stack.back())
        {
            keep = callback(static_cast<int>(ref_stack.size()) - 1, parse_event_t::array_end, *ref_stack.back());
            if (keep)
            {
                handle_start_end_pos_for_end_array();

Since object and array do the same thing, that could also be simplified to reduce the number of functions:

    void record_start_pos()
    {
        if (m_lexer_ref != nullptr)
        {
            ref_stack.back()->set_start_position(m_lexer_ref->get_position() - 1);
        }
    }
    void record_end_pos()
    {
        if (m_lexer_ref != nullptr)
        {
            ref_stack.back()->set_end_position(m_lexer_ref->get_position());
        }
    }

why not adding another macro like DIAGNOSTIC_POSITIONS and implement this without inheritance?

That's certainly a possibility. I thought the base class might be cleaner, but it changes the types, so maybe it isn't. We'd need to encode this define into the intermediate namespace selection so that we don't link against something with the wrong definition, as we do for JSON_DIAGNOSTICS.

https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/abi_macros.hpp#L33-L37

sushshring · 2024-12-11T18:08:51Z

A very wild idea: why not adding another macro like DIAGNOSTIC_POSITIONS and implement this without inheritance?

This seems feasible, thanks for the note on the intermediate namespace @gregmarr. We are prototyping this locally as an option and will post the next iteration based on our findings.

Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>

sushshring · 2024-12-12T00:44:24Z

@nlohmann pushed a new commit that implements the diagnostic style macro. I personally think this looks cleaner as well, PTAL.

gregmarr · 2024-12-12T00:45:55Z

include/nlohmann/detail/abi_macros.hpp

@@ -26,6 +26,10 @@
    #define JSON_DIAGNOSTICS 0
 #endif

+#ifndef DIAGNOSTIC_POSITIONS


Should this define start with JSON_ like the others?

Yes, and we should also have a brief first version of a documentation ready like https://raw.githubusercontent.com/nlohmann/json/develop/docs/mkdocs/docs/api/macros/json_diagnostics.md - I can take care about the rest.

gregmarr · 2024-12-12T02:30:32Z

include/nlohmann/detail/input/json_sax.hpp


 #include <nlohmann/detail/exceptions.hpp>
 #include <nlohmann/detail/macro_scope.hpp>
 #include <nlohmann/detail/string_concat.hpp>
-
+#include <nlohmann/detail/input/lexer.hpp>


Lost a blank line here.

gregmarr · 2024-12-12T02:35:06Z

include/nlohmann/json.hpp

+#if DIAGNOSTIC_POSITIONS
+        start_position = val.start_position;
+        end_position = val.end_position;
+#endif


blank line after?

gregmarr · 2024-12-12T02:35:13Z

include/nlohmann/json.hpp

@@ -1204,15 +1209,24 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
            default:
                break;
        }
+#if DIAGNOSTIC_POSITIONS


blank line before?

gregmarr · 2024-12-12T02:43:03Z

include/nlohmann/json.hpp

+    }
+  private:
+    /// the start position of the value
+    size_t start_position = std::string::npos;


I think these data members should be next to the other data members, down around 4265, probably after the existing ones so as to not unnecessarily change the order of data members. We also need to make sure that the members are initialized in the constructors in same order, as compilers will warn if they're not in the same order.

Made a judgement call on the location, thanks for providing the appropriate one. I’ll make sure to run the ci tools locally for the compiler checks

gregmarr · 2024-12-12T02:45:43Z

include/nlohmann/detail/input/input_adapters.hpp

@@ -417,6 +417,9 @@ typename container_input_adapter_factory_impl::container_input_adapter_factory<C
    return container_input_adapter_factory_impl::container_input_adapter_factory<ContainerType>::create(container);
 }

+// specialization for std::string
+using string_input_adapter = decltype(input_adapter(std::declval<std::string>()));


using string_input_adapter_type =?

gregmarr · 2024-12-12T02:45:50Z

tests/src/unit-cbor.cpp

@@ -1631,7 +1631,8 @@ TEST_CASE("CBOR")
                };

                json j;
-                auto cbp = nlohmann::detail::json_sax_dom_callback_parser<json>(j, callback, true);
+                auto ia = nlohmann::detail::input_adapter(input);
+                auto cbp = nlohmann::detail::json_sax_dom_callback_parser<json, decltype(ia)>(j, callback, true);


string_input_adapter_type?

gregmarr · 2024-12-12T02:45:57Z

tests/src/unit-class_parser.cpp

@@ -219,7 +219,8 @@ json parser_helper(const std::string& s)
    CHECK(j_nothrow == j);

    json j_sax;
-    nlohmann::detail::json_sax_dom_parser<json> sdp(j_sax);
+    auto ia = nlohmann::detail::input_adapter(s);
+    nlohmann::detail::json_sax_dom_parser<json, decltype(ia)> sdp(j_sax);


string_input_adapter_type?

gregmarr · 2024-12-12T02:46:46Z

tests/src/unit-class_parser_diagnostic_positions.cpp

+
+    json j_sax;
+    auto ia = nlohmann::detail::input_adapter(s);
+    nlohmann::detail::json_sax_dom_parser<json, decltype(ia)> sdp(j_sax);


string_input_adapter_type?

gregmarr · 2024-12-12T02:48:49Z

tests/src/unit-regression2.cpp

 {
  public:
    explicit sax_no_exception(json& j)
-        : nlohmann::detail::json_sax_dom_parser<json>(j, false)
+        : nlohmann::detail::json_sax_dom_parser<json, decltype(nlohmann::detail::input_adapter(""))>(j, false)


string_input_adapter_type?

sushshring · 2024-12-12T06:02:41Z

The ci_clang_tidy job keeps complaining about the parser callback being passed by value in json_sax, but that’s not really any code we’re modifying in this PR. What’s the suggestion for that?

nlohmann · 2024-12-12T06:55:01Z

The ci_clang_tidy job keeps complaining about the parser callback being passed by value in json_sax, but that’s not really any code we’re modifying in this PR. What’s the suggestion for that?

I think everywhere else we pass it as non-const parameter, which seems to be OK for Clang-Tidy.

sushshring and others added 18 commits October 29, 2024 15:24

Add implementation to retrieve start and end positions of json during…

02f5bca

… parse

Add more unit tests and add start/stop parsing for arrays

311861f

Merge branch 'nlohmann:develop' into develop

226d79b

Add raw value for all types

b3f6499

Merge branch 'develop' of https://github.com/sushshring/json into dev…

ab744aa

…elop

Add more tests and fix compiler warning

d321cdb

Amalgamate

64ad6ce

Fix CLang GCC warnings

e820747

Fix error in build

3629ceb

Style using astyle 3.1

b42036b

Fix whitespace changes

2575678

revert

9de6ed1

more whitespace reverts

3bbca5e

Address PR comments

fa32e81

Fix failing issues

b806d44

More whitespace reverts

7d662ec

Address remaining PR comments

3625875

Address comments

9359441

sushshring requested a review from nlohmann as a code owner November 25, 2024 23:29

github-actions bot added L tests labels Nov 25, 2024

Merge remote-tracking branch 'nlohmann/develop' into develop

79e6513

nlohmann added the state: please discuss please discuss the issue or vote for your favorite option label Nov 26, 2024

sushshring commented Nov 27, 2024

View reviewed changes

include/nlohmann/detail/input/json_sax.hpp Outdated Show resolved Hide resolved

Switch to using custom base class instead of default basic_json

a31d8b8

gregmarr reviewed Dec 4, 2024

View reviewed changes

tests/src/unit-disabled_exceptions.cpp Outdated Show resolved Hide resolved

gregmarr reviewed Dec 4, 2024

View reviewed changes

tests/src/unit-deserialization.cpp Outdated Show resolved Hide resolved

gregmarr reviewed Dec 4, 2024

View reviewed changes

include/nlohmann/detail/json_base_class_with_start_end_markers.hpp Outdated Show resolved Hide resolved

gregmarr reviewed Dec 4, 2024

View reviewed changes

include/nlohmann/detail/json_base_class_with_start_end_markers.hpp Outdated Show resolved Hide resolved

nlohmann requested changes Dec 5, 2024

View reviewed changes

include/nlohmann/detail/input/binary_reader.hpp Show resolved Hide resolved

include/nlohmann/detail/input/json_sax.hpp Outdated Show resolved Hide resolved

Adding a basic using for a json using the new base class. Also addres…

1d70d2b

…s PR comments and fix CI failures

Address decltype comments

814f367

gregmarr reviewed Dec 7, 2024

View reviewed changes

tests/src/unit-deserialization.cpp Outdated Show resolved Hide resolved

nlohmann reviewed Dec 10, 2024

View reviewed changes

nlohmann mentioned this pull request Dec 10, 2024

Allow ordered_json in NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE macro #4528

Open

Diagnostic positions macro (#4)

4986e99

Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>

Fix missed include deletion

b96a5d1

gregmarr reviewed Dec 12, 2024

View reviewed changes

json start/end position implementation #4517

Are you sure you want to change the base?

json start/end position implementation #4517

Conversation

sushshring commented Nov 25, 2024

Abstract

Motivation

Changes proposed

Memory considerations

Validation

Pull request checklist

Please don't

coveralls commented Nov 26, 2024 • edited Loading

nlohmann commented Nov 26, 2024

gregmarr commented Nov 26, 2024

sushshring commented Nov 26, 2024 • edited Loading

nlohmann left a comment

Choose a reason for hiding this comment

sushshring commented Dec 6, 2024

raphgianotti commented Dec 9, 2024

gregmarr commented Dec 9, 2024

sushshring commented Dec 10, 2024

gregmarr commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sushshring Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sushshring commented Dec 11, 2024

nlohmann commented Dec 11, 2024

gregmarr commented Dec 11, 2024

sushshring commented Dec 11, 2024

sushshring commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sushshring commented Dec 12, 2024

nlohmann commented Dec 12, 2024

coveralls commented Nov 26, 2024 •

edited

Loading

sushshring commented Nov 26, 2024 •

edited

Loading

sushshring Dec 11, 2024 •

edited

Loading