json::parse throws std::invalid_argument when processing string generated by json::dump()

[chenhongbao]

Bug Report

• What is the issue you have?
  I generate a JSON string with json::dump(), and pass to another thread. The thread calls json::parse() to get an object from the JSON string and it throws std::invalid_argument exception.

The following codes throws the exception:

void unexpect(typename lexer::token_type t) const
        {
            if (t == last_token)
            {
                std::string error_msg = "parse error - unexpected ";
                error_msg += (last_token == lexer::token_type::parse_error ? ("'" +  m_lexer.get_token_string() +
                              "'") :
                              lexer::token_type_name(last_token));
                JSON_THROW(std::invalid_argument(error_msg));
            }
        }

The JSON string:

{"Data":[{"Bin":205.0,"DataName":"VolumeDistributionParameters","Expandor":0.100000001490116,"Factor":1.16281127035618,"InstrumentID":"MA805","M":0.932498933747411,"Mse":0.00820842096151519,"Sigma":0.837557822465897,"Theta":0.0,"UpdateDay":"20171208","UpdateTime":"14:51:08"}],"ErrorCode":0,"ErrorMessage":"正确","Receiver":"ANY_RECEIVER","Result":"OK","Sender":"VOLDIST","ServiceName":"VOLDIST"}

• Please describe the steps to reproduce the issue. Can you provide a small but working code example?

Try the following code and exception will show up:

#include "json.hpp"
int main() {
	nlohmann::json jsonObj;
	std::string jstr = R"({"Data":[{"Bin":1790.0,"DataName":"VolumeDistributionParameters","Expandor":0.100000001490116,"Factor":0.831193974614143,"InstrumentID":"rb1805","M":0.640040159225464,"Mse":0.00439472794626309,"Sigma":0.747419960796833,"Theta":0.0,"UpdateDay":"20171208","UpdateTime":"14:55:08"}],"ErrorCode":0,"ErrorMessage":"正确","Receiver":"ANY_RECEIVER","Result":"OK","Sender":"VOLDIST","ServiceName":"VOLDIST"})";
	try
	{
		jsonObj = nlohmann::json::parse(jstr);
	}
	catch (std::invalid_argument & invalid)
	{
		std::cout << invalid.what() << std::endl;
	}
    return 0;
}

• What is the expected behavior?

Parse the JSON string and return an object.

• And what is the actual behavior instead?

Throwing an exception, the std::invalid_argument

• Which compiler and operating system are you using? Is it a supported compiler?

VS 2017 Community

• Did you use a released version of the library or the version from the develop branch?

Release 2.1.1

• If you experience a compilation error: can you compile and run the unit tests?

No.

Feature Request

• Describe the feature in as much detail as possible.

Above is enough maybe.

• Include sample usage where appropriate.

See above.

[nlohmann]

• What is the exact exception string? What was the unexpected token?
• Could you try the develop branch which should provide better error messages.
• I tried your example with GCC and got no error.

[chenhongbao]

Thanks for quick response.

• What is the exact exception string? What was the unexpected token?

exception string is:

"parse error - unexpected '"'"

unexpected token is:

parse_error (14)

• Could you try the develop branch which should provide better error messages.

I have tried it and still see the exception. The information looks like below:

"syntax error - invalid string: ill-formed UTF-8 byte; last read: '"正'"

Please note that I use '\uXXXX' notation for Chinese characters in source code.

[chenhongbao]

The Chinese version of Visual Studio encodes the source files in gb2312 (like Excel and others), while the GCC default encoding is UTF-8. I think that is the cause of the exception.

I tried "Save as..." to save source file as Unicode (Code Page 1200), Unicode (UTF-8 With Signature， Code Page 65001), the problem still exits.

If so, it is really a sticky problem. >_<

[nlohmann]

The library only supports UTF-8 encoded strings. If a string with a different encoding is stored, the resulting serialization may be incompliant JSON. We discuss in #838 whether we shall throw an exception in case non-UTF-8 encodings are detected during serialization.

[nlohmann]

Merged #870: Trying to serialize a non-UTF-8 encoded string now throws an exception.