Dump throwing exception

[Duchex]

Hi
If I run the following code, the dump method throws an exception:

#include "json.hpp"

nlohmann::json d;
std::string test = "Test string [1094:!èéçßëêâú_{}_[+!£$#*=`~]_&34;B31-TestTest] - Test";
d["test"] = test;

std::cout << d.dump(4) << std::endl;

I understand that the json library uses UTF8, but why does it not encode what I give it into its correct format please? (or am I expcected to do that?)

[nlohmann]

Yes, the library only supports UTF-8, see the README:

Note the library only supports UTF-8. When you store strings with different encodings in the library, calling dump() may throw an exception unless json::error_handler_t::replace or json::error_handler_t::ignore are used as error handlers.

The error handlers are described here.

The library does not do re-encoding, because this is a very hard thing to do.

[Duchex]

So, just to clarify please, any strings I pass to the json object must be UTF-8 encoded beforehand?

[nlohmann]

Yes.

[jaredgrubb]

The encoding used for string literals ("this") is chosen by the compiler. When your compiler is digesting the bytes of your source files, it's making choices about what those bytes and, equally important, how they will be represented in the sections of your binary. By the time std::string or nlohmann::json see the strings, they have a fixed encoding, and if that's not UTF-8, then you'll see the weirdenss you see.

Many OS's (and their compilers) natively use UTF-8 so all of this just works.

However, I'm guessing you're on Windows, which has a rich history on the topics of codepages, Unicode, etc. I don't know if there's a way to ask MSVC to read your source files as UTF-8, but you could look around (eg, this StackOverflow question). Doing that could cuase you other headaches as this could break how you interact with other non-UTF-8 libraries. You might have some luck using the C++11 u8 string prefix (u8"this"),but I've never used it personally so I'm not sure how that would interact with the fact that your source files would still be interpreted by the compiler in some other code-page.

[nlohmann]

@Duchex Do you need further assistance?

[Duchex]

Thank you for the help all. Encoding to UTF-8 before passing to the JSON object solved the issue.

It might be worth clarifying this in the readme since when you say

Note the library only supports UTF-8. When you store strings with different encodings in the library, calling dump() may throw an exception unless json::error_handler_t::replace or json::error_handler_t::ignore are used as error handlers.

It does not expliciltly say that in order to get dump to work, you must use UTF-8 encoded strings in the JSON object when strings are used.

Anyway, thank you for a great library. It works really well.

[nlohmann]

Well, "only supports UTF-8" and "strings with different encodings" should be clear.