parse json OOM

[ddxxsx]

Description

I have a string, which is about 10MB in memory. However, when I use json::parse(), the memory cost is 1.6GB.
I wonder if it's a bug?

Reproduction steps

A long string with chinese charater encoded with utf-8.
call
json j = json::parse(str);
can trigger this bug.

Expected vs. actual results

lower memory cost.

Minimal code example

No response

Error messages

No response

Compiler and operating system

windows

Library version

branch: develop, commit hash: ee32bfc

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[nlohmann]

Please share the input to check. Please also make sure you compile in release mode.

[ddxxsx]

The input file is too large to upload(github limit 25MB), but the file is 64MB large.
The release mode costs less memory(600MB), while debug mode costs almost 1GB memory.

My code is like this:

ifstream file("C:\\input.txt");
std::stringstream buffer;
buffer << file.rdbuf();

std::string content = buffer.str();
json j = json::parse(content);

Is there a way to upload the input file here?

[nlohmann]

I would assume using xz would make the file small enough.

[ddxxsx]

input.zip

Github doesn't support xz file, I attach a zip file instead.

Memory cost:

[nlohmann]

I see 250 MB for that file. Not sure what your compiler is doing there.

My code for reference:

#include <fstream>
#include <iostream>
#include <nlohmann/json.hpp>

int main() {
	std::ifstream f("/Users/niels/Downloads/input.txt");
	auto x = nlohmann::json::parse(f);

	 do  {
	 	std::cout << '\n' << "Press a key to continue...";
	 } while (std::cin.get() != '\n');
}

g++ -I include -O3 -DNDEBUG=1 -std=c++11 parse.cpp