serialize std::variant<...>

[MattiasEppler]

Hi.
I try to serialize std::variant<double,int> to json.
I look for an easy and comfortable way. Is there any solution available?

Or any idea how to do this.

[nlohmann]

This section from the README should help you.

[nlohmann]

@MattiasEppler Do you need further assistance?

[MattiasEppler]

Hi.
I already read this section.

I do it in this way

string value = "";

if(const auto pdouble = std::get_if<double>(&data.Value))
    value = std::to_string(*pdouble);
else if(const auto pint= std::get_if<int>(&data.Value))
    value = std::to_string(*pint);

I thought maybe there is an easier solution.

But if not you can close this issue.

Thanks

[nlohmann]

I'm afraid not - at least I am not aware of a generic way to collect all the possible values from a variant.

[theodelrieu]

You could do something like:

namespace nlohmann {
template <typename ...Args>
struct adl_serializer<std::variant<Args...>> {
  static void to_json(json& j, std::variant<Args...> const& v) {
    std::visit([&](auto&& value) {
      j = std::forward<decltype(value)>(value);
    }, v);
  }
};
}

I don't think it's possible to provide a generic from_json here. Even if we serialize the index, it will be known at runtime, so switching on it would require to write N case, N being sizeof...(Args) - 1...

What I would do is setting the index in the json only when serializing an enclosing type:

struct Example {
  std::variant<int, std::string, float> var;
};

void to_json(json& j, Example const& e) {
  j = {{"index", e.var.index()}, {"variant", e.var}};
}

void from_json(json const&j, Example& e) {
  auto const index = j.at("index").get<int>();
  switch (index) {
    case 0:
      e.var = j.get<int>(); break;
    case 1:
      e.var = j.get<std::string>(); break;
    case 2:
      e.var = j.get<float>(); break;
    default: throw std::runtime_error{""};
  }
}

[theodelrieu]

On a second thought, there might be a way to make it generic, I'll try it out and post the results.

[theodelrieu]

Here is a complete example with the generic variant implementation.

Note that it is far from being efficient, it might be improved with something similar to for_each_args.

#include <iostream>
#include <stdexcept>
#include <string>
#include <variant>

#include <nlohmann/json.hpp>

using json = nlohmann::json;

namespace detail
{
template <std::size_t N>
struct variant_switch
{
  template <typename Variant>
  void operator()(int index, json const &value, Variant &v) const
  {
    if (index == N)
      v = value.get<std::variant_alternative_t<N, Variant>>();
    else
      variant_switch<N - 1>{}(index, value, v);
  }
};

template <>
struct variant_switch<0>
{
  template <typename Variant>
  void operator()(int index, json const &value, Variant &v) const
  {
    if (index == 0)
      v = value.get<std::variant_alternative_t<0, Variant>>();
    else
    {
      throw std::runtime_error(
          "while converting json to variant: invalid index");
    }
  }
};
}

namespace nlohmann
{
template <typename ...Args>
struct adl_serializer<std::variant<Args...>>
{
  static void to_json(json& j, std::variant<Args...> const& v)
  {
    std::visit([&](auto&& value) {
      j["index"] = v.index();
      j["value"] = std::forward<decltype(value)>(value);
    }, v);
  }

  static void from_json(json const& j, std::variant<Args...>& v)
  {
    auto const index = j.at("index").get<int>();
    ::detail::variant_switch<sizeof...(Args) - 1>{}(index, j.at("value"), v);
  }
};
}

struct A
{
  int val;
};

bool operator==(A const& lhs, A const& rhs) noexcept
{
  return lhs.val == rhs.val;
}

bool operator!=(A const& lhs, A const& rhs) noexcept
{
  return !(lhs == rhs);
}

void to_json(json& j, A a)
{
  j["val"] = a.val;
}

void from_json(json const& j, A& a)
{
  a.val = j.at("val").get<int>();
}

int main(int argc, char const *argv[])
{
  std::variant<int, std::string, float, A> v(A{42});

  nlohmann::json j(v);

  auto v2 = j.get<decltype(v)>();
  assert(v == v2);

  std::cout << j.dump(2) << std::endl;
  v = std::string("Hello");
  j = v;
  std::cout << j.dump(2) << std::endl;

  // invalidate the index
  j["index"] = 42;
  try
  {
    auto _ = j.get<decltype(v)>();
  }
  catch (std::runtime_error const& e)
  {
    std::cout << "Caught exception: " << e.what() << std::endl;
  }
}

[MattiasEppler]

Oh.... Thanks a lot for spending your time. I will try it.

[nlohmann]

@MattiasEppler Did you try?

[MattiasEppler]

Hi. Oh sorry.
Yes It works. Thanks to theodelrieu.

Will this be added to nlohmann\json?

[theodelrieu]

Will this be added to nlohmann\json?

I guess we could add it, I'd like users to be able to do the following:

std::variant<int, float, MyType> v(2);

json j = v;
auto i = j.get<int>();
auto v2 = j.get<decltype(v)>();

For this to work, we need to add a library-reserved field in the variant's alternative serialization:

template <typename ...Args>
void to_json(json& j, std::variant<Args...> const& v) {
  std::visit([&](auto const& a) { 
                      j = a;
                    }, v);
  j["__nlohmann_variant_index"] = v.index();
}

template <typename ...Args>
void from_json(json const& j, std::variant<Args...>& v) {
  auto const index = j.at("__nlohmann_variant_index").get<int>();
  // recursive ugliness to get the correct index...
}

[MattiasEppler]

In my case its not necessary to serialize the index.
So maybe it should be possible to set serializeIndex = false.

[theodelrieu]

After giving it a bit of thought, I think we should stay away from supporting std::variant natively. There is no correct solution to fit every use-case, i.e. the generic implementation I posted is very inefficient, it instantiates a lot of templates, it's more a hack than anything else.

So maybe it should be possible to set serializeIndex = false.

I'd like to avoid such compile-time toggle to enable/disable some specific conversion as well.

[nlohmann]

I agree with @theodelrieu. In a lot of cases, a variant is not the sole variable to be serialized, so type information may be derived implicitly from other members.

@MattiasEppler Is it OK to close the issue?

[MattiasEppler]

Hi. Yes its ok.

[ibkevg]

Edit: answered below

QUESTION: @theodelrieu @nlohmann after writing the to_json() and from_json() routines to serialize a std::variant I've found that the json constructor can't find my custom to_json() function. I wondered if the problem was that the to/from functions had to be in the std namespace but they don't compile if I do. I noticed above that you stuck the std::variant inside a struct but that isn't possible in my use case.

What do you recommend as the cleanest way to use std::variant? (I should add that my ultimate goal is to be able to json-ify a map<string, variant<uint64_t, double, string>>)
(edit: I'm realizing that an adl_serializer may be involved)

Here is a small example (which I used on the latest Visual Studio 15.9)

#include <nlohmann/json.hpp>
#include <variant>

using json = nlohmann::json;

typedef std::variant<int64_t, std::string>  TestVariant;

void to_json(json& j, TestVariant const& v)
{
    if (auto pInt64 = std::get_if<int64_t>(&v)) {
        j = json{ {"type", 0}, {"int64_t", *pInt64 } };
    else if (auto pStdStr = std::get_if<std::string>(&v)) {
        j = json{ {"type", 1}, {"string", *pStdStr} };
    } else {
        throw std::bad_variant_access();
    }
}

void from_json(json const& j, TestVariant& v)
{
    auto const type = j.at("type").get<int>();
    switch (type) {
    case 0:
        v = TestVariant(j.at("int64_t").get<uint64_t>()); break;
    case 1:
        v = TestVariant(j.at("string").get<std::string>()); break;
    default:
        throw std::runtime_error{ "Unexpected FT:Variant" };
    }
}

void test()
{
    auto v1 = TestVariant(100);

#if 0
    // Gives compile error C2440 below
    json j = v1;
#else
    // This compiles/works
    json j;
    to_json(j1, vInt);
#endif

    std::cout << j << std::endl;
    TestVariant v2 = j; // Gives compile error C2440 below
    cout << std::get<uint64_t>(v2) << endl;
}

error C2440: 'initializing': cannot convert from 'TestVariant' to 'nlohmann::basic_jsonstd::map,std::vector,std::string,bool,int64_t,uint64_t,double,std::allocator,nlohmann::adl_serializer'
note: No constructor could take the source type, or constructor overload resolution was ambiguous

error C2440: 'initializing': cannot convert from 'json' to 'std::variant<int64_t,std::string>'
note: No constructor could take the source type, or constructor overload resolution was ambiguous

ANSWER: here is the glue needed:

namespace nlohmann {

template <>
struct adl_serializer<TestVariant> {
    static void to_json(json& j, const TestVariant& value) {
        ::to_json(j, value);
    }

    static void from_json(const json& j, TestVariant& value) {
        ::from_json(j, value);
    }
};

}

[theodelrieu]

I would advise putting the conversion code inside the specialization, not calling your free function in the global namespace. Otherwise, it seems correct.

[zhongcy]

@nlohmann @theodelrieu Hi, nlohmann has not support C++17 Variant? Have any Variant examples?

[jeffreyscottgraham]

The alternative 'cereal" library has support for boost variant by adding a separate includes file for those that need it.

[jeffreyscottgraham]

Any update on supporting std::variant?
What about boost variant via a separate #include file?

[nlohmann]

Well, there is the code in #1261 (comment). Apart of this, I would be happy to see a PR!

[chakpongchung]

Also want to see a variant example here

[beached]

just a comment. JSON can have odd ways of expressing what type will be in a member. Take something similar to geojson where we have a type field that tells us what the data field is

[{
  "type": "point",
  "data": [1.0, 2.0]
},
{
  "type": "poly",
  "data": [1.0,2.0,3.0,4.0]
}]

They are the same in terms of JSON types, but when deserializing they could be in a c++ structure like

struct point { double x; double y; };
struct polygon { std::vector<point> points; };

struct geo {
  geo_type type;
  std::variant<point, polygon> data;
};

[narnaud]

My take on this: https://www.kdab.com/jsonify-with-nlohmann-json/

I needed a way to serialize/deserialize variant without saving the type.
This requires the types to be "exclusive".

// Try to set the value of type T into the variant data
// if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data)
{
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct nlohmann::adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data)
    {
        // Will call j = v automatically for the right type
        std::visit(
            [&j](const auto &v) {
                j = v;
            },
            data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data)
    {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};

It will guess the value based on the data. For example, a struct { std::vector<int>, bool> data; }; is serialize like this:

{
    data: true
}
or
{
    data: [1, 2, 3, 4, 5]
}

[MattiasEppler]

Hi narnaud
Thanks for your example.

Can you give an example how to deserialize.
I have implemented the custom adl_serializer like above.
And try to use it in this way: but this do not work

using valuetype = std::variant<bool, double, int>;

inline void from_json(const json& j, valuetype& value)
{
   value = j.at("value").get<valuetype>();
}

[MattiasEppler]

Sorry just want to ask a question.

[narnaud]

@MattiasEppler

I made a mistake in my previous code, the variant_from_json should have been outside the adl_serializer struct (fixed).
The code is now the one I'm using in production, and it should work with your example.

[MattiasEppler]

@narnaud
I move the function variant_from_json outside of the adl_serializer. But still get an error

Error C2672: 'nlohmann::basic_json<std::map,std::vector,std::string,bool,int64_t,uint64_t,double,std::allocator,nlohmann::adl_serializer,std::vector<uint8_t,std::allocator<_Ty>>>::get': no matching overloaded function found (346)

on line

value = j.at("value").get<valuetype>();

[narnaud]

@MattiasEppler Sorry, my fault, I forget the namespace nlohmann for adl_serializer (that's what happens when copy/pasting part of the code).
Fixed it in the original comment.

[MattiasEppler]

@narnaud
That was not the problem. I added the adl_serializer already to the namespace nlohman. :-(
Still the same error.

[MattiasEppler]

sooooory. Just forgot to include the file where the adl_serializer is defined :-(

[MattiasEppler]

Maybe somone an idea to an alternative of switch case. If i try to deserialize a variant with known an index not included in json

switch (index) {
    case 0:
      e.var = j.get<int>(); break;
    case 1:
      e.var = j.get<std::string>(); break;
    case 2:
      e.var = j.get<float>(); break;
    default: throw std::runtime_error{""};
  }

[mqnc]

@ibkevg @nlohmann

QUESTION: @theodelrieu @nlohmann after writing the to_json() and from_json() routines to serialize a std::variant I've found that the json constructor can't find my custom to_json() function. I wondered if the problem was that the to/from functions had to be in the std namespace but they don't compile if I do. I noticed above that you stuck the std::variant inside a struct but that isn't possible in my use case.

ANSWER: here is the glue needed:

namespace nlohmann {

template <>
struct adl_serializer<TestVariant> {
    static void to_json(json& j, const TestVariant& value) {
        ::to_json(j, value);
    }

    static void from_json(const json& j, TestVariant& value) {
        ::from_json(j, value);
    }
};

}

Why is this necessary? I got really stuck on this until I found this ANSWER... Why can every other type be converted with free to_json functions but it doesn't work with variants?

[gregmarr]

@mqnc covered here: https://github.com/nlohmann/json#how-do-i-convert-third-party-types

[mqnc]

Ah because it's in the std namespace... Thank you!

[lkotsonis]

My take on this: kdab.com/jsonify-with-nlohmann-json

I needed a way to serialize/deserialize variant without saving the type. This requires the types to be "exclusive".

   // ...

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data)
    {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};

This worked partially for me because I've been using NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT for the variant types inside std::variant and on top of that my std::variant has a std::monostate. Also I've been keeping the index within the json when serialization in to_json. Hence, I had to add some extra logic to variant_from_json since this:

data = j.get<T>();

doesn't seem to throw even if T is not corresponding to the data in the json object. The logic is along the lines of using monostate as a fallback whenever the index doesn't match the variant type index, to avoid overwriting data with a default initialized variant type.

[vglavnyy]

I found a little bit better solution for ser/deser of the std::variant<Ts...>.
This is a comination of two ideas:

1. Is it possible to print a variable's type in standard C++?
2. serialize std::variant<...> #1261 (comment)

The first part is converting invariant's type to contexpr string:

namespace utils
{
  namespace details
  {
    template <typename T>
    constexpr auto make_type_name() {
      using namespace std::string_view_literals;
    #ifdef __clang__
      std::string_view name = __PRETTY_FUNCTION__;
      name.remove_prefix("auto utils::details::make_type_name() [T = "sv.size());
      name.remove_suffix("]"sv.size());
    #endif
      return name;
    }
  } // details
  
  /**
  * Convert type T into constexpr string.
  */
  template<typename T>
  constexpr auto type_name_sv = details::make_type_name<T>();
  
  static_assert(type_name_sv<int> == std::string_view{"int"});
} // utils

The second is serialization using pack foldind as proposed narnaud:

namespace nlohmann
{
  namespace
  {
      template <typename T, typename... Ts>
      bool variant_from_json(const nlohmann::json& j, std::variant<Ts...>& data) {
        if (j.at("type").get<std::string_view>() != utils::type_name_sv<T>)
          return false;
        data = j.at("data").get<T>();
        return true;
      }
   } 

  template <typename... Ts>
  struct adl_serializer<std::variant<Ts...>>
  {
    using Variant = std::variant<Ts...>;
    using Json = nlohmann::json;
    
    static void to_json(Json& j, const Variant& data) {
      std::visit(
        [&j](const auto& v) {
          using T = std::decay_t<decltype(v)>;
          j["type"] = utils::type_name_sv<T>;
          j["data"] = v;
        },
        data);
    }
    
    static void from_json(const Json& j, Variant& data) {
      // Call variant_from_json for all types, only one will succeed
      bool found = (variant_from_json<Ts>(j, data) || ...);
      if (!found)
        throw std::bad_variant_access();
    }
  };
} // nlohmann

This approach is linear in time without exceptions during testing variants.