Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to serialize a generic Map<String, Any> in JVM/JS as a JSON object? #296

Closed
sedovalx opened this issue Dec 6, 2018 · 23 comments
Labels
In next release Committed but not published as artifact yet question

Comments

@sedovalx
Copy link

sedovalx commented Dec 6, 2018

I'm trying to figure out a way to serialize an arbitrary Map<String, Any> into JSON. In my case, I can guarantee that in runtime any value of the map is either a primitive, a list or a map. In case of lists and maps, the values of them are either primitives or lists or maps of the same pattern. Seems, it lays nicely on a json object.

I do not need to deserialize such an object, serialization only.

For now, I'm trying to write a custom serializer for such case but no success yet. How can it be done for both JVM and JS?

@sandwwraith
Copy link
Member

This can be done via using JsonElement, see #175 and #276.
This functionality has been implemented and will be shipped with next release.

@sandwwraith sandwwraith added In next release Committed but not published as artifact yet question labels Dec 7, 2018
@hotchemi
Copy link

@sandwwraith Can you possibly share an example code to deal with Map<String, Any>? 🙇

@sandwwraith
Copy link
Member

@hotchemi You can replace Map type with JsonObject

@dri94
Copy link

dri94 commented Feb 20, 2019

@sandwwraith this isnt really a solution for orgs that already have maps heavily ingrained, which is more likely than having JsonObject everywhere. There should be some support for Map<String, Any> without converting it to a JsonObject or converting every map to <String, JsonElement>.

@sandwwraith
Copy link
Member

JsonObject implements Map<String, JsonElement>. I believe that Any is not very type-safe and much more cumbersome to handle, but this JsonElement itself can be easily converted to Any recursively.

@dri94
Copy link

dri94 commented Feb 21, 2019

Its about converting Any to JsonElement, not the other way around. Any isnt very type-safe but provides a big use case with Maps. As with other serializers it would just thrown an JsonException on unsupported types.

@francos
Copy link

francos commented Jul 7, 2020

@dri94 did you find a way to convert Any to JsonElement in order to convert a Map<String, Any> to JsonObject?

@gunslingor
Copy link

gunslingor commented Aug 22, 2020

It appears there is no way to serialize a dashed variable in Kotlin JS, JSONObject only works on JVM: JetBrains/kotlin-wrappers#339

ERROR: "name contains illegal identifiers that can't appear in javascript identifiers"

        styleManager = jsObject{
            clearProperties = true
        }
        plugins = arrayOf(
            "grapesjs-lory-slider",
            "grapesjs-tabs",
            "grapesjs-custom-code",
            "grapesjs-touch",
            "grapesjs-parser-postcss",
            "grapesjs-tooltip",
            "grapesjs-tui-image-editor",
            "grapesjs-typed",
            "grapesjs-style-bg",
            "grapesjs-preset-webpage"
            //"grapesjs-plugin-filestack"
        )
        pluginsOpts = jsObject<dynamic> {
            `grapesjs-lory-slider` = jsObject<dynamic> {
                sliderBlock = jsObject<dynamic> {
                    category = "Extra"
                }
            }
        }

EDIT/SOLVED: this[ "grapesjs-lory-slider"]

@kabirsaheb
Copy link

This is how I am doing Map to JsonElement for now. I take all primitive to be string (In my case it is fine).


fun List<*>.toJsonElement(): JsonElement {
    val list: MutableList<JsonElement> = mutableListOf()
    this.forEach {
        val value = it as? Any ?: return@forEach
        when(value) {
            is Map<*, *> -> list.add((value).toJsonElement())
            is List<*> -> list.add(value.toJsonElement())
            else -> list.add(JsonPrimitive(value.toString()))
        }
    }
    return JsonArray(list)
}

fun Map<*, *>.toJsonElement(): JsonElement {
    val map: MutableMap<String, JsonElement> = mutableMapOf()
    this.forEach {
        val key = it.key as? String ?: return@forEach
        val value = it.value ?: return@forEach
        when(value) {
            is Map<*, *> -> map[key] = (value).toJsonElement()
            is List<*> -> map[key] = value.toJsonElement()
            else -> map[key] = JsonPrimitive(value.toString())
        }
    }
    return JsonObject(map)
}

@3v1n0
Copy link

3v1n0 commented Jan 8, 2021

I was expecting to be able to use Map<String, Serializable?> instead to get the proper serializer, but looks like this not working either.

No way to get a generic interface for all the possible serializable types?

@3v1n0
Copy link

3v1n0 commented Jan 10, 2021

Well to keep this more generic, I'm using a Map where all the elements have a serializer and we can use the kotlin reflection to get the right one for the handled type, so we can get an object out of the map with just:

fun buildJsonObject(other: Map<String, Any?>) : JsonElement {
    val jsonEncoder = Json{ encodeDefaults = true } // Set this accordingly to your needs
    val map = emptyMap<String, JsonElement>().toMutableMap()

    other.forEach {
        map[it.key] = if (it.value != null)
            jsonEncoder.encodeToJsonElement(serializer(it.value!!::class.starProjectedType), it.value)
        else JsonNull
    }

    return JsonObject(map)
}

And this will still throw a SerializationException in case the value type has not a serializer available.

However, I'm not still fully happy as I'd prefer some more generic serializable type so that can be used with binary when using CBOR serialization, and so where the encoding happens only at the moment we call the Json.encodeToString or Cbor.encodeToByteArray depending whether the serializer supports or not the binary format.

Not to mention that a such built object would just fail with Cbor (Got an error while parsing: java.lang.IllegalStateException: This serializer can be used only with Json format.Expected Encoder to be JsonEncoder, got class kotlinx.serialization.cbor.internal.CborMapWriter).

@3v1n0
Copy link

3v1n0 commented Jan 11, 2021

So, to handle the generic serializer case (such as binary ones), I've crafted some raw KSerializer for Any? that manually serializes the dynamic type of the element other than the value itself.

So basically mimicking what Polymorphic does, I'm not using any experimental or internal APIs but some of them could improve the result, like reusing the type serialName if any (even though, I'm not sure how i can deserialize that).

In the JSON case it could be probably optimized removing the type at all when using a Primitive one. (EDIT: this is done now)

Here's a gist, but suggestions are welcome: https://gist.github.com/3v1n0/ecbc5e825e2921bd0022611d7046690b

@vonox7
Copy link

vonox7 commented Aug 23, 2021

See https://youtrack.jetbrains.com/issue/KTOR-3063, also thanks to @kabirsaheb for the first draft of toJsonElement(). The linked youtrack issue has a more advanced mitigation strategy for this issue, where serializing Map, List, String, Number, Boolean, Enum and null is supported.

@migueltorcha
Copy link

migueltorcha commented Mar 3, 2022

Based on @kabirsaheb, I use the next:

fun Any?.toJsonElement(): JsonElement =
  when (this) {
    null -> JsonNull
    is Map<*, *> -> toJsonElement()
    is Collection<*> -> toJsonElement()
    is Boolean -> JsonPrimitive(this)
    is Number -> JsonPrimitive(this)
    is String -> JsonPrimitive(this)
    is Enum<*> -> JsonPrimitive(this.toString())
    else -> throw IllegalStateException("Can't serialize unknown type: $this")
  }

private fun Collection<*>.toJsonElement(): JsonElement {
  val list: MutableList<JsonElement> = mutableListOf()
  this.forEach { value ->
    when (value) {
      null -> list.add(JsonNull)
      is Map<*, *> -> list.add(value.toJsonElement())
      is Collection<*> -> list.add(value.toJsonElement())
      is Boolean -> list.add(JsonPrimitive(value))
      is Number -> list.add(JsonPrimitive(value))
      is String -> list.add(JsonPrimitive(value))
      is Enum<*> -> list.add(JsonPrimitive(value.toString()))
      else -> throw IllegalStateException("Can't serialize unknown collection type: $value")
    }
  }
  return JsonArray(list)
}

private fun Map<*, *>.toJsonElement(): JsonElement {
  val map: MutableMap<String, JsonElement> = mutableMapOf()
  this.forEach { (key, value) ->
    key as String
    when (value) {
      null -> map[key] = JsonNull
      is Map<*, *> -> map[key] = value.toJsonElement()
      is Collection<*> -> map[key] = value.toJsonElement()
      is Boolean -> map[key] = JsonPrimitive(value)
      is Number -> map[key] = JsonPrimitive(value)
      is String -> map[key] = JsonPrimitive(value)
      is Enum<*> -> map[key] = JsonPrimitive(value.toString())
      else -> throw IllegalStateException("Can't serialize unknown type: $value")
    }
  }
  return JsonObject(map)
}

@BaLaLaLs
Copy link

like is code

    @kotlinx.serialization.Serializable
    data class MyDataClass(var s: String)
    println(mapOf("a" to 1, "b" to "2", "c" to listOf(MyDataClass("3"))).toJsonElement2())

will be throw IllegalStateException

@ghost
Copy link

ghost commented Apr 18, 2022

I'm surprised that such conversion is not supported by the library, it seems to be quite common when working with Java code. Thanks for the workarounds though!

@ss04661
Copy link

ss04661 commented Apr 30, 2022

package test2

import kotlinx.serialization.*
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.json.*
import test1.printlnIt
import java.time.OffsetDateTime
import java.time.format.DateTimeFormatter
import kotlin.test.Test


@Serializable
data class AnyData(
    val name: String,
    @Serializable(with = AnyValueSerializer::class)
    val anyValue: Any?,
    val anyList: List<@Serializable(with = AnyValueSerializer::class) Any?>,
    val anyMap: Map<@Serializable(with = AnyValueSerializer::class) Any?, @Serializable(with = AnySerializer::class) Any?>,
    @Serializable(with = DataFrameSerializer::class)
    val dataFrame: Map<String,List<@Contextual Any?>>, // {date:List<OffsetDateTime>, price:List<Double>}
)

private fun Any?.toJsonPrimitive(): JsonPrimitive {
    return when (this) {
        null -> JsonNull
        is JsonPrimitive -> this
        is Boolean -> JsonPrimitive(this)
        is Number -> JsonPrimitive(this)
        is String -> JsonPrimitive(this)
        // add custom convert
        else -> throw Exception("不支持类型:${this::class}")
    }
}
private fun JsonPrimitive.toAnyValue():Any?{
    val content = this.content
    if (this.isString){
        // add custom string convert
        return content
    }
    if (content.equals("null", ignoreCase = true)){
        return null
    }
    if (content.equals("true", ignoreCase = true)){
        return true
    }
    if (content.equals("false", ignoreCase = true)){
        return false
    }
    val intValue = content.toIntOrNull()
    if (intValue!=null){
        return intValue
    }
    val longValue = content.toLongOrNull()
    if (longValue!=null){
        return longValue
    }
    val doubleValue = content.toDoubleOrNull()
    if (doubleValue!=null){
        return doubleValue
    }
    throw Exception("未知值:${content}")
}

object AnyValueSerializer : KSerializer<Any?> {
    private val delegateSerializer = JsonPrimitive.serializer()
    override val descriptor = delegateSerializer.descriptor
    override fun serialize(encoder: Encoder, value: Any?) {
        encoder.encodeSerializableValue(delegateSerializer, value.toJsonPrimitive())
    }
    override fun deserialize(decoder: Decoder): Any? {
        val jsonPrimitive = decoder.decodeSerializableValue(delegateSerializer)
        return jsonPrimitive.toAnyValue()
    }
}

/**
 * Convert Any? to JsonElement
 */
private fun Any?.toJsonElement(): JsonElement{
    return when (this) {
        null -> JsonNull
        is JsonElement -> this
        is Boolean -> JsonPrimitive(this)
        is Number -> JsonPrimitive(this)
        is String -> JsonPrimitive(this)
        is Iterable<*> -> JsonArray(this.map { it.toJsonElement() })
        // !!! key simply converted to string
        is Map<*, *> -> JsonObject(this.map { it.key.toString() to it.value.toJsonElement() }.toMap())
        // add custom convert
        else -> throw Exception("不支持类型 ${this::class}=${this}}")
    }
}
private fun JsonElement.toAnyOrNull():Any?{
    return when (this) {
        is JsonNull -> null
        is JsonPrimitive -> toAnyValue()
        // !!! key convert back custom object
        is JsonObject -> this.map { it.key to it.value.toAnyOrNull() }.toMap()
        is JsonArray -> this.map { it.toAnyOrNull() }
    }
}

object AnySerializer : KSerializer<Any?> {
    private val delegateSerializer = JsonElement.serializer()
    override val descriptor = delegateSerializer.descriptor
    override fun serialize(encoder: Encoder, value: Any?) {
        encoder.encodeSerializableValue(delegateSerializer, value.toJsonElement())
    }
    override fun deserialize(decoder: Decoder): Any? {
        val jsonPrimitive = decoder.decodeSerializableValue(delegateSerializer)
        return jsonPrimitive.toAnyOrNull()
    }
}

object DataFrameSerializer : KSerializer<Map<String,List<Any?>>> {
    private val stdDateTimeFormatter: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ssZ")
    private val delegateSerializer = JsonObject.serializer()
    override val descriptor = delegateSerializer.descriptor
    override fun serialize(encoder: Encoder, value: Map<String,List<Any?>>) {
        val jsonObject = JsonObject(value.mapValues {
            when(it.key){
                "date" -> JsonArray(it.value.map { v ->
                    JsonPrimitive(stdDateTimeFormatter.format(v as OffsetDateTime))
                })
                "price" -> JsonArray(it.value.map { v ->
                    JsonPrimitive(v as Double)
                })
                else -> throw IllegalStateException("Unknown key${it.key}")
            }
        })
        encoder.encodeSerializableValue(delegateSerializer, jsonObject)
    }
    override fun deserialize(decoder: Decoder): Map<String,List<Any?>> {
        val jsonObject = decoder.decodeSerializableValue(delegateSerializer)
        val map = jsonObject.mapValues {
            when (it.key) {
                "date" -> it.value.jsonArray.map { v ->
                    OffsetDateTime.parse(v.jsonPrimitive.content, stdDateTimeFormatter)
                }
                "price" -> it.value.jsonArray.map { v -> v.jsonPrimitive.double }
                else -> throw IllegalStateException("Unknown key${it.key}")
            }
        }
        return map
    }
}


class SerialTest3 {
    @Test
    fun test1() {
        val anyData = AnyData(
            name = "hello",
            anyValue = null,
            anyList = listOf("hi", 123, Long.MAX_VALUE, 2.7, true, null),
            anyMap = mapOf(
                true to listOf(123, 25.0, false),
                false to listOf(1.2, 1.3, 0.5),
                "RussianDoll" to mapOf(true to listOf(123, 25.0, false), false to listOf(1.2, 1.3, 0.5))
            ),
            dataFrame = mapOf(
                "date" to listOf(OffsetDateTime.now(), OffsetDateTime.now().plusDays(1), OffsetDateTime.now().plusDays(2)),
                "price" to listOf(1.2, 1.3, 0.5)
            ),
        )

        val json = Json.encodeToString(anyData)
        println("json: $json")
        println("class: ${Json.decodeFromString<AnyData>(json)}")
    }
}
output:

json: {"name":"hello","anyValue":null,"anyList":["hi",123,9223372036854775807,2.7,true,null],"anyMap":{"true":[123,25.0,false],"false":[1.2,1.3,0.5],"RussianDoll":{"true":[123,25.0,false],"false":[1.2,1.3,0.5]}},"dataFrame":{"date":["2022-04-30 11:42:06+0800","2022-05-01 11:42:06+0800","2022-05-02 11:42:06+0800"],"price":[1.2,1.3,0.5]}}

class: AnyData(name=hello, anyValue=null, anyList=[hi, 123, 9223372036854775807, 2.7, true, null], anyMap={true=[123, 25.0, false], false=[1.2, 1.3, 0.5], RussianDoll={true=[123, 25.0, false], false=[1.2, 1.3, 0.5]}}, dataFrame={date=[2022-04-30T11:42:06+08:00, 2022-05-01T11:42:06+08:00, 2022-05-02T11:42:06+08:00], price=[1.2, 1.3, 0.5]})

@AdRyAniP
Copy link

Thanks, @kabirsaheb for your solution.
Here another approach based on your solution and the @migueltorcha one:

fun Collection<*>.toJsonElement(): JsonElement = JsonArray(mapNotNull { it.toJsonElement() })

fun Map<*, *>.toJsonElement(): JsonElement = JsonObject(
    mapNotNull {
        (it.key as? String ?: return@mapNotNull null) to it.value.toJsonElement()
    }.toMap(),
)

fun Any?.toJsonElement(): JsonElement = when (this) {
    null -> JsonNull
    is Map<*, *> -> toJsonElement()
    is Collection<*> -> toJsonElement()
    else -> JsonPrimitive(toString())
}

@xuanswe
Copy link

xuanswe commented Feb 24, 2023

fun Collection<*>.toJsonElement(): JsonElement = JsonArray(mapNotNull { it.toJsonElement() })

fun Map<*, *>.toJsonElement(): JsonElement = JsonObject(
    mapNotNull {
        (it.key as? String ?: return@mapNotNull null) to it.value.toJsonElement()
    }.toMap(),
)

fun Any?.toJsonElement(): JsonElement = when (this) {
    null -> JsonNull
    is Map<*, *> -> toJsonElement()
    is Collection<*> -> toJsonElement()
    else -> JsonPrimitive(toString())
}

Hi, do I need to configure anything else to make this work in ktor js client?

I have this error:

'StandaloneCoroutine is cancelling', Caused by: 'Fail to prepare request body for sending. 
The body type is: class JsonObject, with Content-Type: null.

If you expect serialized body, please check that you have installed the corresponding plugin(like `ContentNegotiation`) and set `Content-Type` header.'

Here is my configuration:

val httpClient = HttpClient {
  expectSuccess = true
  install(ContentNegotiation) {
    json()
  }
}

@xuanswe
Copy link

xuanswe commented Feb 24, 2023

Ignore my comment, I need to set manually contentType(ContentType.Application.Json) when creating the request.
I would expect the content negotiation plugin do it automatically.

@Invincibl-e
Copy link

try this:

    @OptIn(InternalSerializationApi::class)
    fun Any?.toJsonElement(): JsonElement =
        when (this) {
            null -> JsonNull
            is Map<*, *> -> toJsonElement()
            is Collection<*> -> toJsonElement()
            is Boolean -> JsonPrimitive(this)
            is Number -> JsonPrimitive(this)
            is String -> JsonPrimitive(this)
            is Enum<*> -> JsonPrimitive(this.toString())
            else -> this.javaClass.kotlin.serializer().let { json.encodeToJsonElement(it, this) }
        }

    private fun Collection<*>.toJsonElement(): JsonElement =
        JsonArray(this.map { it.toJsonElement() })

    private fun Map<String, Any?>.toJsonElement(): JsonElement {
        return JsonObject(this.mapValues { it.value.toJsonElement() })
    }

@yidafu
Copy link

yidafu commented Dec 18, 2023

Base on @migueltorcha

considering Array<*> and XxxArray.

fun Any?.toJsonElement(): JsonElement = when(this) {
    null -> JsonNull
    is Map<*, *> -> toJsonElement()
    is Collection<*> -> toJsonElement()
    is ByteArray -> this.toList().toJsonElement()
    is CharArray -> this.toList().toJsonElement()
    is ShortArray -> this.toList().toJsonElement()
    is IntArray -> this.toList().toJsonElement()
    is LongArray -> this.toList().toJsonElement()
    is FloatArray -> this.toList().toJsonElement()
    is DoubleArray -> this.toList().toJsonElement()
    is BooleanArray -> this.toList().toJsonElement()
    is Array<*> -> toJsonElement()
    is Boolean -> JsonPrimitive(this)
    is Number -> JsonPrimitive(this)
    is String -> JsonPrimitive(this)
    is Enum<*> -> JsonPrimitive(this.toString())
    else -> {
            throw IllegalStateException("Can't serialize unknown type: $this")
    }
}

fun  Map<*, *>.toJsonElement(): JsonElement {
    val map = mutableMapOf<String, JsonElement>()
    this.forEach {key, value ->
        key as String
        map[key] = value.toJsonElement()
    }
    return JsonObject(map)
}

fun Collection<*>.toJsonElement(): JsonElement {
    return JsonArray(this.map { it.toJsonElement() })
}

fun Array<*>.toJsonElement(): JsonElement {
    return JsonArray(this.map { it.toJsonElement() })
}

fun main(args: Array<String>) {
    val obj = mapOf<String, Any>(
        "int" to 1,
        "bool" to  true,
        "float" to 1.2f,
        "double" to 1.2,
        "arrayInt" to intArrayOf(1, 2,3),
        "arrayInt2" to arrayOf(1, 2,3),
        "arrayString" to arrayOf("foo", "bar"),
        "listDouble" to listOf(1.1, 2.2, 3.3),
        "listString" to listOf("goo", "baz"),
        "mapInt" to mapOf("1" to 1, "2" to 2),
    ).toJsonElement()
    println(Json.encodeToString(obj))
}

output

{
    "int":1,
    "bool":true,
    "float":1.2,
    "double":1.2,
    "arrayInt":[
        1,
        2,
        3
    ],
    "arrayInt2":[
        1,
        2,
        3
    ],
    "arrayString":[
        "foo",
        "bar"
    ],
    "listDouble":[
        1.1,
        2.2,
        3.3
    ],
    "listString":[
        "goo",
        "baz"
    ],
    "mapInt":{
        "1":1,
        "2":2
    }
}

samuelAndalon added a commit to ExpediaGroup/graphql-kotlin that referenced this issue Mar 14, 2024
### 📝 Description
Switch from `jackson` to `kotlinx.serialization` for
serialization/deserialization of `GraphQLServerRequest` types.

After running benchmarks we where able to identify that deserializing
`GraphQLServerRequest` with `kotlinx.serialization` is quite faster than
doing it with `jackson`, the reason ? possibly because jackson relies on
reflections to identify deserialization process.

On the other hand, serialization/deserialization of
`GraphQLServerReponse` type is still faster if done with `jackson`,
possibly because of how `kotlinx.serialization` library was designed and
the poor support for serializing `Any` type:
Kotlin/kotlinx.serialization#296, which causes
a lot of memory comsumption.

As part of this PR also including the benchmarks. For that, i created a
separate set of types that are marked with both `jackson` and
`kotlinx.serialization` annotations.


Benchmarks results:
Executed on a MacBookPro 2.6 GHz 6-Core Intel Core i7.

#### GraphQLServerRequest Deserialization

`GraphQLBatchRequest`
4 batched operations, each operation is aprox: 30kb
<img width="1260" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/06e5b218-a35e-4baa-a25e-2be1b3c27a95">

`GraphQLRequest`
<img width="1231" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/e5ecba01-fd41-4872-b3e8-5519414cc918">

#### GraphQLServerResponse Serialization

`GraphQLBatchResponse`
<img width="1240" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/ee84bfa4-d7d1-46b4-b4a8-b3c220998a03">

`GraphQLResponse`
<img width="1197" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/c217e05f-45fc-460e-a059-7667975ee49f">
samuelAndalon added a commit to ExpediaGroup/graphql-kotlin that referenced this issue Apr 1, 2024
Switch from `jackson` to `kotlinx.serialization` for
serialization/deserialization of `GraphQLServerRequest` types.

After running benchmarks we where able to identify that deserializing
`GraphQLServerRequest` with `kotlinx.serialization` is quite faster than
doing it with `jackson`, the reason ? possibly because jackson relies on
reflections to identify deserialization process.

On the other hand, serialization/deserialization of
`GraphQLServerReponse` type is still faster if done with `jackson`,
possibly because of how `kotlinx.serialization` library was designed and
the poor support for serializing `Any` type:
Kotlin/kotlinx.serialization#296, which causes
a lot of memory comsumption.

As part of this PR also including the benchmarks. For that, i created a
separate set of types that are marked with both `jackson` and
`kotlinx.serialization` annotations.

Benchmarks results:
Executed on a MacBookPro 2.6 GHz 6-Core Intel Core i7.

`GraphQLBatchRequest`
4 batched operations, each operation is aprox: 30kb
<img width="1260" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/06e5b218-a35e-4baa-a25e-2be1b3c27a95">

`GraphQLRequest`
<img width="1231" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/e5ecba01-fd41-4872-b3e8-5519414cc918">

`GraphQLBatchResponse`
<img width="1240" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/ee84bfa4-d7d1-46b4-b4a8-b3c220998a03">

`GraphQLResponse`
<img width="1197" alt="image"
src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/c217e05f-45fc-460e-a059-7667975ee49f">
samuelAndalon added a commit to ExpediaGroup/graphql-kotlin that referenced this issue Apr 1, 2024
…1937) (#1944)

Switch from `jackson` to `kotlinx.serialization` for
serialization/deserialization of `GraphQLServerRequest` types.

After running benchmarks we where able to identify that deserializing
`GraphQLServerRequest` with `kotlinx.serialization` is quite faster than
doing it with `jackson`, the reason ? possibly because jackson relies on
reflections to identify deserialization process.

On the other hand, serialization/deserialization of
`GraphQLServerReponse` type is still faster if done with `jackson`,
possibly because of how `kotlinx.serialization` library was designed and
the poor support for serializing `Any` type:
Kotlin/kotlinx.serialization#296, which causes
a lot of memory comsumption.

As part of this PR also including the benchmarks. For that, i created a
separate set of types that are marked with both `jackson` and
`kotlinx.serialization` annotations.

Benchmarks results:
Executed on a MacBookPro 2.6 GHz 6-Core Intel Core i7.

`GraphQLBatchRequest`
4 batched operations, each operation is aprox: 30kb <img width="1260"
alt="image"

src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/06e5b218-a35e-4baa-a25e-2be1b3c27a95">

`GraphQLRequest`
<img width="1231" alt="image"

src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/e5ecba01-fd41-4872-b3e8-5519414cc918">

`GraphQLBatchResponse`
<img width="1240" alt="image"

src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/ee84bfa4-d7d1-46b4-b4a8-b3c220998a03">

`GraphQLResponse`
<img width="1197" alt="image"

src="https://github.com/ExpediaGroup/graphql-kotlin/assets/6611331/c217e05f-45fc-460e-a059-7667975ee49f">

### 📝 Description


### 🔗 Related Issues
@solonovamax
Copy link

solonovamax commented Sep 13, 2024

Using some of what has been given in this issue as well as messing around with it myself, here is some code that (should) be able to serialize anything that kotlinx.serialization can serialize using Json.encodeToJsonElement:

Do note: @file:OptIn(ExperimentalUnsignedTypes::class, ExperimentalSerializationApi::class) needs to be added at the top.
If you wish to remove this, just edit the code and remove all references to any unsigned types.

internal fun Any?.toJsonElement(): JsonElement {
    val serializer = this?.let { Json.serializersModule.serializerOrNull(this::class.java) }

    return when {
        this == null -> JsonNull
        serializer != null -> Json.encodeToJsonElement(serializer, this)
        this is Map<*, *> -> toJsonElement()
        this is Array<*> -> toJsonElement()
        this is BooleanArray -> toJsonElement()
        this is ByteArray -> toJsonElement()
        this is CharArray -> toJsonElement()
        this is ShortArray -> toJsonElement()
        this is IntArray -> toJsonElement()
        this is LongArray -> toJsonElement()
        this is FloatArray -> toJsonElement()
        this is DoubleArray -> toJsonElement()
        this is UByteArray -> toJsonElement()
        this is UShortArray -> toJsonElement()
        this is UIntArray -> toJsonElement()
        this is ULongArray -> toJsonElement()
        this is Collection<*> -> toJsonElement()
        this is Boolean -> JsonPrimitive(this)
        this is Number -> JsonPrimitive(this)
        this is String -> JsonPrimitive(this)
        this is Enum<*> -> JsonPrimitive(this.name)
        this is Pair<*, *> -> JsonObject(
            mapOf(
                "first" to first.toJsonElement(),
                "second" to second.toJsonElement(),
            )
        )
        this is Triple<*, *, *> -> JsonObject(
            mapOf(
                "first" to first.toJsonElement(),
                "second" to second.toJsonElement(),
                "third" to third.toJsonElement(),
            )
        )
        else -> error("Can't serialize '$this' as it is of an unknown type")
    }
}

internal fun Map<*, *>.toJsonElement(): JsonElement {
    return buildJsonObject {
        forEach { (key, value) ->
            if (key !is String)
                error("Only string keys are supported for maps")

            put(key, value.toJsonElement())
        }
    }
}

internal fun Collection<*>.toJsonElement(): JsonElement = buildJsonArray {
    forEach { element ->
        add(element.toJsonElement())
    }
}

internal fun Array<*>.toJsonElement(): JsonElement = buildJsonArray {
    forEach { element ->
        add(element.toJsonElement())
    }
}

internal fun BooleanArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun ByteArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun CharArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it.toString())) } }
internal fun ShortArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun IntArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun LongArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun FloatArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun DoubleArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }

internal fun UByteArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun UShortArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun UIntArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
internal fun ULongArray.toJsonElement(): JsonElement = buildJsonArray { forEach { add(JsonPrimitive(it)) } }
Output
public fun main() {
    val obj = mapOf(
        "bool" to true,
        "byte" to 1.toByte(),
        "char" to '2',
        "short" to 3.toShort(),
        "int" to 4,
        "long" to 5.toLong(),
        "float" to 1.2f,
        "double" to 1.2,
        "ubyte" to 0.toUByte(),
        "ushort" to 1.toUShort(),
        "uint" to 2u,
        "ulong" to 3uL,
        "enum" to YesNo.YES,
        "pair" to ("foo" to "bar"),
        "triple" to Triple("foo", "bar", "baz"),
        "unit" to Unit,
        "duration" to 1.seconds,
        "boolArray" to booleanArrayOf(true, false, true),
        "byteArray" to byteArrayOf(1, 2, 3),
        "charArray" to charArrayOf('1', '2', '3'),
        "shortArray" to shortArrayOf(1, 2, 3),
        "intArray" to intArrayOf(1, 2, 3),
        "longArray" to longArrayOf(1, 2, 3),
        "floatArray" to floatArrayOf(1.0f, 1.1f, 1.2f, 1.3f),
        "doubleArray" to doubleArrayOf(1.0, 1.1, 1.2, 1.3),
        "ubyteArray" to ubyteArrayOf(1u, 2u, 3u),
        "ushortArray" to ushortArrayOf(1u, 2u, 3u),
        "uintArray" to uintArrayOf(1u, 2u, 3u),
        "ulongArray" to ulongArrayOf(1u, 2u, 3u),
        "arrayOfInt" to arrayOf(1, 2, 3),
        "arrayOfString" to arrayOf("foo", "bar"),
        "listOfDouble" to listOf(1.1, 2.2, 3.3),
        "listOfString" to listOf("foo", "bar"),
        "setOfString" to setOf("foo", "bar", "baz"),
        "mapOfStringInt" to mapOf("1" to 1, "2" to 2),
    ).toJsonElement()
    println(Json.encodeToString(obj))
}

public enum class YesNo {
    YES, NO
}
{
    "bool": true,
    "byte": 1,
    "char": "2",
    "short": 3,
    "int": 4,
    "long": 5,
    "float": 1.2,
    "double": 1.2,
    "ubyte": 0,
    "ushort": 1,
    "uint": 2,
    "ulong": 3,
    "enum": "YES",
    "pair": [ "foo", "bar" ],
    "triple": [ "foo", "bar", "baz" ],
    "unit": {},
    "duration": "PT1S",
    "boolArray": [ true, false, true ],
    "byteArray": [ 1, 2, 3 ],
    "charArray": [ "1", "2", "3" ],
    "shortArray": [ 1, 2, 3 ],
    "intArray": [ 1, 2, 3 ],
    "longArray": [ 1, 2, 3 ],
    "floatArray": [ 1.0, 1.1, 1.2, 1.3 ],
    "doubleArray": [ 1.0, 1.1, 1.2, 1.3 ],
    "ubyteArray": [ 1, 2, 3 ],
    "ushortArray": [ 1, 2, 3 ],
    "uintArray": [ 1, 2, 3 ],
    "ulongArray": [ 1, 2, 3 ],
    "arrayOfInt": [ 1, 2, 3 ],
    "arrayOfString": [ "foo", "bar" ],
    "listOfDouble": [ 1.1, 2.2, 3.3 ],
    "listOfString": [ "foo", "bar" ],
    "setOfString": [ "foo", "bar", "baz" ],
    "mapOfStringInt": {
        "1": 1,
        "2": 2
    }
}

this should also be able to serialize/deserialize Uuids without any issue, however I have yet to test that.

Edit 1: Fixed a bug with CharArray.toJsonElement() serializing to integers instead of strings
Edit 2: Fixed a bug with Pair/Triple serializing to arrays instead of objects (oopsies)
Edit 3: Clarify serialization of Uuids

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
In next release Committed but not published as artifact yet question
Projects
None yet
Development

No branches or pull requests