Support arbitrary Number implementation for Object and Number deserialization

[lyubomyr-shaydariv]

RFC 8259, 6 Numbers allows numbers of arbitrary range and precision specifying that a particular implementation may set some limits for numeric values. As of Gson 2.8.4 uses the following deserialization strategies:

• If the result type is declared as Object, Gson always returns Doubles for all encountered JSON numbers. In this case, the minimum and maximum hold number value fits the limits of Double which cannot hold exact Long values that exceed the Double exact precision limits (2^53+1).
• ‎If the result type is declared as Number and there are no custom Number adapters, Gson always returns LazilyParsedNumbers that hold original string values parsing them to a requested type lazily allowing to use the whole range of Long. However, it does not allow to use numbers of arbitrary range and precision, and does not expose its hold string as a BigDecimal.

In order to fix these limitations and preserve backwards compatibility, some sort of "to-number" strategies might be accepted in GsonBuilder to override the default behavior of Gson. This pull-request introduces such a strategy interface to be used in the built-in Object and Number type adapters. There are also four strategies implemented in this PR (two standard to keep the backwards compatibility and two enhanced to overcome the limitations) using an enumeration:

• ToNumberPolicy.DOUBLE that implements the default behavior of the Object type adapter returning Doubles only.
• ToNumberPolicy.LAZILY_PARSED_NUMBER that implements the default behavior of the Number type adapter.
• ToNumberPolicy.LONG_OR_DOUBLE that tries to parse a number as a Long, otherwise then tries to parse it as a Double, if the number cannot be parsed as a Long.
• ToNumberPolicy.BIG_DECIMAL that can parse numbers of arbitrary range and precision.

Call-site is expected to extract proper values using the methods declared in java.lang.Number.

Examples of use:

Default behavior, backwards-compatible with the previous versions of Gson

Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(ToNumberPolicy.DOUBLE) // explicit default, may be omitted
  .create();
List<Object> actual = gson.fromJson("[null, 10, 10.0]", new TypeToken<List<Object>>() {}.getType());
List<Double> expected = Arrays.asList(null, 10.0, 10.0);
assertEquals(expected, actual);

Gson gson = new GsonBuilder()
    .setNumberToNumberStrategy(ToNumberPolicy.LAZILY_PARSED_NUMBER) // explicit default, may be omitted
    .create();
List<Number> actual = gson.fromJson("[null, 10, 10.0]", new TypeToken<List<Number>>() {}.getType());
List<Object> expected = Arrays.<Object>asList(null, new LazilyParsedNumber("10"), new LazilyParsedNumber("10.0"));
assertEquals(expected, actual);

Object-declared numbers are always LazilyParsedNumbers

Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(ToNumberPolicy.LAZILY_PARSED_NUMBER)
  .create();
List<Object> actual = gson.fromJson("[null, 10, 10.0]", new TypeToken<List<Object>>() {}.getType());
List<LazilyParsedNumber> expected = Arrays.asList(null, new LazilyParsedNumber("10"), new LazilyParsedNumber("10.0"));
assertEquals(expected, actual);

Number-declared numbers are always Doubles

Gson gson = new GsonBuilder()
  .setNumberToNumberStrategy(ToNumberPolicy.DOUBLE)
  .create();
List<Number> actual = gson.fromJson("[null, 10, 10.0]", new TypeToken<List<Number>>() {}.getType());
List<Double> expected = Arrays.asList(null, 10.0, 10.0);
assertEquals(expected, actual);

Object-declared numbers are either Long or Double

Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(ToNumberPolicy.LONG_OR_DOUBLE)
  .create();
List<Object> actual = gson.fromJson("[null, 10, 10.0]", new TypeToken<List<Object>>() {}.getType());
List<? extends Number> expected = Arrays.asList(null, 10L, 10.0);
assertEquals(expected, actual);

Object-declared numbers are BigDecimals

Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(ToNumberPolicy.BIG_DECIMAL)
  .create();
List<Object> actual = gson.fromJson("[null, 3.141592653589793238462643383279, 1e400]", new TypeToken<List<Object>>() {}.getType());
List<BigDecimal> expected = Arrays.asList(null, new BigDecimal("3.141592653589793238462643383279"), new BigDecimal("1e400"));
assertEquals(expected, actual);

Custom bytes-only:

Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(new ToNumberStrategy() {
    @Override
    public Byte toNumber(final JsonReader in)
            throws IOException {
      return (byte) in.nextInt();
    }
  })
  .create();
List<Object> actual = gson.fromJson("[null, 10, 20, 30]", new TypeToken<List<Object>>() {}.getType());
List<Byte> expected = Arrays.asList(null, (byte) 10, (byte) 20, (byte) 30);
assertEquals(expected, actual);

Custom deserialization does not affect Byte-declared deserialization:

ToNumberStrategy fail = new ToNumberStrategy() {
    @Override
    public Byte toNumber(final JsonReader in) {
      throw new AssertionError();
    }
  };
Gson gson = new GsonBuilder()
  .setObjectToNumberStrategy(fail)
  .setNumberToNumberStrategy(fail)
  .create();
List<Object> actual = gson.fromJson("[null, 10, 20, 30]", new TypeToken<List<Byte>>() {}.getType());
List<Byte> expected = Arrays.asList(null, (byte) 10, (byte) 20, (byte) 30);
assertEquals(expected, actual);

[lyubomyr-shaydariv]

The pull request addresses the same issue that's addressed in:

• issues: Numbers are converted to doubles #504, number double or integer #525, integer values of array become a float values #559, BigDecimal support #812, GSON is always converting a long number to scientific notation format. #968, Parsing a long value leads to different value as double #1065, Gson doesn't deserialise Long numbers correctly #1084, ObjectTypeAdapter error when the value type is a Number #1257, Gson serialized integers to decimals #1313, reading json file integer values as double values in ObjectTypeAdapter #1351, Allow JsonReader to tell if it peeked a long or double #1621, integer 、long toObject is deouble,This leads to many problems #1622, Double / Long deserialize #1679, Long upper limit value #1727;
• pull-requests: enhance ObjectTypeBinder to works with number type 'int' #989, Supporting Long numbers during desiralization #1267, Parse JSON numbers to Number type instead of Double. #1288;
• S.O.: 50442637, 51887197, 55286855.

Related:

• issues: BigDecimal equals problem #904, JsonWriter don't work correctly with float #1127, LazilyParsedNumber does not implement Comparable<LazilyParsedNumber> #1579, Gson deserialize Object array problem #1580.

[xavix-yo]

Great!

[arouel]

Are there plans to merge and release it?

[Taipeek]

Still waiting for this to be released :)

[nonight89]

Thank god! Finally you guys fix this! :)

[Catfriend1]

lgtm

[grieser-careoline]

please, apply as soon as possible

[douglasom]

Is this PR ever going to be applied? If not with this enhancement, what is the approach recommended by the Gson team for deserializing numbers?

If I'm deserializing as a Map<String, Object> and I have some Integer values how to avoid for them to be interpreted as a Double?

[roycarser]

is this PR acceptable now ？

[Marcono1234]

Could you please not force push next time? It requires looking at all the files again to find out what changed and it does not show any of the previous review comments in the code anymore.

For a clean commit history, all the commits can then later be squashed when the pull request is merged.

[lyubomyr-shaydariv]

:)

Could you please not force push next time? It requires looking at all the files again to find out what changed and it does not show any of the previous review comments in the code anymore.

https://github.com/lyubomyr-shaydariv/gson/commits/to-number-strategy%2Bhistory

For a clean commit history, all the commits can then later be squashed when the pull request is merged.

It depends on how the repository maintainers do merges. From what I see in the git log graph, squashes are not a must. and I'd like to keep it as clean as possible.

[Marcono1234]

Some more minor things, but it looks pretty good 👍

[Marcono1234]

Sorry, still some more minor things which could possibly be improved (if you like).

[lyubomyr-shaydariv]

Sorry, still some more minor things which could possibly be improved (if you like).

No problem, please provide either an inline diff (or a patch as a comment) or provide a patch on top of the "+history" branch as a PR into my repository (so that I could merge/cherry-pick your suggestions on top of the branch and then I could squash all these changes into this branch) addressing the unresolved comments (3 remaining as I could resolve the first two only). Thank you.

[googlebot]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[Marcono1234]

@googlebot I consent.

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[erdemtpz]

@googlebot I consent.

[Bilecen]

@googlebot I consent.

[kodkafali]

@googlebot I consent.

[alperentoksoz]

@googlebot I consent.

[osmantufekci]

@googlebot I consent.

[yakupdmrr]

@googlebot I consent.

[sorhanselcuk]

@googlebot I consent.

[ogunozdemir]

@googlebot I consent.

[mfg41]

@googlebot I consent.

[rmirabelle]

I don't understand this "feature" of Gson.

Assuming objects of this shape:

data class MyThing(
    val answer: Any? = null
)

And assuming incoming JSON like so:

[
    {
        "answer": "1"
    },
    {
        "answer": 1
    }
    {
        "answer": 1.0
    },
    {
        "answer": null
    }
]

The very obvious solution to the above is for Gson to see the target type of Any?, then infer the destination type based on the JSON value, setting the answer values to String, Int, Double and null respectively. This doesn't strike me as even marginally difficult.

QUITE unfortunately, Gson still doesn't do this and is happy to corrupt incoming JSON values, requiring that AFTER conversion, we manually inspect the resulting object and manually correct all of Gson's errors. To compound this issue, I cannot craft my own custom converter either, because that too, would require selecting a single type only.

[Marcono1234]

The very obvious solution to the above is for Gson to see the target type of Any?, then infer the destination type based on the JSON value, setting the answer values to String, Int, Double and null respectively.

When you tell Gson to deserialize a Java Object / Kotlin Any and it finds a JSON number it deserializes it by default as Double regardless of the format of the number. I assume for historical reasons that could not be changed which is why this PR here was proposed to allow customizing the behavior.

The ToNumberStrategy allows you to implement the functionality you want. A very simple implementation which just checks if the number contains a . could look like this:

val gson = GsonBuilder()
    .setObjectToNumberStrategy { reader ->
        // Read the JSON number as string to inspect its format
        val numberAsString = reader.nextString()

        if (numberAsString.contains('.')) numberAsString.toDouble()
            else numberAsString.toInt()
    }
    .create()

You could also have a look at the implementation of Gson's ToNumberPolicy.LONG_OR_DOUBLE which implements the same concept except for Long and Double, and also rejects non-finite values and produces more helpful exception messages when parsing the number fails.

[rmirabelle]

@Marcono1234 Thanks for the very thoughtful response.