Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Gson.getDelegateAdapter not working properly for JsonAdapter #2435

Conversation

Marcono1234
Copy link
Collaborator

@Marcono1234 Marcono1234 commented Jul 15, 2023

Purpose

Fixes #1028
Supersedes #1690

Description

Unlike #1690 this implementation here tracks which TypeAdapterFactory instances have been created for @JsonAdapter placed on classes, and only for them lets Gson.getDelegateAdapter skip past the JsonAdapterAnnotationTypeAdapterFactory.

This pull request also in general changes the behavior of Gson.getDelegateAdapter to hopefully be more reasonable when skipPast is null (which was previously supported, but undocumented) or when skipPast is not a known factory. In these cases the method now behaves like Gson.getAdapter (instead of assuming that the factory came from @JsonAdapter on a class).

Checklist

  • New code follows the Google Java Style Guide
  • If necessary, new public API validates arguments, for example rejects null
  • New public API has Javadoc
    • Javadoc uses @since $next-version$
      ($next-version$ is a special placeholder which is automatically replaced during release)
  • If necessary, new unit tests have been added
    • Assertions in unit tests use Truth, see existing tests
    • No JUnit 3 features are used (such as extending class TestCase)
    • If this pull request fixes a bug, a new test was added for a situation which failed previously and is now fixed
  • mvn clean verify javadoc:jar passes without errors

* types, our stats factory will not count the number of String or primitives that will be
* read or written.
*
* <p>If {@code skipPast} is {@code null} or a factory which has neither been registered
Copy link
Collaborator Author

@Marcono1234 Marcono1234 Jul 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The section starting here is new; the changes above only remove one redundant leading space after the * (unfortunately the GitHub diff does not indicate that very well).

Comment on lines 92 to 101
T instance = instanceCreator.createInstance(type);
if (instance == null) {
throw new RuntimeException("InstanceCreator " + instanceCreator + " returned null for type " + type);
}

if (!rawType.isInstance(instance)) {
throw new ClassCastException("InstanceCreator " + instanceCreator + " created instance of wrong type;"
+ " expected " + rawType.getName() + " but got instance of unrelated type " + instance.getClass().getName());
}
return instance;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this for sanity, and to avoid some checks in JsonAdapterAnnotationTypeAdapterFactory.isClassJsonAdapterFactory to handle bad InstanceCreators. Most likely Gson was already failing for bad InstanceCreators before, but probably at a completely unrelated place which would have made debugging more difficult.

Copy link
Member

@eamonnmcmanus eamonnmcmanus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Only very minor comments. I'm going to run this through all of Google's internal tests.

@Marcono1234
Copy link
Collaborator Author

Thanks! The fact that JsonAdapterAnnotationTypeAdapterFactory.isClassJsonAdapterFactory also performs the annotation lookup does not feel that clean to me, but I hope it is fine with the current implementation anyway.

I am a bit afraid that users unwittingly rely on the current broken behavior, or that users are currently intentionally calling getDelegateAdapter with an unknown (or null) factory to be guaranteed to get the Gson reflection-based adapter (even though this was an implementation detail).
But hopefully your test run shows if that is really a valid concern.

@eamonnmcmanus
Copy link
Member

Still waiting for test reruns, but looking good so far. (There are so many tests that invariably a few of them fail for unrelated reasons.)

@Marcono1234
Copy link
Collaborator Author

Marcono1234 commented Jul 15, 2023

Did a code search on GitHub now and unfortunately found a few cases of getDelegateAdapter(null, ...):

  • vertigo-io/vertigo-extensions, eutro/Tetraits
    Both seem to work around the limitation that JsonSerializer and JsonDeserializer do not support delegation, by making the eventually created Gson instance accessible through a field and then in the serializer / deserializer use getDelegateAdapter with null because they don't have access to a TypeAdapterFactory there.
    I am afraid with the changes of this pull request this would lead to an infinite recursion because the JsonSerializer / JsonDeserializer would keep retrieving itself.
  • JuiceLabs/kotlin-fhir-model
    Seems to be a custom variant of gson-extras' RuntimeTypeAdapterFactory; not sure why they are using null as skipPast.

These libraries don't seem to be that popular so maybe the risk of causing issues for them is not that big (we can also warn the authors about their problematic getDelegateAdapter usage).
(Edit: Have created issues on vertigo-io/vertigo-extensions and eutro/Tetraits now to inform the authors.)

Also, it appears commit 83e5a49 which introduced the getDelegateAdapter method for Gson did not mention null as supported value, and the code from back then looks like it would have thrown an IllegalArgumentException for null because the workaround for @JsonAdapter did not exist back then. Most likely only since b2c00a3 a null argument would have worked the way it does now. So it seems the authors of those projects above are (wittingly?) relying on undocumented Gson implementation details.

@eamonnmcmanus
Copy link
Member

I'm hesitating a bit here because the internal tests did show up one test failure (or actually a group of failures in related tests, probably having the same cause.

I'm getting a failure like this:

java.lang.ClassCastException: InstanceCreator [...].GsonUtils$ImmutableListTypeAdapterFactory$$Lambda$148/0x00000008012f8000@2926ed77 created instance of wrong type; expected com.google.common.collect.ImmutableList but got instance of unrelated type java.util.ArrayList
        at com.google.gson.internal.ConstructorConstructor.useInstanceCreator(ConstructorConstructor.java:99)
        at com.google.gson.internal.ConstructorConstructor$2.construct(ConstructorConstructor.java:127)
        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:80)
        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
        at [...].GsonUtils$ImmutableListTypeAdapterFactory$1.read(GsonUtils.java:51)
        at [...].AutoValue_ButtonGroupModel$GsonTypeAdapter.read(AutoValue_ButtonGroupModel.java:84)
        at [...].AutoValue_ButtonGroupModel$GsonTypeAdapter.read(AutoValue_ButtonGroupModel.java:27)
        at com.google.gson.TypeAdapter.fromJson(TypeAdapter.java:269)
        at com.google.gson.TypeAdapter.fromJson(TypeAdapter.java:285)

Maybe there is some sort of interaction between this ImmutableListTypeAdapterFactory and the TypeAdapter created by auto-value-gson. This is ImmutableListTypeAdapterFactory:

  private static class ImmutableListTypeAdapterFactory implements TypeAdapterFactory {

    @Nullable
    @Override
    public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
      if (!ImmutableList.class.isAssignableFrom(type.getRawType())) {
        return null;
      }
      final TypeAdapter<T> delegate = gson.getDelegateAdapter(this, type);
      return new TypeAdapter<T>() {
        @Override
        public void write(JsonWriter out, T value) throws IOException {
          delegate.write(out, value);
        }

        @Override
        @SuppressWarnings("unchecked")
        public T read(JsonReader in) throws IOException {
          return (T) ImmutableList.copyOf((List) delegate.read(in));
        }
      };
    }

    public static <E> InstanceCreator<List<E>> newCreator() {
      return type -> new ArrayList<>();
    }
  }

Registered like this:

  private static final GsonBuilder GSON_BUILDER = new GsonBuilder();

  static {
    GSON_BUILDER.registerTypeAdapterFactory(new ImmutableListTypeAdapterFactory());
    GSON_BUILDER.registerTypeAdapter(
        ImmutableList.class, ImmutableListTypeAdapterFactory.newCreator());
  }

  public static Gson get() {
    return GSON_BUILDER.create();
  }

I'm not sure if this code is doing something unusual. I do notice that the code in ConstructorConstructor that is throwing the exception is new with this PR.

That's as much as I had time for here but I may be able to look into it further later. Let me know if this much is enough to get an idea of the problem, and in particular whether it is something other users might run into.

Copy link
Member

@eamonnmcmanus eamonnmcmanus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Marking this as "request changes" so we don't accidentally merge it before the test-failure issue is resolved.)

@Marcono1234
Copy link
Collaborator Author

Marcono1234 commented Jul 24, 2023

Thanks a lot for the feedback!

If I understand the code of ImmutableListTypeAdapterFactory correctly, the idea of the deserialization logic is:

Deserialize as ArrayList and then construct an ImmutableList from that.

However, the way this is implemented relies heavily on Java's type erasure and Gson implementation details (possibly even to a degree where you could call it "misuse"?):

Get the delegate for ImmutableList (the generic Collection adapter factory) and let it handle the deserialization. But because ImmutableList doesn't have a public no-args constructor and is immutable, instead create an ArrayList when requested to create an instance of ImmutableList. So the Collection adapter thinks it is creating an ImmutableList while in reality it creates an ArrayList, which is then used by ImmutableListTypeAdapterFactory to create the ImmutableList.

Most likely these implementation details of ImmutableListTypeAdapterFactory don't leak because the bogus InstanceCreator can only be reached by the adapter of ImmutableListTypeAdapterFactory because all requests for ImmutableList go through it; but this is certainly quite brittle. If Gson would in any other way allow using the InstanceCreators registered for it, a use might see a ClassCastException (instead of a different exception saying that ImmutableList cannot be instantiated).

Maybe a proper implementation would be to have ImmutableListTypeAdapterFactory construct a TypeToken<List<...>> (using TypeToken.getParameterized) based on type (using an approach similar to #1952, respectively your comment there) and use that for requesting the deserialization delegate adapter. Or similar to how it is done here (except using List instead of ArrayList).


Technically the changes to InstanceCreator validation are not needed for this pull request, so I can comment them out, add a TODO comment and add @Ignore to the corresponding test if you want.

@eamonnmcmanus
Copy link
Member

Technically the changes to InstanceCreator validation are not needed for this pull request, so I can comment them out, add a TODO comment and add @Ignore to the corresponding test if you want.

That sounds like a great idea. If those changes are not strictly necessary, I think everything else is pretty uncontroversial. Even taking into account your code-search findings.

We could then have a separate PR for the InstanceCreator validation and discuss there whether it is safe.

@Marcono1234
Copy link
Collaborator Author

Marcono1234 commented Jul 25, 2023

Have reverted the InstanceCreator validation changes now and have extracted them into #2446.

Though while thinking a bit more about these changes, what do you think about rejecting null as skipPast for Gson.getDelegateAdapter? The handling and support of null is undocumented and will be changed by this pull request anyways. Maybe the cleaner solution would then be to reject null. I think this would be consistent with most other Gson methods which also don't support null, and it might also led to a more helpful exception message than a potential StackOverflowError due to infinite recursion for users who currently use getDelegateAdapter(null, ...).

@Marcono1234
Copy link
Collaborator Author

Marcono1234 commented Aug 17, 2023

@eamonnmcmanus, would it be ok if I changed getDelegateAdapter to reject null as skipPast (as proposed above), and could you then please run the Google tests with that change?

@eamonnmcmanus
Copy link
Member

@eamonnmcmanus, would it be ok if I changed getDelegateAdapter to reject null as skipPast (as proposed above), and could you then please run the Google tests with that change?

I can certainly try that. Just adding a second Objects.requireNonNull at the start of Gson.getDelegateAdapter. I see that there are three Gson tests that fail if I do that, but I'll ignore those and see what else might fail.

@Marcono1234
Copy link
Collaborator Author

Ah sorry, I should have been more explicit: Rejecting null also requires a small change in the JsonAdapterAnnotationTypeAdapterFactory class. I have adjusted that now.

If the test results indicate that rejecting null is a problem I can revert the latest commit again. (Though I would be a bit curious then, if with null being allowed but the behavior change for null compared to before, the caller really behaves as expected.)

@eamonnmcmanus
Copy link
Member

I have confirmed that rejecting skipPast == null does not cause any test failures, so I think we can go ahead with it. I can merge it if you confirm it is ready.

Previously `getDelegateAdapter` called `factories.contains(skipPast)`,
but unlike the other comparisons which check for reference equality,
that would have used the `equals` method.
This could lead to spurious "GSON cannot serialize ..." exceptions
if two factory instances compared equal, but the one provided as
`skipPast` had not been registered yet.
@Marcono1234
Copy link
Collaborator Author

Thanks a lot for checking! I have made a few small additional adjustments; sorry for the trouble. It should be functionally nearly identical to the previous implementation (except for corner cases with factories with custom equals); but if you want you can run the tests again to be sure.

I think these changes should be ready for merging now.

@eamonnmcmanus eamonnmcmanus merged commit 7ee5ad6 into google:main Aug 23, 2023
@Marcono1234 Marcono1234 deleted the marcono1234/JsonAdapter-getDelegateAdapter branch August 23, 2023 09:07
tibor-universe pushed a commit to getuniverse/gson that referenced this pull request Sep 14, 2024
…oogle#2435)

* Fix `Gson.getDelegateAdapter` not working properly for `JsonAdapter`

* Address review feedback and add comments regarding thread-safety

* Revert InstanceCreator instance validation

* Disallow `null` as `skipPast`

* Avoid `equals` usage in `getDelegateAdapter` & minor other changes

Previously `getDelegateAdapter` called `factories.contains(skipPast)`,
but unlike the other comparisons which check for reference equality,
that would have used the `equals` method.
This could lead to spurious "GSON cannot serialize ..." exceptions
if two factory instances compared equal, but the one provided as
`skipPast` had not been registered yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

getDelegateAdapter() does not work properly in TypeAdapterFactory used from JsonAdapter annotation
2 participants