Provide opt-in for custom converters to handle null

[layomia]

Motivation

The serializer does not pass null to custom converters for reference types on serialization and deserialization. Instead, it handles null by returning a null instance on deserialization (or throwing JsonException/returning a default value for primitive value types, depending on if there's an internal converter), and writing null directly with the writer on serialization. This is primarily for performance to skip an extra call to the converter. In addition, we didn't want callers to have to check for null at the start of every Read and Write method override.

However, the serializer does pass null tokens on deserialization to converters for value types. This is to support internal logic for deserializing JsonElement and Nullable<T> instances, where null is a valid token.

null is also passed to converters for value types because it is not a valid CLR value for these types. This way, the converter can determine what to do with this "invalid" token.

This feature provides a way for custom converters to handle null, regardless of if they convert value or reference types. It has to be opt-in, otherwise it would be a breaking change.

Users may want to handle null for various reasons, including validation and setting/writing default values instead of null. Some scenarios:

• https://stackoverflow.com/questions/59792850/system-text-json-serialize-null-strings-into-empty-strings-globally
• https://stackoverflow.com/questions/23830206/json-convert-empty-string-instead-of-null
• https://stackoverflow.com/a/58443810/5609121

API Proposal

namespace System.Text.Json.Serialization
{
    public abstract partial class JsonConverter<T> : System.Text.Json.Serialization.JsonConverter
    {
        /// <summary>
        /// Indicates whether <see langword="null"> should be passed to the converter
        /// on serialization, and whether <see cref="JsonTokenType.Null"> should be
        /// passed on deserialization.
        /// The default value is <see langword="true"> for converters for value types,
        /// and <see langword="false"> for converters for reference types.
        /// </summary>
        public virtual bool HandleNull { get { throw null; } }
    }
}

Usage

public class Profile
{
    public string FirstName { get; set; }

    public string LastName { get; set; }

    [JsonConverter(typeof(ProfileUriConverter))]
    public Uri ProfileUri { get; set; }
}

public class ProfileUriConverter : JsonConverter<Uri>
{
    public override bool HandleNull { get; } = true;

    public override Uri Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokeType == JsonTokenType.Null)
        {
            return new Uri("https://mycompany.com");
        }

        string uriString = reader.GetString();
        if (Uri.TryCreate(uriString, UriKind.RelativeOrAbsolute, out Uri? value))
        {
            return value;
        }

        throw new JsonException();
    }

    public override void Write(Utf8JsonWriter writer, Uri value, JsonSerializerOptions options)
    {
        if (value == null)
        {
            value = new Uri("https://mycompany.com");
        }

        writer.WriteStringValue(value.OriginalString);
    }
}

public static void HandleUserProfile()
{
    string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""ProfileUri"":null}";

    Profile profile = JsonSerializer.Deserialize<Profile>(json);
    Console.WriteLine(profile.FirstName); // Jet
    Console.WriteLine(profile.LastName); // Doe
    Console.WriteLine(profile.ProfileUri.OrignalString); // https://mycompany.com

    profile.ProfileUri = null;

    json = JsonSerializer.Serialize(profile);
    Console.WriteLine(json); // {"FirstName":"Jet","LastName":"Doe","ProfileUri":"https://mycompany.com"}
}

Notes

• We can't make HandleNull default to false for value types because that would be a breaking change.

[terrajobst]

Video

• In this proposal the converter is only called when a null value is included in the payload. If the property is missing in the payload, the the convert still won't be called. Does this satisfy the customer request? While this should probably be an independent feature it seems we'll want to make sure the design for both will work well together.
• Should this be on the non-generic base JsonConverter?
  • We concluded no, because the Read/Write methods are also only on JsonConverter<T>

namespace System.Text.Json.Serialization
{
    public partial class JsonConverter<T>
    {
        public virtual bool HandleNull { get; }
    }
}

[bartonjs]

Good as proposed, consider nullable annotations on the callback methods.

    public abstract partial class JsonConverter<T> : System.Text.Json.Serialization.JsonConverter
    {
        // Also update the nullable annotations for the serialize (parameter)/deserialize (return) callbacks
        public virtual bool HandleNull { get { throw null; } }
    }

[nathan-alden-sr]

In case anyone needs a solution to this problem specifically to be able to serialize an empty IList<T> as null instead of [], I created a wrapper class. See here.