[API Proposal]: Allow keys with colons in config

[SteveDunn]

Background and motivation

As described in issue #42643, some users want to be able to have configuration keys that contain a colon character.

However, the colon character is the default separator for config keys, so if I had a config key named https://google.es, that would be translated to a composite key, the first being https, and the second being //google.es.

The purpose of this API suggestion is to enable users to specify their own separator character for configuration so that they can have keys containing colons, or any other character they wish.

API Proposal

The API changes are listed in the PR: https://github.com/dotnet/runtime/pull/66886/files#diff-ec34a58f9fd18f4d4fccabf4efc22dd555b9d32963b26419ed107c872b67356f .

We want overloads to provide the separator character:

public static partial class JsonConfigurationExtensions
{
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, Microsoft.Extensions.FileProviders.IFileProvider? provider, string path, bool optional, bool reloadOnChange) { throw null; }
+   public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, Microsoft.Extensions.FileProviders.IFileProvider? provider, string path, bool optional, bool reloadOnChange, string separator = ":") { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, System.Action<Microsoft.Extensions.Configuration.Json.JsonConfigurationSource>? configureSource) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path) { throw null; }
+   public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional) { throw null; }
+   public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, bool reloadOnChange) { throw null; }
+   public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, bool reloadOnChange, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonStream(this IConfigurationBuilder builder, System.IO.Stream stream) { throw null; }
+   public static IConfigurationBuilder AddJsonStream(this IConfigurationBuilder builder, System.IO.Stream stream, string separator = ":") { throw null; }
}

ConfigurationPath.cs would have a new method to complement Combine, named CombineWith:

  public static string Combine(params string[] pathSegments) { throw null; }
+ public static string CombineWith(string separator, params string[] pathSegments) { throw null; }

... and a new method to complement GetSectionKey, named GetSectionKeyWith:

  public static string? GetSectionKey(string? path) { throw null; }
+ public static string? GetSectionKeyWith(string separator, string? path) { throw null; }

ConfigurationKeyComparer.cs will have an overloaded constructor to specify the separator:

public partial class ConfigurationKeyComparer : System.Collections.Generic.IComparer<string>
{
        public ConfigurationKeyComparer() { }
+       public ConfigurationKeyComparer(string separator) { }
        public static Microsoft.Extensions.Configuration.ConfigurationKeyComparer Instance { get { throw null; } }
        public static Microsoft.Extensions.Configuration.ConfigurationKeyComparer GetInstanceFor(string separator){ throw null; }
        public int Compare(string? x, string? y) { throw null; }
}

ConfigurationProvider.cs will have a new method to expose the key separator that it's using:

public abstract partial class ConfigurationProvider : Microsoft.Extensions.Configuration.IConfigurationProvider
{
    protected ConfigurationProvider() { }
+   public virtual string GetDelimiter() { throw null; }
    protected System.Collections.Generic.IDictionary<string, string?> Data { get { throw null; } set { } }
    public virtual System.Collections.Generic.IEnumerable<string> GetChildKeys(System.Collections.Generic.IEnumerable<string> earlierKeys, string? parentPath) { throw null; }
    public Microsoft.Extensions.Primitives.IChangeToken GetReloadToken() { throw null; }

ConfigurationSection.cs will have an overload that takes the separator:

public partial class ConfigurationSection : Microsoft.Extensions.Configuration.IConfiguration, Microsoft.Extensions.Configuration.IConfigurationSection
{
        public ConfigurationSection(Microsoft.Extensions.Configuration.IConfigurationRoot root, string path,) { }
+       public ConfigurationSection(Microsoft.Extensions.Configuration.IConfigurationRoot root, string path, string separator = ":") { }
        public string? this[string key] { get { throw null; } set { } }
        ...
}

API Usage

Taken from this test, if we had some JSON config:

{
    "auths": {
        "http://google": {
            "uri": "https://www.google.es"
        },
        "http://microsoft": {
            "uri": "https://www.microsoft.es"
        }
    }
}

We could specify a different separator when loading it, e.g. a backtick (`), as that is different to the default separator of colon:

public class MyClass
{
    public Dictionary<string, OtherType> Auths { get; set; }
}

public class OtherType
{
    public string Uri { get; set; }
}

var config = new ConfigurationBuilder()
    .AddJsonFile("json_with_colons_in_keys.json", optional: false, reloadOnChange: true, separator: "`").Build();

var settings = new MyClass();

config.Bind(settings);
Assert.Equal("https://www.google.es", settings.Auths["http://google"].Uri);

Alternative Designs

No response

Risks

I can't see any risks. The default separator is colon (:), so the changes are backwards compatible.

The PR I did for this Issue has this new functionality, and all of the existing tests still pass, without modification.

[ghost]

Tagging subscribers to this area: @dotnet/area-extensions-configuration
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

As described in issue #42643, some users want to be able to have configuration keys that contain a colon character.

However, the colon character is the default separator for config keys, so if I had a config key named https://google.es, that would be translated to a composite key, the first being https, and the second being //google.es.

The purpose of this API suggestion is to enable users to specify their own separator character for configuration so that they can have keys containing colons, or any other character they wish.

API Proposal

The API changes are listed in the PR: https://github.com/dotnet/runtime/pull/66886/files#diff-ec34a58f9fd18f4d4fccabf4efc22dd555b9d32963b26419ed107c872b67356f .

We want overloads to provide the separator character:

public static partial class JsonConfigurationExtensions
{
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, Microsoft.Extensions.FileProviders.IFileProvider? provider, string path, bool optional, bool reloadOnChange) { throw null; }
==> public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, Microsoft.Extensions.FileProviders.IFileProvider? provider, string path, bool optional, bool reloadOnChange, string separator = ":") { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, System.Action<Microsoft.Extensions.Configuration.Json.JsonConfigurationSource>? configureSource) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path) { throw null; }
==> public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional) { throw null; }
==> public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, bool reloadOnChange) { throw null; }
==> public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, string path, bool optional, bool reloadOnChange, string separator) { throw null; }
    public static IConfigurationBuilder AddJsonStream(this IConfigurationBuilder builder, System.IO.Stream stream) { throw null; }
==> public static IConfigurationBuilder AddJsonStream(this IConfigurationBuilder builder, System.IO.Stream stream, string separator = ":") { throw null; }
}

ConfigurationPath.cs would have a new method to complement Combine, named CombineWith:

public static string Combine(params string[] pathSegments) { throw null; }
==> public static string CombineWith(string separator, params string[] pathSegments) { throw null; }

... and a new method to complement GetSectionKey, named GetSectionKeyWith:

public static string? GetSectionKey(string? path) { throw null; }
==> public static string? GetSectionKeyWith(string separator, string? path) { throw null; }

ConfigurationKeyComparer.cs will have an overloaded constructor to specify the separator:

public partial class ConfigurationKeyComparer : System.Collections.Generic.IComparer<string>
{
        public ConfigurationKeyComparer() { }
==> public ConfigurationKeyComparer(string separator) { }
        public static Microsoft.Extensions.Configuration.ConfigurationKeyComparer Instance { get { throw null; } }
        public static Microsoft.Extensions.Configuration.ConfigurationKeyComparer GetInstanceFor(string separator){ throw null; }
        public int Compare(string? x, string? y) { throw null; }
}

ConfigurationProvider.cs will have a new method to expose the key separator that it's using:

public abstract partial class ConfigurationProvider : Microsoft.Extensions.Configuration.IConfigurationProvider
{
    protected ConfigurationProvider() { }
==> public virtual string GetDelimiter() { throw null; }
    protected System.Collections.Generic.IDictionary<string, string?> Data { get { throw null; } set { } }
    public virtual System.Collections.Generic.IEnumerable<string> GetChildKeys(System.Collections.Generic.IEnumerable<string> earlierKeys, string? parentPath) { throw null; }
    public Microsoft.Extensions.Primitives.IChangeToken GetReloadToken() { throw null; }

ConfigurationSection.cs will have an overload that takes the separator:

public partial class ConfigurationSection : Microsoft.Extensions.Configuration.IConfiguration, Microsoft.Extensions.Configuration.IConfigurationSection
{
        public ConfigurationSection(Microsoft.Extensions.Configuration.IConfigurationRoot root, string path,) { }
==>  public ConfigurationSection(Microsoft.Extensions.Configuration.IConfigurationRoot root, string path, string separator = ":") { }
        public string? this[string key] { get { throw null; } set { } }
        ...
}

API Usage

Taken from this test, if we had some JSON config:

{
    "auths": {
        "http://google": {
            "uri": "https://www.google.es"
        },
        "http://microsoft": {
            "uri": "https://www.microsoft.es"
        }
    }
}

We could specify a different separator when loading it, e.g. a backtick (`), as that is different to the default separator of colon:

public class MyClass
{
    public Dictionary<string, OtherType> Auths { get; set; }
}

public class OtherType
{
    public string Uri { get; set; }
}

var config = new ConfigurationBuilder()
    .AddJsonFile("json_with_colons_in_keys.json", optional: false, reloadOnChange: true, separator: "`").Build();

var settings = new MyClass();

config.Bind(settings);
Assert.Equal("https://www.google.es", settings.Auths["http://google"].Uri);

Alternative Designs

No response

Risks

I can't see any risks. The default separator is colon (:), so the changes are backwards compatible.

The PR I did for this Issue has this new functionality, and all of the existing tests still pass, without modification.

Author:	SteveDunn
Assignees:	-
Labels:	api-suggestion, untriaged, area-Extensions-Configuration
Milestone:	-

[SteveDunn]

@maryamariyan - is there any update on this one?

[eerhardt]

This isn't planned for .NET 7 according to #64015. We can consider it in the future.

[SteveDunn]

This isn't planned for .NET 7 according to #64015. We can consider it in the future.

@eerhardt - I think it is. That page (#64015) links to #44517

Built-in configuration providers can do more robust key handling

That page links to Bug #42643

ConfigurationBuilder - Configuration.Json: Binding a Dictionary<string,string> error when Key contains an string url like http://www.google.es

That has a PR #66886.

That PR is closed until this API Proposal is accepted.

[eerhardt]

Thanks @SteveDunn, I had missed that one.

I'll move it back to 7.0 for now, but feature complete is in 2 weeks, so I'm not confident this will make it for 7.0.

[eerhardt]

The "Built-in configuration providers can do more robust key handling" feature #44517 has been moved out of 7.0.

We will consider this in a future release. Moving this issue out of the 7.0 milestone as well.

[SteveDunn]

Glad to see this one back.

[layomia]

Inlining a user scenario for this API proposal where it is desired for string dictionary keys - #42643.

[layomia]

Triage: moving to future given we're in a late period in the release & this would need API review.

[SteveDunn]

@layomia - is this being considered for .NET 9?

[julealgon]

The way this issue is named feels a bit misleading/reductive to me. Could someone consider a rename to something like:

[API Proposal]: Support configurable section delimiter in IConfiguration

?

[SteveDunn]

@layomia - is this being considered for .NET 9?

.net 10 then?