Fix the ICU time format conversion logic

[PopSlime]

Revise the ICU time format conversion logic to support all unquoted literal texts and the B and b pattern symbols.

Fix #103592

[PopSlime]

I'm new to such a big and complex project and not sure how to test these changes, so this is a draft at the moment. Looking forward to help from anyone.

Example affected patterns:

ICU pattern	Result (Old)	Result (New)
a नि h:mm	tt h:mm	tt नि h:mm
ཆུ་ཚོད་h:mm:ss a	h:mm:ss tt	ཆུ་ཚོད་h:mm:ss tt
ཆུ་ཚོད་ h སྐར་མ་ mm a	h mm tt	ཆུ་ཚོད་ h སྐར་མ་ mm tt
Bh:mm:ss	h:mm:ss	tth:mm:ss

More information can be found in the fixing issue.

[mkhamoyan]

@PopSlime thanks for submitting the PR.
Let's add a test case in https://github.com/dotnet/runtime/blob/main/src/libraries/System.Runtime/tests/System.Globalization.Tests/DateTimeFormatInfo/DateTimeFormatInfoLongTimePattern.cs to validate the implemented changes.

[mkhamoyan]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[PopSlime]

@mkhamoyan I'm not sure how to write the test for this because the retrieved time format is device-/platform-dependent.

Instead of testing against some specific patterns, I'm considering adding a test that fails if a 12-hour clock is used without an AM/PM designator in any pattern in any culture. Also we need another test for the unquoted literal texts, which I have no clues how to do yet.

By the way, short time patterns are also affected by this PR.

[mkhamoyan]

@PopSlime I was thinking something like below (same test case for shorttime patterns under here) would validate the change.

[Theory]
[InlineData("zh-TW")]
[InlineData("en-US")]
[InlineData("fr-FR")]
public void LongTimePattern_ValidateAMPMDesignators(string cultureName)
{
    var cultureInfo = new CultureInfo(cultureName);
    var date = DateTime.Today + TimeSpan.FromDays(10) + TimeSpan.FromMinutes(10);
    string formattedDateTime = date.ToString("T", cultureInfo);
    
    Assert.True(formattedDateTime.Contains(cultureInfo.DateTimeFormat.AMDesignator));
}

This should be consistent for all platforms and devices.
Now we see issue only in some android devices because of different CLDR versions.

[PopSlime]

@mkhamoyan Is it safe to assume those cultures use a 12-hour clock? fr-FR uses a 24-hour clock on my side.

[mkhamoyan]

@mkhamoyan Is it safe to assume those cultures use a 12-hour clock? fr-FR uses a 24-hour clock on my side.

My bad, fr-FR shouldn't be there.

[PopSlime]

How about this?

[Fact]
public void LongTimePattern_VerifyTimePatterns()
{
    foreach (var culture in CultureInfo.GetCultures(CultureTypes.AllCultures))
    {
        var pattern = culture.DateTimeFormat.LongTimePattern;
        var segments = pattern.Split('\'');
        bool use12Hour = false;
        bool useAMPM = false;
        for (var i = 0; i < segments.Length; i += 2)
        {
            var segment = segments[i];
            use12Hour |= segment.Contains('h', StringComparison.Ordinal);
            useAMPM |= segment.Contains('t', StringComparison.Ordinal);
        }
        Assert.True(!use12Hour || useAMPM, $"Bad time pattern for culture {culture.Name}: '{pattern}'");
    }
}

By the way, this fails on my PC (Windows) with mi: 'h:mm:ss' and mi-NZ: 'h:mm:ss'. Should we address this in another issue?

[mkhamoyan]

How about this?

Looks good to me.

By the way, this fails on my PC (Windows) with mi: 'h:mm:ss' and mi-NZ: 'h:mm:ss'. Should we address this in another issue?

Yes , we should file an issue for that.
Let's push in the PR and check how it behaves on other platforms.

[PopSlime]

@dotnet-policy-service agree

[tarekgh]

By the way, this fails on my PC (Windows) with mi: 'h:mm:ss' and mi-NZ: 'h:mm:ss'. Should we address this in another issue?
Yes , we should file an issue for that.

I don't think we can fix that as the patterns are picked from Windows. I suggest you mark your new test methods with the attribute and see if this can help?

runtime/src/libraries/System.Runtime/tests/System.Globalization.Tests/DateTimeFormatInfo/DateTimeFormatInfoLongTimePattern.cs

Line 264 in 7934e64

[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsIcuGlobalization))]

[PopSlime]

I'll do that, but I also want to see if similar issues are occurring on other platforms, so let's wait for these checks to be done.

[tarekgh]

this fails on my PC (Windows) with mi: 'h:mm:ss' and mi-NZ: 'h:mm:ss'.

I am trying on my machine, and I am not getting the same results you are getting.

mi-NZ .. Maori (New Zealand) .. h:mm tt .. h:mm:ss tt
mi .. Maori .. h:mm tt .. h:mm:ss tt

running code like:

            var ci = CultureInfo.GetCultureInfo("mi-NZ");
            Console.WriteLine($"{ci.Name} .. {ci.EnglishName} .. {ci.DateTimeFormat.ShortTimePattern} .. {ci.DateTimeFormat.LongTimePattern}");
            ci = CultureInfo.GetCultureInfo("mi");
            Console.WriteLine($"{ci.Name} .. {ci.EnglishName} .. {ci.DateTimeFormat.ShortTimePattern} .. {ci.DateTimeFormat.LongTimePattern}");

What Windows version do you have? I am wondering how you get this result?

[PopSlime]

It seems that the version of the CLDR data installed on my PC is below 36. The time format for mi was fixed in unicode-org/cldr#104.

• Operating system: Windows 10
• Version number: 22H2
• Internal version: 19045.4529

[tarekgh]

ok, then the attribute

runtime/src/libraries/System.Runtime/tests/System.Globalization.Tests/DateTimeFormatInfo/DateTimeFormatInfoLongTimePattern.cs

Line 264 in 7934e64

[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsIcuGlobalization))]

is not going to help. You may try using something like

runtime/src/libraries/System.Runtime/tests/System.Globalization.Tests/NumberFormatInfo/NumberFormatInfoData.cs

Line 10 in 2a779ab

if (PlatformDetection.WindowsVersion >= 10 || PlatformDetection.ICUVersion.Major >= 55 || PlatformDetection.IsHybridGlobalizationOnApplePlatform)

to avoid the failure.

I meant using check like if (PlatformDetection.ICUVersion.Major >= 68) .

[ilonatommy]

@PopSlime the tests are still failing. For now, can you exclude running this test with hype globalization? or special case fr-CA when encountering it? @ilonatommy @mkhamoyan do you have any better suggestion? @PopSlime already logged issue to tracking fixing fr-CA formats.

Current behavior was expected and other HybridGlobalization tests are prepared for that. I am looking if we can provide a generic fix. For now this test can be blocked with PlatformDetection.IsNotHybridGlobalizationOnBrowser. The fix can get in in a separate PR

[mkhamoyan]

/azp run runtime-ioslike,runtime-ioslikesimulator,runtime-maccatalyst

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[mkhamoyan]

Looks good to me.
Let's wait for ios pipelines and then we can merge.

[mkhamoyan]

/azp run runtime-ioslike,runtime-ioslikesimulator,runtime-maccatalyst

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[PopSlime]

/azp run runtime

[azure-pipelines]

Commenter does not have sufficient privileges for PR 103681 in repo dotnet/runtime

[ilonatommy]

/azp run runtime

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).