ManyConcurrentAddsTakes_ForceContentionWithToArray fails intermittently on osx-arm64

[jkotas]

Assert.DoesNotContain() Failure
Found:    0
In value: Int32[] [0]
   at System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(Double seconds) + 0x110
   at System.Collections.Concurrent!<BaseAddress>+0x9fa6bc
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0x148

 {
    "ErrorMessage" : "System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray",
    "BuildRetry": false
 }

Failed in #76499. Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-76499-merge-3c4c964d671349bdbd/System.Collections.Concurrent.Tests/1/console.b1cceddc.log?helixlogtype=result

Report

Build	Definition	Test	Pull Request
74353	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
74067	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
73941	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77934
73002	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
72557	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
71724	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithGetEnumerator
70399	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77770
67757	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
67272	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
67179	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
66666	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#76630
64951	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)	#77522

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	5	12

[jkotas]

Also failed in #75421 (comment)

[jkotas]

cc @VSadov @filipnavara

[filipnavara]

I believe the failure is genuine but I was not able to reliably reproduce it on my machine yet. Should we mark the test with ActiveIssue until we find the root cause?

[jkotas]

"Known Build Error" label turns on automation to monitor and collect statistics about the failure. Let's see how often it fails. If it fails too often, we are going to disable it.

[filipnavara]

It reproduces in less than a minute when running the single test in loop on M1.

[filipnavara]

In fact, it reproduces under regular CoreCLR as well:

./.dotnet/dotnet /Users/filipnavara/Projects/ConcurrentDict/bin/Debug/net7.0/ConcurrentDict.dll 
FAIL
Unhandled exception. System.Exception: Exception of type 'System.Exception' was thrown.
   at Assert.DoesNotContain(Int32 value, Int32[] array) in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 92
   at BringUpTest.Test() in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 50
   at BringUpTest.Main() in /Users/filipnavara/Projects/ConcurrentDict/ConcurrentDict.cs:line 18

ConcurrentDict.cs:

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Diagnostics;
using System.Text;
using System.Collections.Concurrent;
using System.Threading.Tasks;

public class BringUpTest
{
    const int Pass = 100;
    const int Fail = -1;

    public static int Main()
    {
        for (int i = 0; i < 200; i++)
            Test();
        return Pass;
    }

    public static void Test()
    {
        IProducerConsumerCollection<int> c = new ConcurrentQueue<int>();
        const int MaxCount = 4;

        DateTime end = DateTime.UtcNow + TimeSpan.FromSeconds(1);

        Task addsTakes = Task.Factory.StartNew(() =>
        {
            while (DateTime.UtcNow < end)
            {
                for (int i = 1; i <= MaxCount; i++)
                {
                    Assert.True(c.TryAdd(i));
                }
                for (int i = 1; i <= MaxCount; i++)
                {
                    int item;
                    Assert.True(c.TryTake(out item));
                    Assert.InRange(item, 1, MaxCount);
                }
            }
        });

        while (DateTime.UtcNow < end)
        {
            int[] arr = c.ToArray();
            Assert.InRange(arr.Length, 0, MaxCount);
            Assert.DoesNotContain(0, arr); // make sure we didn't get default(T)
        }

        addsTakes.GetAwaiter().GetResult();
        Assert.Equal(0, c.Count);
    }
}

class Assert
{
    public static void Equal(int expected, int actual)
    {
        if (expected != actual)
        {
            Console.WriteLine($"{actual} != {expected}");
            throw new Exception();
        }
    }

    public static void True(bool actual)
    {
        if (!actual)
        {
            Console.WriteLine($"!{actual}");
            throw new Exception();
        }
    }

    public static void InRange(int actual, int min, int max)
    {
        if (actual < min || actual > max)
        {
            Console.WriteLine($"{actual} not in {min}-{max}");
            throw new Exception();
        }
    }

    public static void DoesNotContain(int value, int[] array)
    {
        if (Array.Exists(array, e => e == value))
        {
            Console.WriteLine($"FAIL");
            throw new Exception();
        }
    }
}

[ghost]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Assert.DoesNotContain() Failure
Found:    0
In value: Int32[] [0]
   at System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(Double seconds) + 0x110
   at System.Collections.Concurrent!<BaseAddress>+0x9fa6bc
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0x148

 {
    "ErrorMessage" : "System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray",
    "BuildRetry": false
 }

Failed in #76499. Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-76499-merge-3c4c964d671349bdbd/System.Collections.Concurrent.Tests/1/console.b1cceddc.log?helixlogtype=result

Report

Build	Definition	Test
38234	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)
38068	dotnet/runtime	System.Collections.Concurrent.Tests.ConcurrentQueueTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(seconds: 1)

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	2	2

Author:	jkotas
Assignees:	-
Labels:	arch-arm64, area-System.Collections, os-mac-os-x, tenet-reliability, Known Build Error
Milestone:	8.0.0

[filipnavara]

Note that the failure is also reproducible on 7.0 RC1 so it may be appropriate to change the milestone.

[akoeplinger]

Might be related: #76141

[filipnavara]

Might be related: #76141

Likely not, I already checked the existing issues. This one is ConcurrentQueue and the other one is ConcurrentStack.

[VSadov]

Looks like an ordering issue on arm64. I will take a look

[VSadov]

I am able to reproduce this on OSX-arm64-CoreCLR

[VSadov]

PR with a fix: #78142

I am running the repro in multiple processes for about 20 min now with no failures. It typically fails in under one minute.