Experiment Report: Using BenchmarkDotNet to find methods that can cause longe GC pause times

[adamsitnik]

I was asked by @jkotas if it would be possible to run a simple experiment using BenchmarkDotNet and find methods that can cause long GC pause times. Examples: https://github.com/dotnet/coreclr/issues/27106 https://github.com/dotnet/coreclr/issues/27683

I took the idea from following @jkotas example:

static void Main()
{
    new Thread(() =>
    {
        var src = new int[10_000_000];
        for (;;) Array.Sort(src);
    }).Start();
    for (;;) { Console.WriteLine("GC start"); GC.Collect(); Console.WriteLine("GC end"); Thread.Sleep(1); }
}

And hacked BenchmarkDotNet to:

• start a new thread that runs the benchmark code with a heuristic that never stops running and does not report the results
• start a new benchmark on the startup thread that calls GC.Collect(); Thread.Sleep(1); and reports the time it takes to execute it

So more or less the reported times were GC pause times.

More details: dotnet/BenchmarkDotNet@fb57a14

To verify that it works I've used the following example:

public class SimpleTest
{
    private int[] array = Enumerable.Range(0, 1000).ToArray();

    [Benchmark]
    public void Sort() => Array.Sort(array);

    [Benchmark]
    public void Reverse() => Array.Reverse(array);

    [Benchmark]
    public void BubbleSort()
    {
        var arr = array;
        int temp;
        for (int write = 0; write < arr.Length; write++)
        {
            for (int sort = 0; sort < arr.Length - 1; sort++)
            {
                if (arr[sort] > arr[sort + 1])
                {
                    temp = arr[sort + 1];
                    arr[sort + 1] = arr[sort];
                    arr[sort] = temp;
                }
            }
        }
    }
}

Which has produced the following results for 2.1 (where the Array.Sort still calls native sort implementation and causes the problem):

BenchmarkDotNet=v0.12.0.20191121-develop, OS=Windows 10.0.18363
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-alpha1-013817
[Host] : .NET Core 2.1.13 (CoreCLR 4.6.28008.01, CoreFX 4.6.28008.01), X64 RyuJIT
Job-JTLBSN : .NET Core 2.1.13 (CoreCLR 4.6.28008.01, CoreFX 4.6.28008.01), X64 RyuJIT

IterationTime=250.0000 ms MaxIterationCount=5 MinIterationCount=3
WarmupCount=1

Method	Mean	Error	StdDev
Sort	6,432.459 ms	8,690.5440 ms	2,256.9067 ms
Reverse	2.047 ms	0.2790 ms	0.0724 ms
BubbleSort	1.999 ms	0.0237 ms	0.0013 ms

So as you can see the Reverse and BubbleSort despite doing more work for already sorted array report an order of magnitude less time than Sort.

[adamsitnik]

In the next step, I've run all the benchmarks we have in this repo and wrote a small parser that sorted them descending by average reported time.

The full results can be obtained here: https://gist.github.com/adamsitnik/e31284814b44bcb3078825e0018a1ea3#file-windows_results-txt

Top:

avg(ms) max(ms) benchmark name
1053.50 4187.44 Double.Cbrt
 645.41 2096.09 Single.Asinh
 573.59 1578.11 StackWalk.Walk
 536.83 1265.10 Single.Cbrt
 484.42 1909.27 Double.ScaleB
 366.69 1392.27 Single.Log2
 303.23  842.58 Double.Asinh
 283.50  729.35 Double.Sinh
 270.24  597.75 Double.Atan2
 244.80  809.71 Single.Atanh
 221.97  453.83 Double.Acosh
 219.49  487.13 Double.Cos
 213.72  351.59 Single.Acosh
 205.71  736.87 Double.Atanh
 176.60  312.45 CastingPerf.CheckArrayIsInterfaceNo
 172.84  232.45 Single.Cosh
 172.26  329.22 Single.ScaleB
 151.65  177.71 CastingPerf.CheckArrayIsNonvariantGenericInterfaceNo
 149.82  313.83 Double.Cosh
 148.71  250.00 Double.Pow
 144.82  277.03 Double.Acos
 143.20  299.31 Single.Pow
 141.61  267.00 Single.Asin
 136.72  296.02 CastingPerf.ObjObjIsFoo
 132.12  174.78 CastingPerf.ObjrefValueTypeObj
 130.95  165.99 EnumPerf.ObjectGetType
 129.63  444.53 Single.Tanh
 125.79  257.73 CastingPerf.CheckArrayIsNonvariantGenericInterface
 125.62  254.93 Double.Atan
 119.54  208.26 CastingPerf.FooObjIsFoo
 113.96  129.86 CastingPerf.IFooObjIsDescendantOfIFoo
 111.42  146.47 CastingPerf.IFooObjIsIFooInterAlia
 111.22  164.04 CastingPerf.CheckListIsVariantGenericInterface
 110.59  175.14 Double.Log
 108.62  140.54 CtorDefaultSize<Int32>.ImmutableHashSet
 108.19  191.36 CtorDefaultSize<Int32>.ImmutableList
 105.86  136.60 CastingPerf.CheckArrayIsArrayByVariance
 104.87  147.38 Double.Tanh
 104.77  154.65 CastingPerf.AssignArrayElementByVariance
  99.81  238.94 Single.Atan2
  99.32  143.65 Single.Tan
  99.23  124.81 CastingPerf.FooObjCastIfIsa
  98.55  131.12 Single.Sinh
  96.77  171.64 Single.Cos
  95.48  235.33 Single.Atan
  94.92  175.73 CtorDefaultSize<Int32>.ImmutableSortedSet
  94.55  155.46 CtorDefaultSize<Int32>.ImmutableQueue
  93.37  124.96 IndexerSet<String>.IList
  93.23  129.64 CastingPerf.ScalarValueTypeObj
  92.16  136.81 Single.Exp
  90.44  148.36 Double.Sin
  89.68  318.98 Single.Acos
  89.18  155.97 CastingPerf.CheckArrayIsVariantGenericInterfaceNo
  88.48   88.96 Perf_Timer.ShortScheduleAndDisposeWithFiringTimers
  88.16  178.82 Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace
  84.76  135.37 CastingPerf.IntObj
  83.84  102.90 Double.Exp
  83.61  163.98 CastingPerf.IFooObjIsDescendantOfIFoo
  83.21  136.63 EnumPerf.ObjectGetTypeNoBoxing
  82.88  191.45 Perf_CultureInfo.GetInvariantCulture
  82.87  109.11 LowLevelPerf.TypeReflectionObjectGetType
  82.71  101.22 CtorDefaultSize<Int32>.ImmutableStack
  82.32  104.15 Trap.Trap
  82.21  175.72 Single.Log10
  81.64  117.15 CastingPerf.IFooObjIsIFoo
  78.92  186.64 CastingPerf.CheckArrayIsInterfaceYes
  77.24   81.85 Perf_Timer.ScheduleManyThenDisposeMany
  74.49   95.36 Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace
  73.15  102.08 Double.ILogB
  70.71  109.34 CastingPerf.FooObjIsFoo2
  69.73  117.34 Single.Sin
  69.54  117.12 CtorDefaultSize<Int32>.ImmutableDictionary
  67.63   89.25 Simpsn.Simpsn
  66.80   97.24 CastingPerf.IFooObjIsIFoo
  66.01  109.33 LowLevelPerf.TypeReflectionArrayGetType
  65.56  271.93 Double.Log2
  65.31   90.58 NewtE.NewtE
  64.83  122.36 CastingPerf.FooObjIsDescendant
  63.68   85.91 CastingPerf.IFooObjIsIFooInterAlia
  63.30   92.92 CtorDefaultSize<Int32>.ImmutableSortedDictionary
  62.89   85.52 CastingPerf.FooObjIsDescendant
  57.37   71.22 ReadOnlyMemory<Char>.Pin
  55.87  105.43 CtorDefaultSize<String>.ImmutableHashSet
  53.66   80.81 ReadOnlyMemory<Byte>.Pin

[adamsitnik]

I've taken a brief look at the results to verify that they are correct.

The worst is Double.Cbrt which calls Math.Cbrt in a tight loop:

performance/src/benchmarks/micro/coreclr/Math/Functions/Double/Cbrt.cs

Lines 24 to 28 in 64e1e28

	for (var iteration = 0; iteration < MathTests.Iterations; iteration++)
	{
	result += Math.Cbrt(value);
	value += cbrtDelta;
	}

And internally performs an fcall to COMDouble.Cbrt

https://github.com/dotnet/coreclr/blob/a49296e266ae9aa0bee760f2fcf51d3497ba818d/src/classlibnative/float/floatdouble.cpp#L121-L124

Which means that calling some of the Math.* APIs in a tight loop might produce long GC pauses.

[adamsitnik]

@jkotas please let me know if you want some more information or confirmation. I am not going to investigate all the methods right now because you most probably know how they are implemented and can quickly tell where the problems are and what is by design ;)

[tannergooding]

We currently have an issue with many of the math functions that different platform/architecture combinations have different implementations and so compute different results and can have wildly different performance characteristics.

So it might be worth also getting results on Linux and macOS for these to help compare/contrast.

[stephentoub]

Nice experiment and analysis, @adamsitnik.

[VSadov]

Very nice!

[VSadov]

So, looking at the fact that calling Asinh in a loop can cause 2sec GC pause.
Is that because FCALL invocation is not a GC safepoint? (I kind of assumed that it is similar to a regular call in that sense...)

I can easily imagine a loop doing just math and trivial FCALLs - more math, typesystem/VM helpers like casts and threadstatics, GetTickCount, etc ...

@jkotas - How is that actually handled?

[jkotas]

FCALLs are supposed to poll for GC to avoid GC starvation. Many FCALLs are missing the polls.

We can fix this by either adding the GC polls to all FCALLs that we find problematic; or teaching the JIT to generate the GC poll around every FCALL call (ie use the same GCPoll logic as SuppressGCTransition that was introduced recently).

[jkotas]

I have opened https://github.com/dotnet/coreclr/issues/27931 and
https://github.com/dotnet/coreclr/issues/27930 on the worst offenders.

[adamsitnik]

I've run the experiment for Ubuntu 18.04 using exactly same hardware (my PC with dual boot)

Full results: https://gist.github.com/adamsitnik/e31284814b44bcb3078825e0018a1ea3#file-linux_results-txt

Top 100:

 329.60  560.94 StackWalk.Walk
 119.10  130.06 CtorDefaultSize<String>.ConcurrentBag
 109.36  109.88 CtorDefaultSize<Int32>.ConcurrentBag
  79.73   85.14 Perf_Timer.ShortScheduleAndDisposeWithFiringTimers
  69.34   71.48 Perf_Timer.ScheduleManyThenDisposeMany
  48.39   72.92 Double.Pow
  37.15   53.83 Perf_DateTime.GetUtcNow
  36.46   42.97 Perf_ThreadStatic.GetThreadStatic
  32.29   79.36 Single.Cbrt
  28.56   34.57 Double.Cosh
  27.51   49.37 Double.Cbrt
  24.92   31.19 Single.Sinh
  24.14   30.47 Double.Sinh
  23.88   27.73 Double.Atan
  23.77   29.26 Double.Cos
  23.68   27.30 Double.Acosh
  23.17   34.92 Perf_Array.ArrayResize
  23.06   30.95 Double.Log10
  22.03   25.54 Perf_BinaryFormatter.DeserializeLargeList
  22.02   28.53 Perf_Object.GetType
  20.81   25.45 Double.Tan
  20.31   25.93 Double.Atan2
  20.27   37.16 Double.Log
  20.13   24.90 EnumPerf.ObjectGetTypeNoBoxing
  19.55   21.59 EnumPerf.ObjectGetType
  19.16   22.69 Perf_Interlocked.CompareExchange_object_NoMatch
  18.91   22.75 CastingPerf.CheckArrayIsNonvariantGenericInterface
  18.88   23.87 Single.Cosh
  18.40   24.30 Single.Tan
  18.21   21.06 Double.Exp
  17.14   21.03 Double.Acos
  16.60   21.80 Perf_CultureInfo.GetInvariantCulture
  15.93   17.54 Perf_Interlocked.CompareExchange_object_Match
  15.88   17.79 Double.Sin
  15.54   23.74 Perf_Type.GetTypeFromHandle
  15.29   20.86 CtorDefaultSize<Int32>.ImmutableSortedSet
  15.10   20.79 Double.Asinh
  15.06   21.59 Perf_Interlocked.CompareExchange_long
  14.89   18.67 Single.Pow
  14.78   17.51 Double.Asin
  14.44   17.52 Perf_Interlocked.CompareExchange_int
  14.33   19.32 CtorDefaultSize<Int32>.ImmutableStack
  14.16   16.88 CastingPerf.ScalarValueTypeObj
  13.89   18.05 CtorDefaultSize<Int32>.ImmutableDictionary
  13.75   18.11 CastingPerf.FooObjIsFoo
  13.74   17.39 CastingPerf.IFooObjIsIFoo
  13.69   17.37 CastingPerf.ObjrefValueTypeObj
  13.64   16.48 Single.Asinh
  13.60   16.52 CastingPerf.IFooObjIsIFooInterAlia
  13.54   15.56 CastingPerf.IntObj
  13.40   17.04 Single.Exp
  13.33   16.30 CtorDefaultSize<Int32>.ImmutableHashSet
  13.30   15.46 CastingPerf.IFooObjIsDescendantOfIFoo
  13.28   15.43 Single.Atan2
  13.16   14.17 CtorDefaultSize<Int32>.ImmutableQueue
  13.03   15.14 CastingPerf.AssignArrayElementByVariance
  12.98   14.78 Trap.Trap
  12.93   20.61 Double.Atanh
  12.75   16.13 CastingPerf.FooObjIsDescendant
  12.69   14.96 CtorDefaultSize<Int32>.ImmutableList
  12.64   14.78 CastingPerf.FooObjIsFoo2
  12.60   14.59 CastingPerf.CheckArrayIsNonvariantGenericInterfaceNo
  12.58   14.42 Double.Tanh
  12.49   15.32 CtorDefaultSize<Int32>.ImmutableSortedDictionary
  12.40   15.89 CastingPerf.ObjObjIsFoo
  12.19   13.85 Single.Atanh
  12.02   12.22 Perf_Timer.SynchronousContention
  11.81   14.01 CastingPerf.CheckArrayIsArrayByVariance
  11.56   12.17 CastingPerf.CheckArrayIsInterfaceNo
  11.56   15.30 CastingPerf.CheckListIsVariantGenericInterface
  11.39   13.75 Single.Tanh
  10.98   16.05 BenchMk2.BenchMk2
  10.97   12.79 CastingPerf.FooObjCastIfIsa
  10.92   11.68 CtorDefaultSize<String>.ImmutableHashSet
  10.65   11.57 Perf_Timer.AsynchronousContention
  10.43   12.31 Single.Acosh
  10.27   10.64 Perf_Regex_Cache.IsMatch_Multithreading
  10.14   11.65 Simpsn.Simpsn
  10.14   11.35 LowLevelPerf.TypeReflectionArrayGetType
  10.00   11.18 LowLevelPerf.TypeReflectionObjectGetType
   9.66   11.21 Perf_Regex_Cache.IsMatch_Multithreading
   9.60   10.26 Perf_Regex_Cache.IsMatch_Multithreading
   9.53    9.80 Perf_Regex_Cache.IsMatch_Multithreading
   9.48   10.41 Perf_Strings.WriteStringsUtf16
   9.40   11.66 CastingPerf.CheckArrayIsVariantGenericInterfaceNo
   9.34   12.34 CastingPerf.CheckArrayIsInterfaceYes
   9.27   13.08 BenchMrk.BenchMrk
   9.27    9.71 Perf_Regex_Cache.IsMatch
   9.17   11.38 CastingPerf.IFooObjIsIFoo
   9.15    9.82 NewtE.NewtE
   9.11   10.01 Perf_Regex_Cache.IsMatch_Multithreading
   9.03    9.37 Perf_Strings.WriteStringsUtf8
   8.96    9.97 Perf_Strings.WriteStringsUtf16
   8.96   10.09 Double.Log2
   8.95   10.27 CtorDefaultSize<String>.ImmutableSortedSet
   8.92   10.36 Perf_Strings.WriteStringsUtf8
   8.92   11.69 CtorDefaultSize<Int32>.ImmutableArray
   8.86    9.23 Perf_Strings.WriteStringsUtf16
   8.85    9.63 Perf_Strings.WriteStringsUtf8

[adamsitnik]

Linux vs Windows comparison, sorted descending by sum of medians: https://gist.github.com/adamsitnik/e31284814b44bcb3078825e0018a1ea3#file-linux_vs_windows-txt

top 100:

win(ms) lin(ms) benchmark name
 421.88  357.62 PerfLabTests.StackWalk.Walk
 218.91   23.39 System.MathBenchmarks.Single.Cbrt
 213.62   24.70 System.MathBenchmarks.Double.Sinh
 202.48   19.95 System.MathBenchmarks.Double.Atan2
 177.22   14.50 System.MathBenchmarks.Single.Pow
 164.61   23.57 System.MathBenchmarks.Double.Acosh
 168.35   19.78 System.MathBenchmarks.Single.Cosh
 174.66   10.76 System.MathBenchmarks.Single.Acosh
 141.59   43.80 System.MathBenchmarks.Double.Pow
 176.94    8.39 System.MathBenchmarks.Single.ScaleB
 161.11   12.62 PerfLabTests.CastingPerf.CheckArrayIsNonvariantGenericInterfaceNo
 159.82   13.69 System.MathBenchmarks.Double.Asinh
  50.68  119.30 System.Collections.CtorDefaultSize<String>.ConcurrentBag
 144.92   24.13 System.MathBenchmarks.Double.Cbrt
 158.82    8.48 System.MathBenchmarks.Double.ScaleB
  88.41   78.28 System.Threading.Tests.Perf_Timer.ShortScheduleAndDisposeWithFiringTimers
  46.84  109.35 System.Collections.CtorDefaultSize<Int32>.ConcurrentBag
 147.40    5.70 System.MathBenchmarks.Single.Asin
 120.00   31.24 System.MathBenchmarks.Double.Cosh
 117.45   29.44 System.MathBenchmarks.Single.Sinh
  76.18   68.34 System.Threading.Tests.Perf_Timer.ScheduleManyThenDisposeMany
 138.43    5.83 System.MathBenchmarks.Single.Log2
 130.16   13.36 System.Collections.CtorDefaultSize<Int32>.ImmutableHashSet
 130.61   12.64 System.MathBenchmarks.Single.Atanh
 124.14   18.29 PerfLabTests.CastingPerf.CheckArrayIsNonvariantGenericInterface
 124.68   15.83 System.MathBenchmarks.Double.Acos
 127.16   13.06 System.MathBenchmarks.Double.Tanh
 118.12   20.36 PerfLabTests.EnumPerf.ObjectGetType
 124.70   12.86 PerfLabTests.CastingPerf.AssignArrayElementByVariance
 123.91   12.99 PerfLabTests.CastingPerf.ObjrefValueTypeObj
 124.85   11.57 PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo
 113.00   22.99 System.MathBenchmarks.Double.Atan
 112.43   19.57 System.MathBenchmarks.Single.Tan
 113.08   13.65 System.MathBenchmarks.Single.Asinh
 111.96   14.09 PerfLabTests.CastingPerf.IFooObjIsDescendantOfIFoo
 109.19   11.60 PerfLabTests.CastingPerf.ObjObjIsFoo
 108.54   10.74 PerfLabTests.CastingPerf.FooObjCastIfIsa
 113.30    5.44 System.MathBenchmarks.Single.Cos
 103.17   13.83 PerfLabTests.CastingPerf.IFooObjIsIFooInterAlia
 104.38   10.62 PerfLabTests.CastingPerf.CheckListIsVariantGenericInterface
 103.97   10.17 PerfLabTests.LowLevelPerf.TypeReflectionObjectGetType
 101.48   12.53 PerfLabTests.CastingPerf.FooObjIsFoo
  99.32   13.89 System.MathBenchmarks.Single.Exp
  98.82   13.92 PerfLabTests.CastingPerf.ScalarValueTypeObj
  97.54   13.17 System.Collections.CtorDefaultSize<Int32>.ImmutableList
  86.29   23.80 System.MathBenchmarks.Double.Cos
  96.32   11.50 PerfLabTests.CastingPerf.CheckArrayIsArrayByVariance
  92.94   13.43 System.Collections.CtorDefaultSize<Int32>.ImmutableDictionary
  85.80   19.26 System.MathBenchmarks.Double.Log
  93.65    8.49 System.Collections.IndexerSet<String>.IList(Size: 512)
  87.86   13.45 System.Collections.CtorDefaultSize<Int32>.ImmutableStack
  85.83   13.74 PerfLabTests.CastingPerf.IntObj
  81.23   17.77 System.MathBenchmarks.Double.Exp
  83.25   13.93 System.Collections.CtorDefaultSize<Int32>.ImmutableQueue
  79.34   15.99 System.MathBenchmarks.Double.Sin
  79.69   14.71 System.MathBenchmarks.Single.Atan2
  83.20    7.61 System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)
  69.93   18.25 PerfLabTests.EnumPerf.ObjectGetTypeNoBoxing
  74.20   13.12 Benchstone.BenchF.Trap.Test
  79.96    6.04 System.MathBenchmarks.Double.ILogB
  77.30    7.99 PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsDescendantOfIFoo
  66.37   17.24 System.Collections.CtorDefaultSize<Int32>.ImmutableSortedSet
  76.28    7.33 System.MathBenchmarks.Single.Log10
  70.27    9.80 PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo
  62.46   13.82 PerfLabTests.CastingPerf.IFooObjIsIFoo
  66.19    9.87 Benchstone.BenchF.Simpsn.Test
  69.24    6.57 System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)
  70.15    4.64 System.MathBenchmarks.Single.Sin
  65.59    9.10 Benchstone.BenchF.NewtE.Test
  62.09   12.60 PerfLabTests.CastingPerf.FooObjIsFoo2
  66.01    8.06 PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia
  39.30   34.53 System.Tests.Perf_Array.ArrayResize
  66.31    5.97 PerfLabTests.CastingPerf2.CastingPerf.FooObjIsDescendant
  64.35    7.36 System.MathBenchmarks.Single.Atan
  62.31    9.02 PerfLabTests.CastingPerf.CheckArrayIsInterfaceYes
  52.06   16.80 System.Globalization.Tests.Perf_CultureInfo.GetInvariantCulture
  54.62   13.89 System.Collections.CtorDefaultSize<Int32>.ImmutableSortedDictionary
  57.26    9.12 PerfLabTests.CastingPerf.CheckArrayIsVariantGenericInterfaceNo
  58.20    7.84 System.Memory.ReadOnlyMemory<Char>.Pin(Size: 512)
  54.58    9.81 PerfLabTests.LowLevelPerf.TypeReflectionArrayGetType
  52.31   11.72 System.MathBenchmarks.Single.Tanh
  56.24    5.91 System.Globalization.Tests.Perf_CultureInfo.GetCurrentCulture
  49.67   11.48 System.MathBenchmarks.Double.Atanh
  53.49    7.03 System.Tests.Perf_TimeSpan.ctor_int_int_int
  51.65    8.29 Benchstone.BenchF.BenchMrk.Test
  48.82    9.62 Benchstone.BenchF.BenchMk2.Test
  46.84   11.54 System.Collections.CtorDefaultSize<String>.ImmutableHashSet
  45.85   12.31 PerfLabTests.CastingPerf.FooObjIsDescendant
  50.69    6.71 System.Collections.CtorDefaultSize<String>.ImmutableSortedDictionary
  50.69    5.85 PerfLabTests.CastingPerf.CheckArrayIsVariantGenericInterfaceReflection
  51.95    4.04 PerfLabTests.CastingPerf.CheckIsInstAnyIsInterfaceNo
  47.72    7.54 System.Memory.ReadOnlyMemory<Byte>.Pin(Size: 512)
  18.28   36.95 System.Tests.Perf_DateTime.GetUtcNow
  32.74   21.97 System.MathBenchmarks.Double.Log10
  46.40    8.10 System.Memory.Memory<Byte>.Pin(Size: 512)
  30.16   21.57 System.Runtime.Serialization.Formatters.Tests.Perf_BinaryFormatter.DeserializeLargeList
  46.20    5.13 System.MathBenchmarks.Single.ILogB
  41.39    9.35 System.Collections.CtorDefaultSize<String>.ImmutableSortedSet
  44.52    6.22 System.MathBenchmarks.Single.Acos
  29.99   19.83 System.MathBenchmarks.Double.Tan

[adamsitnik]

I've reported an issue I've hit when I was running the benchmarks: dotnet/runtime#314

@jkotas pleas let me know if you need any more data.

[jkotas]

Thanks! We have issues opened on the top problems found by this experiment.