Switch to the dotnet/performance repo

[adamsitnik]

The benchmarks were ported to BenchmarkDotNet and added to the dotnet/performance repository.

The new workflow is described at https://github.com/dotnet/performance/blob/master/docs/benchmarking-workflow-corefx.md

The benchmarks can be found in https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx

The BenchView link: https://benchview/trendline?jobgroup=.NET%20Performance&branchId=762&jobgroup=.NET%20Performance&branchId=762&jobtype=rolling&count=200&rtids=[1713]&archids=[9,4]&mpids=[1292]&cfgids=[3394,3391,3392,3393]&

Details:

• System.Collections benchmarks were rewritten in Collections benchmarks rewrite performance#92 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Collections
  • System.Collections.PerformanceTests
  • System.Collections.Concurrent.PerformanceTests
  • System.Collections.NonGeneric.PerformanceTests
• System.ComponentModel.TypeConverter.PerformanceTests benchmarks were ported and fixed (they contained Assert that was included in the measurements and was taking 6/7 of the reported time) in port TypeDescriptor benchmarks performance#83 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.ComponentModel.TypeConverter
• System.Console.PerformanceTests benchmarks were ported in Port Console benchmarks performance#85 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Console
• System.Diagnostics.Process.PerformanceTests were rewritten (they were disabled for more than a year due to design issues, they were spawning houndreds of processes) in System.Diagnostics.Process performance#111 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Diagnostics
• System.Drawing benchmarks were ported and fixed (they were claiming to test bmp, jpg and png but were testing only bmp) in Port System.Drawing benchmarks from CoreFX performance#350 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Drawing
  • System.Drawing.Common.PerformanceTests
  • System.Drawing.Primitives.PerformanceTests
• System.Globalization.PerformanceTests benchmarks were ported in Globalization performance#115 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Globalization
• System.IO.PerformanceTests benchmarks which contained single type got ported and now are available at https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/corefx/System.Runtime.Extensions/Perf.StreamWriter.cs
• System.IO.Compression.PerformanceTests benchmarks were ported and fixed (they had invalid OS-dependent IDs which made OS comparison impossible) in System.IO.Compression performance#105 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.IO.Compression
  • System.IO.Compression.PerformanceTests
  • System.IO.Compression.Brotli.PerformanceTests
• System.IO.FileSystem.PerformanceTests benchmarks were ported and partially redesigned (1 benchmark was testing few different things based on arguments) in File system performance#107 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.IO.FileSystem
• System.IO.MemoryMappedFiles.PerformanceTests benchmarks were ported in Memory mapped files performance#106 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.IO.MemoryMappedFiles
• System.IO.Pipelines.PerformanceTests benchmarks were ported in Port System.IO.Pipelines benchmarks from CoreFX performance#369 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.IO.Pipelines
• System.IO.Pipes.PerformanceTests benchmarks were ported in Port System.IO.Pipes performance#108 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.IO.Pipes
• System.Linq.PerformanceTests benchmarks were ported in Port Port System.Linq.Performance.Tests  performance#90 and improved (they were using a very expensive .ToArray call to materialize every LINQ which was dominating the reported time) in don't use very expensive .ToArray() to measure other LINQ extenion methods performance performance#127 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Linq
• System.Linq.Parallel.PerformanceTests got ported (https://github.com/adamsitnik/performance/tree/parallelLinq) but not merged. They need a redesign Redesign System.Linq.Parallel benchmarks from CoreFX performance#359
• System.Memory.PerformanceTests benchmarks were rewritten (to take advantage of port to BDN) in System.Memory performance#94 and port missing System.Memory benchmarks performance#377 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Memory
• System.Net.Http.PerformanceTests benchmarks were ported in Port System.Net.Http benchmarks performance#100 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Net.Http
• System.Net.Primitives.PerformanceTests benchmarks were ported and fixed (1 benchmark was using hardocded value instead of the passed argument) in Net.Primitives performance#116 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Net.Primitives
• System.Net.Sockets.PerformanceTests benchmarks were ported in Sockets performance#101 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Net.Sockets
• System.Numerics.Vectors.PerformanceTests benchmarks were ported in Port System.Numerics.Vectors.Performance.Tests  performance#87 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Numerics.Vectors
• System.Runtime.PerformanceTests benchmarks were ported and fixed (few of them had houndreds of test cases and needed almost an hour to run, I added a validator to make sure we don't repeat same mistake again) in System.Runtime performance#112 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Runtime
• System.Runtime.Extensions.PerformanceTests benchmarks were ported in Runtime extensions performance#117 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Runtime.Extensions
• System.Runtime.Numerics.PerformanceTests benchmarks were ported inhttps://github.com/Port and fix BigInteger benchmarks performance#99 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Runtime.Numerics
• System.Runtime.Serialization benchmarks were rewritten (to take advantage of port to BDN) and extended with third party serializers in Serializer benchmarks performance#23 and System.Runtime.Serialization.Xml performance#110, now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/Serializers
  • System.Runtime.Serialization.Json.PerformanceTests
  • System.Runtime.Serialization.Xml.PerformanceTests
  • System.Xml.XmlSerializer.PerformanceTests
• System.Text.Encoding.PerformanceTests benchmarks were ported in Port System.Text.Encoding.Performance.Tests  performance#81 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Text.Encoding
• System.Text.RegularExpressions.PerformanceTests benchmarks were ported in Port RegularExpressions benchmarks performance#84 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Text.RegularExpressions
• System.Threading.PerformanceTests benchmarks were ported (and some duplicates covered by CoreCLR benchmarks were removed) in Port System.Threading.Performance.Tests and remove duplicated benchmarks performance#80 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Threading
• System.Threading.Channels.PerformanceTests benchmarks were ported in Port Channels benchmarks performance#79 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Threading.Channels
• System.Threading.Tasks.Extensions.PerformanceTests benchmarks were ported in Port value task benchmarks performance#75 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Threading.Tasks.Extensions There was a huge difference (x81) in the time reported by xunit-performance, which was due to a xunit-performance bug.
• System.Security.Cryptography.Primitives.PerformanceTests benchmarks were ported in Port System.Security.Cryptography.Primitives benchmarks performance#86 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Security.Cryptography.Primitives
• System.Xml.XmlDocument.PerformanceTests benchmarks were ported in Port Port System.Xml.XmlDocument.Performance.Tests  performance#88 and now are available at https://github.com/dotnet/performance/tree/master/src/benchmarks/micro/corefx/System.Xml.XmlDocument
• Marvin benchmarks were testing a local copy of Marvin, which was bad because they would not catch any regression introduced to Marvin. They were rewritten in Marvin  performance#114 and now are available at https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/corefx/System/Hashing.cs

[ViktorHofer]

There's also some cleanup in arcade possible when this is removed. I will submit a PR so that you can merge both when you think this should go into master.

[ViktorHofer]

So what about SocketPerformanceAsyncTests? Keeping the trait on it means it will never run. Either remove the trait or remove the whole project.

[adamsitnik]

So what about SocketPerformanceAsyncTests? Keeping the trait on it means it will never run. Either remove the trait or remove the whole project.

They are stored in "ManualPerformanceTests" folder which means that somebody did this on purpose. The problem is that git blame shows only @ViktorHofer on the contributors list so I am not sure who is the owner that could make such decision ;)

[ViktorHofer]

I don't think you need a "owner decision" for that. Sure, if we just want to run them manually periodically then the Trait can stay.

[jkotas]

Could you please also update all repo docs that talk about performance tests, e.g.:

https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/performance-tests.md
https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md
https://github.com/dotnet/corefx/blob/master/eng/build.ps1#L31
https://github.com/dotnet/corefx/blob/master/eng/build.sh#L31

[ahsonkhan]

Could you please also update all repo docs that talk about performance tests, e.g.:

And https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/benchmarking.md

[adamsitnik]

The last missing benchmarks have been ported and merged into the performance repo (dotnet/performance#381).

@danmosemsft @stephentoub @ViktorHofer this PR is ready for review.

[ViktorHofer]

@adamsitnik Thanks a lot!