Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update BenchmarkDotNet to latest version to take advantage of improvements #511

Merged
merged 9 commits into from
Aug 14, 2019
Merged

update BenchmarkDotNet to latest version to take advantage of improvements #511

merged 9 commits into from
Aug 14, 2019

Conversation

adamsitnik
Copy link
Member

The update contains a lof of minor improvements and bug fixes, but the most important are:

"Use new .NET Core 3.0 API to get the total number of allocated bytes for all threads"

dotnet/BenchmarkDotNet@f54055a So far BDN was using GC.GetAllocatedBytesForCurrentThread which returns the value for current thread only. Recently new API was added, GC.GetTotalAllocatedBytes which returns the value of allocated managed memory for all threads. By using it we get the proper numbers. For multi-threaded benchmarks (few dozens in the enitre portfolio) we are going to get new, bigger numbers. This is a change of reported values, but also a bug fix (cc @stephentoub).

"Use Median instead of Mean for overhead calculations"

dotnet/BenchmarkDotNet@d9901ba

One of the techniques BDN uses to allow for accurate nano-benchmarking is overhead deduction. BDN measures the overhead by benchmarking a method of same return type and arguments as the benchmarked method. The difference is that the "overhead" method does nothing for void and returns default for methods returning anything.

So far the actual results was:

Result(n) = Actual(n) - Average(Overhead)

But as @AndyAyersMS has spotted, using Average for deducing the overhead is bad when we have some outliers that affect the Average:

1.69
1.65
1.70
1.83
2.60 <-- an outlier -->
1.76
1.69
1.69
1.73
1.64
1.66
1.67
1.71
1.68
1.64

The implementation was changed, now BDN uses Median.

How it's going to affect the reported results? The nano-benchmarks are going to get more stable results, we are going to have fewer results affected by outliers.

Power Plans

By default BDN is now enforcing High Performance Power plan when running benchmarks on Windows. dotnet/BenchmarkDotNet@0dfa37e

To make sure it does not affect the results reported in our lab, I have disabled this behaviour in the RecommendedConfig type (.DontEnforcePowerPlan()). So this new feature is not going to affect the results, we might consider enabling it in the futre.

Other improvements

To see the diff for the update from 0.11.3.1003 to 0.11.5.1090 please click here

@adamsitnik adamsitnik requested a review from billwert May 27, 2019 15:26
@billwert
Copy link
Member

billwert commented May 27, 2019 •
edited
Loading

Thanks @adamsitnik!

I have done a comparison between our current version of BDN and 0.11.5.1086, so very close to this diff.

I have put the total results here: https://gist.github.com/billwert/9469538fb651b8b022e7bebdf8c7295f

Of particular interest:

Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Memory.Constructors.MemoryMarshalCreateSpan 12.53 0.34 0.03
System.Memory.Constructors_ValueTypesOnly.ReadOnlyFromPointerLength 12.12 0.41 0.03 multimodal

These seem to be impacted by the shift from Mean to Median. Here's what the raw results look like for MemoryMarshalCreateSpan:

Candidate

OverheadActual 1: 152565216 op, 242377100.00 ns, 1.5887 ns/op
OverheadActual 2: 152565216 op, 242047300.00 ns, 1.5865 ns/op
OverheadActual 3: 152565216 op, 244870100.00 ns, 1.6050 ns/op
OverheadActual 4: 152565216 op, 241318900.00 ns, 1.5817 ns/op
OverheadActual 5: 152565216 op, 243208200.00 ns, 1.5941 ns/op
OverheadActual 6: 152565216 op, 242559400.00 ns, 1.5899 ns/op
OverheadActual 7: 152565216 op, 247181200.00 ns, 1.6202 ns/op
OverheadActual 8: 152565216 op, 244912500.00 ns, 1.6053 ns/op
OverheadActual 9: 152565216 op, 242285700.00 ns, 1.5881 ns/op
OverheadActual 10: 152565216 op, 242764900.00 ns, 1.5912 ns/op
OverheadActual 11: 152565216 op, 246956300.00 ns, 1.6187 ns/op
OverheadActual 12: 152565216 op, 242917100.00 ns, 1.5922 ns/op
OverheadActual 13: 152565216 op, 242545600.00 ns, 1.5898 ns/op
OverheadActual 14: 152565216 op, 244401300.00 ns, 1.6019 ns/op
OverheadActual 15: 152565216 op, 245350300.00 ns, 1.6082 ns/op

WorkloadWarmup 1: 152565216 op, 243468100.00 ns, 1.5958 ns/op

// BeforeActualRun
WorkloadActual 1: 152565216 op, 242658700.00 ns, 1.5905 ns/op
WorkloadActual 2: 152565216 op, 242035800.00 ns, 1.5864 ns/op
WorkloadActual 3: 152565216 op, 245612600.00 ns, 1.6099 ns/op
WorkloadActual 4: 152565216 op, 242780700.00 ns, 1.5913 ns/op
WorkloadActual 5: 152565216 op, 244314200.00 ns, 1.6014 ns/op
WorkloadActual 6: 152565216 op, 242844700.00 ns, 1.5917 ns/op
WorkloadActual 7: 152565216 op, 242693400.00 ns, 1.5908 ns/op
WorkloadActual 8: 152565216 op, 242765900.00 ns, 1.5912 ns/op
WorkloadActual 9: 152565216 op, 241449100.00 ns, 1.5826 ns/op
WorkloadActual 10: 152565216 op, 241861600.00 ns, 1.5853 ns/op
WorkloadActual 11: 152565216 op, 247192900.00 ns, 1.6202 ns/op
WorkloadActual 12: 152565216 op, 248041500.00 ns, 1.6258 ns/op
WorkloadActual 13: 152565216 op, 247349100.00 ns, 1.6213 ns/op
WorkloadActual 14: 152565216 op, 244272200.00 ns, 1.6011 ns/op
WorkloadActual 15: 152565216 op, 242933700.00 ns, 1.5923 ns/op

// AfterActualRun
WorkloadResult 1: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 2: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 3: 152565216 op, 2695500.00 ns, 0.0177 ns/op
WorkloadResult 4: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 5: 152565216 op, 1397100.00 ns, 0.0092 ns/op
WorkloadResult 6: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 7: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 8: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 9: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 10: 152565216 op, 0.00 ns, 0.0000 ns/op
WorkloadResult 11: 152565216 op, 4275800.00 ns, 0.0280 ns/op
WorkloadResult 12: 152565216 op, 5124400.00 ns, 0.0336 ns/op
WorkloadResult 13: 152565216 op, 4432000.00 ns, 0.0290 ns/op
WorkloadResult 14: 152565216 op, 1355100.00 ns, 0.0089 ns/op
WorkloadResult 15: 152565216 op, 16600.00 ns, 0.0001 ns/op

Baseline

OverheadActual 1: 124610960 op, 192520900.00 ns, 1.5450 ns/op
OverheadActual 2: 124610960 op, 255380400.00 ns, 2.0494 ns/op
OverheadActual 3: 124610960 op, 203979000.00 ns, 1.6369 ns/op
OverheadActual 4: 124610960 op, 193032100.00 ns, 1.5491 ns/op
OverheadActual 5: 124610960 op, 192476100.00 ns, 1.5446 ns/op
OverheadActual 6: 124610960 op, 193009900.00 ns, 1.5489 ns/op
OverheadActual 7: 124610960 op, 217588500.00 ns, 1.7461 ns/op
OverheadActual 8: 124610960 op, 216896600.00 ns, 1.7406 ns/op
OverheadActual 9: 124610960 op, 194498000.00 ns, 1.5608 ns/op
OverheadActual 10: 124610960 op, 202146800.00 ns, 1.6222 ns/op
OverheadActual 11: 124610960 op, 201542200.00 ns, 1.6174 ns/op
OverheadActual 12: 124610960 op, 185823800.00 ns, 1.4912 ns/op
OverheadActual 13: 124610960 op, 211242300.00 ns, 1.6952 ns/op
OverheadActual 14: 124610960 op, 194089800.00 ns, 1.5576 ns/op
OverheadActual 15: 124610960 op, 206162600.00 ns, 1.6544 ns/op
OverheadActual 16: 124610960 op, 223671300.00 ns, 1.7950 ns/op
OverheadActual 17: 124610960 op, 198295600.00 ns, 1.5913 ns/op
OverheadActual 18: 124610960 op, 213353900.00 ns, 1.7122 ns/op
OverheadActual 19: 124610960 op, 222495800.00 ns, 1.7855 ns/op
OverheadActual 20: 124610960 op, 216971000.00 ns, 1.7412 ns/op

WorkloadWarmup 1: 124610960 op, 262845500.00 ns, 2.1093 ns/op

// BeforeActualRun
WorkloadActual 1: 124610960 op, 247317800.00 ns, 1.9847 ns/op
WorkloadActual 2: 124610960 op, 265753300.00 ns, 2.1327 ns/op
WorkloadActual 3: 124610960 op, 252514900.00 ns, 2.0264 ns/op
WorkloadActual 4: 124610960 op, 268808100.00 ns, 2.1572 ns/op
WorkloadActual 5: 124610960 op, 271620900.00 ns, 2.1798 ns/op
WorkloadActual 6: 124610960 op, 262705200.00 ns, 2.1082 ns/op
WorkloadActual 7: 124610960 op, 240089500.00 ns, 1.9267 ns/op
WorkloadActual 8: 124610960 op, 261646300.00 ns, 2.0997 ns/op
WorkloadActual 9: 124610960 op, 274081100.00 ns, 2.1995 ns/op
WorkloadActual 10: 124610960 op, 238811500.00 ns, 1.9165 ns/op
WorkloadActual 11: 124610960 op, 227333800.00 ns, 1.8243 ns/op
WorkloadActual 12: 124610960 op, 257076200.00 ns, 2.0630 ns/op
WorkloadActual 13: 124610960 op, 263583000.00 ns, 2.1152 ns/op
WorkloadActual 14: 124610960 op, 238869900.00 ns, 1.9169 ns/op
WorkloadActual 15: 124610960 op, 242160900.00 ns, 1.9433 ns/op
WorkloadActual 16: 124610960 op, 254169800.00 ns, 2.0397 ns/op
WorkloadActual 17: 124610960 op, 238191600.00 ns, 1.9115 ns/op
WorkloadActual 18: 124610960 op, 225440600.00 ns, 1.8092 ns/op
WorkloadActual 19: 124610960 op, 242206900.00 ns, 1.9437 ns/op
WorkloadActual 20: 124610960 op, 252964200.00 ns, 2.0300 ns/op

// AfterActualRun
WorkloadResult 1: 124610960 op, 40558970.00 ns, 0.3255 ns/op
WorkloadResult 2: 124610960 op, 58994470.00 ns, 0.4734 ns/op
WorkloadResult 3: 124610960 op, 45756070.00 ns, 0.3672 ns/op
WorkloadResult 4: 124610960 op, 62049270.00 ns, 0.4979 ns/op
WorkloadResult 5: 124610960 op, 64862070.00 ns, 0.5205 ns/op
WorkloadResult 6: 124610960 op, 55946370.00 ns, 0.4490 ns/op
WorkloadResult 7: 124610960 op, 33330670.00 ns, 0.2675 ns/op
WorkloadResult 8: 124610960 op, 54887470.00 ns, 0.4405 ns/op
WorkloadResult 9: 124610960 op, 67322270.00 ns, 0.5403 ns/op
WorkloadResult 10: 124610960 op, 32052670.00 ns, 0.2572 ns/op
WorkloadResult 11: 124610960 op, 20574970.00 ns, 0.1651 ns/op
WorkloadResult 12: 124610960 op, 50317370.00 ns, 0.4038 ns/op
WorkloadResult 13: 124610960 op, 56824170.00 ns, 0.4560 ns/op
WorkloadResult 14: 124610960 op, 32111070.00 ns, 0.2577 ns/op
WorkloadResult 15: 124610960 op, 35402070.00 ns, 0.2841 ns/op
WorkloadResult 16: 124610960 op, 47410970.00 ns, 0.3805 ns/op
WorkloadResult 17: 124610960 op, 31432770.00 ns, 0.2522 ns/op
WorkloadResult 18: 124610960 op, 18681770.00 ns, 0.1499 ns/op
WorkloadResult 19: 124610960 op, 35448070.00 ns, 0.2845 ns/op
WorkloadResult 20: 124610960 op, 46205370.00 ns, 0.3708 ns/op

Note the actual results are often zero with thew new BDN. How should we handle this sort of test? Should we add inner iterations perhaps?

@adamsitnik
Copy link
Member Author

I think that this particular update has not introduced a new problem, but instead revealed an existing one.

This update changes only the way we do the math, not the way we run the benchmarks.

I have put the "Candidate" numbers to excel to show the difference:

Overhead(i) Average Median Workload(i) Old Result New Result New - Old
1,5887 1,597433333 1,5922 1,5905 -0,006933333 -0,0017 0,005233333
1,5865 1,5864 -0,011033333 -0,0058 0,005233333
1,605 1,6099 0,012466667 0,0177 0,005233333
1,5817 1,5913 -0,006133333 -0,0009 0,005233333
1,5941 1,6014 0,003966667 0,0092 0,005233333
1,5899 1,5917 -0,005733333 -0,0005 0,005233333
1,6202 1,5908 -0,006633333 -0,0014 0,005233333
1,6053 1,5912 -0,006233333 -0,001 0,005233333
1,5881 1,5826 -0,014833333 -0,0096 0,005233333
1,5912 1,5853 -0,012133333 -0,0069 0,005233333
1,6187 1,6202 0,022766667 0,028 0,005233333
1,5922 1,6258 0,028366667 0,0336 0,005233333
1,5898 1,6213 0,023866667 0,0291 0,005233333
1,6019 1,6011 0,003666667 0,0089 0,005233333
1,6082 1,5923 -0,005133333 1E-04 0,005233333

As you can see, here there was no outliers and Average is almost same as Median. The difference is very small (0,005233333). The results are not affected.

The same for "Baseline" numbers:

Overhead(i) Average Median Workload(i) Old Result New Result New - Old
1,545 1,65923 1,62955 1,9847 0,32547 0,35515 0,02968
2,0494 2,1327 0,47347 0,50315 0,02968
1,6369 2,0264 0,36717 0,39685 0,02968
1,5491 2,1572 0,49797 0,52765 0,02968
1,5446 2,1798 0,52057 0,55025 0,02968
1,5489 2,1082 0,44897 0,47865 0,02968
1,7461 1,9267 0,26747 0,29715 0,02968
1,7406 2,0997 0,44047 0,47015 0,02968
1,5608 2,1995 0,54027 0,56995 0,02968
1,6222 1,9165 0,25727 0,28695 0,02968
1,6174 1,8243 0,16507 0,19475 0,02968
1,4912 2,063 0,40377 0,43345 0,02968
1,6952 2,1152 0,45597 0,48565 0,02968
1,5576 1,9169 0,25767 0,28735 0,02968
1,6544 1,9433 0,28407 0,31375 0,02968
1,795 2,0397 0,38047 0,41015 0,02968
1,5913 1,9115 0,25227 0,28195 0,02968
1,7122 1,8092 0,14997 0,17965 0,02968
1,7855 1,9437 0,28447 0,31415 0,02968
1,7412 2,03 0,37077 0,40045 0,02968

Here, we had some outliers (bold numbers) and the difference between Average and Median is noticable: 0,02968ns (it's still a little difference). This is the case @AndyAyersMS has reported and @AndreyAkinshin has fixed by using the Median instead of Average.

I will try to find some time this week and answer the most important question: why do we get zeros? Are the two mentioned benchmarks optimized by JIT to a const? Are they single instruction and BDN can't handle that and we should rewrite them? Are they very dependent on the array alignment and we should rewrite them? Is BDN generating the right code for the overhead method that returns Span?

@billwert
Copy link
Member

billwert commented May 27, 2019 via email

This was referenced May 30, 2019
Closed
Merged
@billwert
Copy link
Member

@adamsitnik any updates?

@adamsitnik adamsitnik mentioned this pull request Jul 29, 2019
Merged
@adamsitnik
Copy link
Member Author

@billwert the diff that you have uploaded contains a few different kinds of benchmarks:

  • unstable benchmarks like the old Clear benchmarks that have|had a bad design
  • multimodal benchmarks that rely on IO like Sockets
  • multimodal benchmarks that do sth in parallel like ArrayPool benchmarks
  • benchmarks dependent heavily on the code alignment (like IniArray)

I did not have the time to go through the list and look at each of them, but when I was doing the 2.2 vs 3.0 comparison and some of them became a real pain I've redesigned them. So the list should be definitely shorter now.

Right now I think that the best thing to do would be:

  • run all the benchmarks twice using the currently used version of BDN. Do the diff which should contain only the unstable benchmarks. Create an issue to track them and improve in the future.
  • run all the benchmarks again using latest BDN, do the diff and compare it against the list of unstable benchmarks. Investigate every benchmark that was not on the list of unstable benchmarks from step 1.

@billwert does it sound like a good plan to you?

I'll try to run all the benchmarks again this week and make some progress here.

@billwert
Copy link
Member

@adamsitnik I think this is a good plan.

@adamsitnik adamsitnik mentioned this pull request Aug 14, 2019
Closed
@billwert
Copy link
Member

@adiaaida to review the yml change and make sure it's inline with the plan.

@adamsitnik
Copy link
Member Author

This is a short version of the report to unblock the team.

I did run the benchmarks twice using exactly the same IL, OS, Hardware and BDN version. Then I created a diff which has shown the list of unstable benchmarks. Some were expected (IO, Multithreaded) some were not (like Parsing the numbers). The details below:

summary:
better: 62, geomean: 1.172
worse: 66, geomean: 1.133
total diff: 131

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.Tests.Perf_Int64.TryParseSpan(value: "9223372036854775807") 1.42 26.64 37.82
System.Collections.TryAddDefaultSize.ConcurrentDictionary(Count: 512) 1.37 60276.05 82299.68
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 1.35 2741.89 3701.74 several?
System.Tests.Perf_UInt64.TryParse(value: "12345") 1.31 15.12 19.79
System.Tests.Perf_Int32.ParseHex(value: "7FFFFFFF") 1.29 22.46 28.98 several?
System.Tests.Perf_StringComparer.CompareSame(Count: 262144, Comparison: OrdinalI 1.29 72260.85 93050.78 bimodal
System.Collections.CtorDefaultSize.ConcurrentBag 1.27 621.89 790.22 bimodal
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: 1.27 211917.09 268100.00
System.Collections.ContainsKeyFalse<Int32, Int32>.IDictionary(Size: 512) 1.23 6125.14 7563.73
System.Collections.Tests.Add_Remove_SteadyState.ConcurrentBag(Count: 512) 1.23 51.91 63.80 bimodal
System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 12345) 1.20 7.01 8.39
System.Tests.Perf_UInt32.Parse(value: "4294967295") 1.19 19.64 23.42 several?
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromResult_ConfigureAwai 1.17 27.04 31.62
System.Memory.ReadOnlySequence.Slice_StartPosition_And_EndPosition(Segment: Sing 1.17 5.88 6.85
System.Tests.Perf_Int64.TryParse(value: "12345") 1.17 15.23 17.76 several?
System.IO.Tests.Perf_Directory.CreateDirectory 1.16 251203.50 292590.00 bimodal
System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch_Multithreading(tot 1.16 110743525.00 128140500.00
System.Collections.ContainsFalse.Array(Size: 512) 1.15 712327.90 822477.88
System.Threading.Channels.Tests.BoundedChannelPerfTests.ReadAsyncThenWriteAsync 1.15 120.09 138.40
System.Collections.Concurrent.IsEmpty.Bag(Size: 512) 1.15 11.88 13.68 several?
System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512) 1.15 5635.71 6477.28
System.Collections.CtorFromCollection.ImmutableSortedSet(Size: 512) 1.14 335712.84 382262.94
System.Collections.ContainsFalse.HashSet(Size: 512) 1.13 4744.52 5380.95
System.IO.Tests.Perf_FileStream.Write(BufferSize: 512, TotalSize: 200000) 1.13 920653.79 1042994.32
System.Collections.ContainsKeyFalse<String, String>.ImmutableSortedDictionary(Si 1.13 309988.96 349684.99
System.Collections.CtorFromCollection.ConcurrentDictionary(Size: 512) 1.13 77709.89 87455.25
PerfLabTests.EnumPerf.EnumEquals 1.12 7.78 8.74
System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 2) 1.12 103552350.00 115978950.00
Devirtualization.EqualityComparer.ValueTupleCompare 1.12 2.84 3.18 bimodal
System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructAndReverseBE 1.12 15.36 17.17 bimodal
System.MathBenchmarks.Double.Atanh 1.12 77153.16 86118.09
System.Collections.CopyTo.List(Size: 2048) 1.11 477.09 530.76
System.Buffers.Text.Tests.Utf8ParserTests.TryParseSByte(value: 127) 1.11 6.47 7.19
System.Buffers.Tests.RentReturnArrayPoolTests.SingleSerial(RentalSize: 409 1.10 39.67 43.75
System.Collections.CreateAddAndClear.ConcurrentDictionary(Size: 512) 1.10 75109.28 82796.67
MicroBenchmarks.Serializers.Binary_FromStream.BinaryFor 1.10 1052782.33 1159862.16
MicroBenchmarks.Serializers.Json_FromStream.Jil_ 1.10 1298.87 1429.59
System.Collections.TryGetValueFalse<Int32, Int32>.Dictionary(Size: 512) 1.10 6742.98 7416.21
MicroBenchmarks.Serializers.Json_FromString.Jil_ 1.10 298479.03 328142.11
System.Tests.Perf_Guid.ctor_str 1.10 88.72 97.37
System.Threading.Channels.Tests.UnboundedChannelPerfTests.ReadAsyncThenWriteAsyn 1.09 141.16 153.41
System.Collections.TryGetValueFalse<String, String>.SortedDictionary(Size: 512) 1.09 347281.81 377157.01
LinqBenchmarks.Where01LinqQueryX 1.09 313470250.00 340118750.00
System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_NoEscaping(NumberOfByt 1.08 279.21 302.70
Benchstone.BenchI.NDhrystone.Test 1.08 524897200.00 568898500.00
MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ 1.08 29900.30 32351.11
System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests.ReadAsyncThenWrite 1.08 118.68 128.39
System.Collections.CreateAddAndRemove.LinkedList(Size: 512) 1.08 11528.33 12441.47
System.Tests.Perf_UInt64.TryParse(value: "0") 1.08 11.21 12.10
System.Tests.Perf_Char.Char_ToLower(c: '?', cultureName: zh-Hans) 1.08 45.32 48.86
System.Threading.Channels.Tests.UnboundedChannelPerfTests.WriteAsyncThenReadAsyn 1.08 89.58 96.47
System.Memory.ReadOnlyMemory.ToArray(Size: 512) 1.08 41.88 45.10
System.Collections.CreateAddAndClear.Dictionary(Size: 512) 1.08 23375.79 25159.47
System.Collections.ContainsTrue.List(Size: 512) 1.08 459594.34 494493.06
System.Collections.Concurrent.IsEmpty.Dictionary(Size: 0) 1.07 194.46 208.94
System.Memory.Span.SequenceCompareTo(Size: 512) 1.07 21.64 23.24
System.Collections.CtorFromCollection.SortedList(Size: 512) 1.07 381577.34 409405.15
MicroBenchmarks.Serializers.Json_ToStream.Jil_ 1.07 264.23 283.06
Microsoft.AspNetCore.Server.Kestrel.Performance.PipeThroughputBenchmark.Parse_Pa 1.07 598.86 640.54
System.Globalization.Tests.Perf_CompareInfo.IsSuffix(culture: en-US, source: "St 1.07 10.71 11.43
System.Memory.ReadOnlySequence.Slice_StartPosition(Segment: Single) 1.07 6.37 6.78 several?
System.Globalization.Tests.Perf_CompareInfo.LastIndexOf(culture: en-US, source: 1.06 35.86 38.17
System.Collections.TryGetValueFalse<String, String>.ImmutableSortedDictionary(Si 1.06 317408.70 336162.67
System.Tests.Perf_Int32.ParseHex(value: "80000000") 1.06 21.88 23.17 several?
System.Tests.Perf_Double.Parse(value: "12345") 1.06 71.13 75.30
System.Collections.CreateAddAndClear.ConcurrentBag(Size: 512) 1.06 14796.67 15617.50
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Collections.Concurrent.Count.Dictionary(Size: 512) 3.98 12103.26 3040.69
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: 1.80 138300.00 76796.75 several?
Benchstone.BenchF.Simpsn.Test 1.62 229020850.00 141394950.00
System.Collections.AddGivenSize.HashSet(Size: 512) 1.57 12157.97 7747.33
System.Collections.IterateForEach.ConcurrentDictionary(Size: 512) 1.32 20292.32 15426.52
System.Collections.CreateAddAndRemove.HashSet(Size: 512) 1.29 26408.74 20440.17
System.Collections.Concurrent.IsEmpty.Bag(Size: 0) 1.28 15.69 12.29 several?
System.Memory.Span.Clear(Size: 512) 1.27 31.03 24.47
System.Tests.Perf_UInt64.ParseSpan(value: "12345") 1.25 20.21 16.15
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 1.24 1376.54 1106.00 several?
System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii") 1.22 24.08 19.79
System.Tests.Perf_UInt16.TryParse(value: "12345") 1.20 16.92 14.16
System.Drawing.Tests.Perf_Graphics_DrawBeziers.DrawBezier_Points 1.18 1952.38 1649.64
System.Tests.Perf_Int32.TryParse(value: "4") 1.18 12.15 10.28
System.Tests.Perf_UInt32.ParseSpan(value: "12345") 1.17 18.92 16.16
System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8") 1.16 31.98 27.49
System.Buffers.Tests.ReadOnlySequenceTests.FirstTenSegments 1.16 6.29 5.42
System.Tests.Perf_UInt32.TryParse(value: "0") 1.16 11.64 10.04
System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE 1.15 16.22 14.09 several?
System.Collections.CopyTo.Span(Size: 2048) 1.15 584.43 509.69
System.Tests.Perf_Int32.ParseSpan(value: "4") 1.15 15.31 13.37
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_Config 1.14 41.25 36.03
System.Tests.Perf_Int64.TryParse(value: "9223372036854775807") 1.14 30.54 26.80
System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512) 1.14 6962.87 6112.16
System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808") 1.14 29.79 26.22
System.IO.Compression.Brotli.Compress(level: Optimal, file: "TestDocument.pdf") 1.13 14636.03 12901.99 several?
System.Collections.CtorDefaultSize.List 1.13 10.97 9.68
System.Collections.TryGetValueTrue<String, String>.SortedList(Size: 512) 1.13 286087.45 252710.72
System.MathBenchmarks.Double.Exp 1.13 36254.95 32120.53
System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 1000 1.13 10809.57 9586.45
System.Threading.Tests.Perf_Timer.ShortScheduleAndDisposeWithFiringTimers 1.11 189.75 170.22
System.Collections.CtorDefaultSize.SortedList 1.11 10.53 9.46
System.Collections.CreateAddAndRemove.SortedSet(Size: 512) 1.11 590103.83 529869.60
System.Perf_Convert.ChangeType 1.11 46.70 42.21
System.Tests.Perf_Int32.ParseSpan(value: "2147483647") 1.11 23.53 21.27
System.Threading.Tests.Perf_CancellationToken.CreateLinkedTokenSource3 1.11 173.62 157.12
System.Linq.Tests.Perf_Enumerable.OrderBy(input: IEnumerable) 1.10 4762.38 4317.93 bimodal
LinqBenchmarks.Order00ManualX 1.10 155893350.00 141501400.00
System.Collections.IterateForEach.ImmutableList(Size: 512) 1.10 44633.66 40585.19
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 1.10 1392.59 1270.88
System.Tests.Perf_UInt16.TryParse(value: "0") 1.09 12.45 11.38
SeekUnroll.Test(boxedIndex: 19) 1.09 1656663500.00 1514774100.00
System.Buffers.Tests.ReadOnlySequenceTests.SliceTenSegments 1.09 32.56 29.77
System.Threading.Tests.Perf_CancellationToken.CancelAfter 1.09 120.66 110.49
System.Linq.Tests.Perf_Enumerable.OrderByThenBy(input: IEnumerable) 1.09 4988.15 4577.01
System.Collections.IndexerSet.IList(Size: 512) 1.09 7063.79 6498.92
System.Collections.IndexerSet.Dictionary(Size: 512) 1.09 6394.40 5892.44
System.Collections.IterateFor.ImmutableSortedSet(Size: 512) 1.08 13310.56 12293.24
System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 1.08 253249.81 235047.85
System.Collections.CtorDefaultSize.SortedSet 1.08 13.67 12.70
System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512) 1.08 22.93 21.31
System.Net.Primitives.Tests.CredentialCacheTests.ForEach(uriCount: 0, hostPortCo 1.08 33.69 31.32
System.Tests.Perf_Int64.Parse(value: "12345") 1.07 16.05 15.01
System.Collections.Concurrent.IsEmpty.Bag(Size: 0) 1.07 13.61 12.74
System.Threading.Tasks.ValueTaskPerfTest.Await_FromCompletedTask 1.06 31.27 29.38
SeekUnroll.Test(boxedIndex: 27) 1.06 2156615600.00 2027346400.00
System.Collections.CtorDefaultSize.ConcurrentQueue 1.06 66.16 62.28
System.Collections.Sort.LinqQuery(Size: 512) 1.06 372426.33 350670.78
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSo 1.06 49.55 46.68
Exceptions.Handling.MultipleNestedTryCatch_LastCatches(kind: Hardware) 1.06 22083.41 20874.07
System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "ascii") 1.06 106.23 100.43
System.Collections.CtorFromCollectionNonGeneric.Stack(Size: 512) 1.06 31676.41 29969.21 bimodal

@michellemcdaniel
Copy link
Contributor

michellemcdaniel commented Aug 14, 2019 •
edited
Loading

We need to add netcoreapp5.0 to the private jobs too (I rearranged the yaml to make more sense to me, but that means that the private builds are further down -- line 192 or so)

-- I made this change.

@adamsitnik
Copy link
Member Author

Then I've run the benchmarks using latest version of BDN and did the diff again.

The benchmarks in the diff:

I did not find any BDN bugs or other things that have affected the results in negative way

summary:
better: 147, geomean: 1.141
worse: 162, geomean: 1.185
total diff: 311

Slower diff/base Base Median (ns) Diff Median (ns) Modality
MicroBenchmarks.Serializers.Json_ToString.Jil_ 8.03 416.96 3350.00 several?
System.Numerics.Tests.Perf_Vector2.MultiplyOperatorBenchmark 2.81 0.20 0.55 several?
System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512) 2.55 550.99 1407.06
System.Tests.Perf_Boolean.ToString(value: True) 2.13 0.78 1.67
MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ 1.91 336726.95 643600.00
System.Tests.Perf_Int32.TryParseSpan(value: "2147483647") 1.71 17.86 30.48
System.Collections.CtorDefaultSize.ConcurrentBag 1.64 621.89 1021.23 bimodal
System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648") 1.60 19.02 30.51 several?
System.Collections.Tests.Add_Remove_SteadyState.Queue(Count: 512) 1.58 6.43 10.19
System.Collections.CtorDefaultSize.ConcurrentBag 1.56 661.23 1031.35 bimodal
System.Collections.Concurrent.AddRemoveFromDifferentThreads.ConcurrentSt 1.49 168500235.00 251658850.00 bimodal
System.Memory.Span.StartsWith(Size: 512) 1.48 20.49 30.25
System.Collections.CopyTo.ReadOnlySpan(Size: 2048) 1.46 100.49 147.11 several?
System.Tests.Perf_Boolean.ToString(value: False) 1.44 0.82 1.18
System.Tests.Perf_Int64.Parse(value: "9223372036854775807") 1.41 28.25 39.96 bimodal
System.Tests.Perf_Int64.TryParseSpan(value: "12345") 1.40 13.90 19.39
Benchstone.BenchF.NewtR.Test 1.39 232838400.00 324191350.00
System.Collections.TryAddDefaultSize.ConcurrentDictionary(Count: 512) 1.38 60276.05 83461.49
System.Memory.Span.LastIndexOfAnyValues(Size: 512) 1.36 16.77 22.88
System.Tests.Perf_UInt32.Parse(value: "4294967295") 1.36 19.64 26.68
System.IO.Tests.Perf_Path.HasExtension 1.35 4.77 6.45 several?
System.Tests.Perf_StringComparer.CompareSame(Count: 262144, Comparison: OrdinalI 1.29 72260.85 93346.91 bimodal
System.Memory.Span.SequenceCompareTo(Size: 512) 1.27 13.26 16.82 bimodal
System.Net.Security.Tests.SslStreamTests.ConcurrentReadWriteLargeBuffer 1.26 19406.64 24359.86
System.Tests.Perf_String.IndexerCheckPathLength 1.25 118.50 148.38
System.Net.Sockets.Tests.SocketSendReceivePerfTest.ReceiveAsyncThenSendAsync_Soc 1.25 120213125.00 149957550.00
PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodSwitchCallType 1.24 713900.25 883040.99
System.Threading.Tasks.Tests.Perf_AsyncMethods.EmptyAsyncMethodInvocation 1.23 13.11 16.16
System.Tests.Perf_Array.IndexOfChar 1.23 9.38 11.53
System.Collections.IterateFor.IList(Size: 512) 1.23 2219.36 2720.23
System.Numerics.Tests.Perf_Vector2.DistanceSquaredBenchmark 1.23 1.49 1.83
System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808") 1.22 29.79 36.39
System.Collections.CtorDefaultSizeNonGeneric.SortedList 1.21 21.91 26.62 several?
System.Collections.CreateAddAndClear.ConcurrentDictionary(Size: 512) 1.21 93698.74 113424.00
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: 1.21 211917.09 255450.00
System.Memory.Constructors.ReadOnlySpanImplicitCastFromArraySegment 1.21 2.05 2.48
System.Collections.Tests.Perf_BitArray.BitArrayCopyToIntArray(Size: 512) 1.20 30.71 36.91
System.Collections.CreateAddAndClear.HashSet(Size: 512) 1.20 11666.90 13990.76
System.Tests.Perf_Char.Char_ToUpper(c: 'a', cultureName: en-US) 1.20 3.99 4.77
System.Tests.Perf_Int32.ToStringHex(value: 2147483647) 1.19 30.08 35.90 several?
System.Perf_Convert.FromBase64String 1.19 84.15 100.01
System.Perf_Convert.FromBase64Chars 1.19 86.90 103.11 several?
System.Linq.Tests.Perf_Enumerable.Prepend(input: IEnumerable) 1.19 3399.46 4032.42
System.IO.Compression.Brotli.Compress(level: Optimal, file: "alice29.txt") 1.18 21925.50 25934.83 bimodal
System.Collections.CtorDefaultSize.Stack 1.18 11.46 13.49
System.Collections.CreateAddAndClear.SortedSet(Size: 512) 1.17 276993.11 322909.31
System.Memory.Span.SequenceEqual(Size: 512) 1.16 41.93 48.78
System.Collections.Concurrent.IsEmpty.Bag(Size: 512) 1.16 11.91 13.84
System.Perf_Convert.GetTypeCode 1.16 3.72 4.33
System.Collections.AddGivenSize.SortedList(Size: 512) 1.16 316115.63 367224.78
System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch_Multithreading(tot 1.16 110743525.00 128616900.00
System.Collections.CtorFromCollection.SortedSet(Size: 512) 1.16 330359.26 383050.44 several?
System.Globalization.Tests.Perf_CompareInfo.Compare(culture: , string1: "XXXXXXX 1.16 4.51 5.22
System.Collections.IndexerSetReverse.Span(Size: 512) 1.15 205.07 236.80
GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.33 pD=0.67) 1.15 4.13 4.76
System.Collections.CreateAddAndRemove.SortedList(Size: 512) 1.15 557246.20 642556.25 bimodal
System.Tests.Perf_Int16.ToString(value: -32768) 1.15 35.91 41.39
System.Tests.Perf_String.Contains(text: "This is a very nice sentence", value: " 1.15 122.31 140.73
System.Memory.Span.IndexOfAnyFourValues(Size: 512) 1.15 52.11 59.86
System.Linq.Tests.Perf_Enumerable.SelectToArray(input: Array) 1.15 238.82 274.32
System.MathBenchmarks.Single.Asin 1.15 37821.29 43414.09
System.Tests.Perf_String.Compare(strings: ["The quick brown fox", "THE QUICK BRO 1.15 5.25 6.02
System.Tests.Perf_String.IndexerCheckBoundCheckHoist 1.15 33.27 38.13
System.Collections.TryGetValueFalse<String, String>.ConcurrentDictionary(Size: 5 1.14 16817.63 19254.25
System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt16(value: 0) 1.14 4.29 4.91 several?
System.Collections.IndexerSetReverse.Span(Size: 512) 1.14 245.84 281.08
System.Memory.Span.EndsWith(Size: 512) 1.14 7.71 8.80 several?
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 1.14 292.35 333.36 bimodal
System.Linq.Tests.Perf_Enumerable.SelectToArray(input: Range) 1.14 230.35 262.54
GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.00 pD=0.33) 1.14 4.17 4.74
PerfLabTests.CastingPerf2.CastingPerf.FooObjIsFoo 1.13 507640.28 574300.69
System.Tests.Perf_StringComparer.CompareSame(Count: 128, Comparison: OrdinalIgno 1.13 48.04 54.28
System.Tests.Perf_Double.TryParse(value: "12345") 1.13 72.34 81.46
MicroBenchmarks.Serializers.Binary_FromStream.BinaryFormatter_ 1.13 3551.88 3997.49
System.Tests.Perf_UInt32.TryParseHex(value: "FFFFFFFF") 1.12 15.27 17.16
Benchstone.BenchF.DMath.Test 1.12 672272600.00 755072450.00
System.Collections.IterateForEachNonGeneric.Stack(Size: 512) 1.12 3525.93 3957.59
PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull 1.11 425407.80 473636.36
System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable) 1.11 546.38 607.93
System.Tests.Perf_Environment.GetFolderPath(folder: System, option: None) 1.11 668.45 742.80
System.Tests.Perf_Char.Char_ToUpper(c: 'A', cultureName: en-US) 1.11 4.31 4.78
System.Collections.IterateForEach.LinkedList(Size: 512) 1.11 2903.28 3221.36
System.Memory.Span.SequenceCompareTo(Size: 512) 1.11 21.64 24.01
System.Collections.CtorDefaultSize.SortedList 1.11 23.87 26.47 several?
System.Collections.ContainsTrue.ImmutableSortedSet(Size: 512) 1.11 25143.33 27867.65 several?
System.Collections.Concurrent.Count.Queue(Size: 512) 1.11 7.06 7.82 multimodal
System.Collections.CtorDefaultSize.Queue 1.11 12.44 13.78 bimodal
PerfLabTests.CastingPerf2.CastingPerf.FooObjIsDescendant 1.11 449787.01 498104.88
PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod 1.11 137687.70 152295.68 bimodal
LinqBenchmarks.Where00LinqMethodX 1.10 565521100.00 624446050.00
System.Globalization.Tests.Perf_CompareInfo.IsSuffix(culture: en-US, source: "St 1.10 10.71 11.80
System.Collections.CtorFromCollection.ConcurrentDictionary(Size: 512) 1.10 77709.89 85506.88
System.Linq.Tests.Perf_Enumerable.Concat_Once(input: IEnumerable) 1.10 1652.42 1816.88 several?
Benchstone.BenchI.NDhrystone.Test 1.10 524897200.00 576435700.00
System.Collections.ContainsTrue.Array(Size: 512) 1.10 34122.00 37469.19
System.Collections.CreateAddAndClear.ConcurrentBag(Size: 512) 1.10 20308.54 22275.73
System.IO.Tests.Perf_StreamWriter.WriteFormat 1.10 169.05 185.39
System.Collections.IterateForEachNonGeneric.Hashtable(Size: 512) 1.10 10310.64 11302.34
System.Net.NetworkInformation.Tests.PhysicalAddressTests.PALong 1.10 586.90 642.93
System.Tests.Perf_Single.Parse(value: "12345") 1.10 73.07 80.02
System.Collections.TryGetValueFalse<String, String>.ImmutableSortedDictionary(Si 1.09 317408.70 347296.59
System.Collections.TryGetValueFalse<Int32, Int32>.ImmutableDictionary(Size: 512) 1.09 16353.44 17892.01 several?
System.Collections.ContainsTrue.ImmutableArray(Size: 512) 1.09 35197.62 38419.95
System.Memory.Slice.MemoryStartLengthSpan 1.09 10.03 10.94
System.Tests.Perf_String.Split(s: "A B C D E F G H I J K L M N O P Q R S T U V W 1.09 417.61 454.93
System.Tests.Perf_String.Contains(text: "This is a very nice sentence", value: " 1.09 163.57 178.14
System.Collections.ContainsTrue.SortedSet(Size: 512) 1.09 26794.71 29179.36
System.Linq.Tests.Perf_Enumerable.Reverse(input: IEnumerable) 1.09 1195.59 1300.14
Benchstone.BenchI.Permutate.Test 1.09 580158500.00 630657300.00
System.Collections.IterateForEach.List(Size: 512) 1.09 1979.56 2151.79
System.Memory.ReadOnlySequence.Slice_Start_And_EndPosition(Segment: Multiple) 1.09 12.12 13.18
System.Linq.Tests.Perf_Enumerable.EmptyTakeSelectToArray 1.09 23.97 26.01
PerfLabTests.LowLevelPerf.MeasureEvents 1.09 18030830.77 19568750.00
System.Tests.Perf_String.IndexerCheckLengthHoisting 1.08 34.14 37.03
PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj 1.08 439309.56 476198.30
System.Collections.CopyTo.ReadOnlySpan(Size: 2048) 1.08 507.44 549.58
System.Tests.Perf_Random.Next_int 1.08 9.75 10.56
System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 12345) 1.08 7.01 7.58
System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: I 1.08 542.53 587.10
System.Linq.Tests.Perf_Enumerable.ToList(input: ICollection) 1.08 64.69 69.98
System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512) 1.08 21.79 23.55 several?
System.Collections.CtorDefaultSize.SortedSet 1.08 4.90 5.29
System.Globalization.Tests.Perf_CompareInfo.IsSuffix(culture: ja-JP, source: "XX 1.08 11.90 12.84
System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512) 1.08 21.73 23.44
System.Linq.Tests.Perf_Enumerable.SelectToList(input: Range) 1.08 292.32 315.12
GuardedDevirtualization.ThreeClassVirtual.Call(testInput: pB=0.33 pD=0.00) 1.08 4.13 4.45
System.Threading.Tests.Perf_CancellationToken.Cancel 1.08 146.31 157.66
MicroBenchmarks.Serializers.Binary_FromStream.MessagePa 1.08 79276.46 85376.95
System.Collections.ContainsFalse.ImmutableHashSet(Size: 512) 1.07 37778.19 40582.55
System.Collections.Tests.Perf_BitArray.BitArrayLengthValueCtor(Size: 512) 1.07 17.26 18.54
System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing( 1.07 7046.62 7568.19 several?
System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_HostPort(host: "n 1.07 67.74 72.73 several?
System.Memory.Span.SequenceEqual(Size: 512) 1.07 19.07 20.44
System.Collections.IterateForEach.Dictionary(Size: 512) 1.07 2648.52 2837.22
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.07 937.53 1004.09
System.Tests.Perf_String.StartsWith(size: 100) 1.07 104.43 111.78
PerfLabTests.LowLevelPerf.InterfaceInterfaceMethod 1.07 3318103.36 3550551.25
System.Memory.Span.IndexOfAnyFourValues(Size: 512) 1.07 34.01 36.40
System.Globalization.Tests.Perf_CompareInfo.IsSuffix(culture: , source: "XXXXXXX 1.07 11.04 11.81
System.Memory.Constructors.ArrayAsSpan 1.07 11.86 12.68
System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 512, TotalSize: 200000) 1.07 1406144.06 1503116.67
MicroBenchmarks.Serializers.Json_FromString.Jil_ 1.07 298479.03 319025.80
System.Collections.AddGivenSize.IDictionary(Size: 512) 1.07 7107.35 7593.14
System.Globalization.Tests.Perf_CompareInfo.LastIndexOf(culture: , source: "foob 1.07 110.21 117.63
System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: 12345) 1.07 36.70 39.13
System.Tests.Perf_DateTime.ParseR 1.07 100.25 106.91
Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Hardware) 1.07 20600.13 21957.53
System.Net.Sockets.Tests.SocketSendReceivePerfTest.ReceiveAsyncThenSendAsync_Tas 1.07 112308450.00 119703900.00
System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests.ReadAsyncThenWrite 1.07 118.68 126.44
Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: Hardware) 1.07 20805.71 22166.65
GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.50) 1.06 5.37 5.71
GuardedDevirtualization.TwoClassInterface.Call(testInput: pB = 0.70) 1.06 5.42 5.76 several?
System.Collections.IterateForEach.IEnumerable(Size: 512) 1.06 2192.34 2326.10
System.Collections.ContainsTrue.ICollection(Size: 512) 1.06 35357.80 37477.94
System.Tests.Perf_UInt32.ParseSpan(value: "4294967295") 1.06 20.34 21.56
System.Linq.Tests.Perf_Enumerable.Zip(input: IEnumerable) 1.06 1521.33 1612.31
PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo 1.06 445205.96 471770.93
System.Tests.Perf_String.Compare(strings: ["Thé quick brown fox", "Thé quick BRO 1.06 5.96 6.32
System.Collections.AddGivenSize.Stack(Size: 512) 1.06 2190.63 2319.81
System.Collections.CtorFromCollection.ConcurrentStack(Size: 512) 1.06 5381.82 5682.71
System.Reflection.Activator.CreateInstanceType 1.05 29.30 30.89
System.Linq.Tests.Perf_Enumerable.AppendPrepend(input: IEnumerable) 1.04 4031.01 4211.75 several?
Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Collections.Concurrent.Count.Dictionary(Size: 512) 1.99 12103.26 6069.31
Benchstone.BenchF.Simpsn.Test 1.60 229020850.00 142927300.00
System.Numerics.Tests.Perf_Vector4.DistanceSquaredBenchmark 1.59 1.11 0.70
System.Numerics.Tests.Perf_Vector4.DistanceSquaredJitOptimizeCanaryBenchmark 1.53 1.07 0.70
System.Numerics.Tests.Perf_Vector4.DistanceBenchmark 1.46 1.03 0.71
System.Numerics.Tests.Perf_Vector3.LengthBenchmark 1.43 1.02 0.71
System.Tests.Perf_String.Trim(s: " Te st ") 1.41 23.36 16.52 bimodal
System.Memory.Span.Clear(Size: 512) 1.35 31.03 23.05
System.Collections.Concurrent.IsEmpty.Bag(Size: 0) 1.27 15.69 12.32 several?
System.Collections.TryGetValueTrue<String, String>.SortedList(Size: 512) 1.27 286087.45 225580.45
System.Collections.AddGivenSize.Queue(Size: 512) 1.27 3113.05 2457.27
System.Collections.AddGivenSize.HashSet(Size: 512) 1.26 12157.97 9638.50
System.Tests.Perf_String.TrimStart(s: "Test") 1.24 4.41 3.55
System.Collections.Concurrent.IsEmpty.Dictionary(Size: 512) 1.24 2.74 2.21
System.Collections.ContainsFalse.ImmutableArray(Size: 512) 1.22 1053724.96 862075.99
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.22 159.77 130.83
System.Memory.Span.Fill(Size: 512) 1.22 21.74 17.86
System.Tests.Perf_Array.ArrayCopy3D 1.21 301.55 248.38
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.21 156.76 129.14
System.Tests.Perf_UInt16.TryParse(value: "12345") 1.21 16.92 14.02
System.Collections.CopyTo.Span(Size: 2048) 1.20 584.43 485.80
System.Tests.Perf_UInt16.TryParse(value: "0") 1.20 12.45 10.36
System.Collections.TryGetValueFalse<Int32, Int32>.IDictionary(Size: 512) 1.20 6952.27 5784.65
System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii") 1.20 24.08 20.06
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.20 155.28 129.45
System.Collections.CopyTo.Array(Size: 2048) 1.19 561.08 469.70
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.19 154.24 129.35
System.Drawing.Tests.Perf_Graphics_DrawBeziers.DrawBezier_Points 1.19 1952.38 1638.48
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.19 154.55 129.78
System.Tests.Perf_String.TrimEnd(s: "Test") 1.19 4.88 4.12
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.18 153.85 129.91
System.Tests.Perf_String.Contains(text: "This is a very nice sentence", value: " 1.17 13.48 11.54
System.Tests.Perf_Int32.TryParse(value: "4") 1.16 12.15 10.43
System.Drawing.Tests.Perf_Color.FromArgb_AlphaColor 1.16 5.56 4.78
System.Collections.IterateForEach.ConcurrentQueue(Size: 512) 1.16 5520.88 4745.06
System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8") 1.16 31.98 27.49
PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod 1.16 177614.30 152943.81
System.Buffers.Tests.ReadOnlySequenceTests.FirstTenSegments 1.16 6.29 5.42
System.Collections.Sort.LinqQuery(Size: 512) 1.16 372426.33 321968.39
System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldBE 1.16 16.22 14.02 several?
System.Tests.Perf_UInt32.TryParse(value: "0") 1.16 11.64 10.06
System.Collections.ContainsFalse.ICollection(Size: 512) 1.16 953500.43 825229.61
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSo 1.15 49.55 42.91
System.Collections.ContainsFalse.List(Size: 512) 1.15 990982.64 860970.72
System.Collections.IterateForEach.ImmutableQueue(Size: 512) 1.15 5110.91 4445.46
System.Tests.Perf_UInt64.ParseSpan(value: "12345") 1.15 20.21 17.59
System.Collections.CtorDefaultSize.ConcurrentStack 1.15 3.10 2.71
System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [143, 2 1.15 6.03 5.26
System.Security.Cryptography.Primitives.Tests.Performance.Perf_FixedTimeEquals.F 1.15 153.73 134.18
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSo 1.15 57.49 50.19
System.Collections.CreateAddAndRemove.HashSet(Size: 512) 1.15 26408.74 23055.65
System.Memory.Span.Clear(Size: 512) 1.14 22.06 19.31 several?
System.Tests.Perf_String.Trim(s: "Test") 1.13 6.06 5.35
System.Collections.ContainsTrue.Queue(Size: 512) 1.13 38493.35 34030.88
System.Collections.IterateFor.ImmutableSortedSet(Size: 512) 1.13 13310.56 11776.46
System.Threading.Tests.Perf_ThreadStatic.GetThreadStatic 1.13 2.87 2.54
System.MathBenchmarks.Double.Exp 1.13 36254.95 32129.32
System.Tests.Perf_Array.IndexOfShort 1.13 11.45 10.15
System.Buffers.Tests.RentReturnArrayPoolTests.SingleParallel(RentalSize: 4 1.13 91.43 81.19
System.Collections.CtorFromCollection.ImmutableSortedDictionary(Size: 51 1.13 375994.97 333964.44
System.Collections.CtorFromCollection.Dictionary(Size: 512) 1.13 7927.33 7041.75
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345) 1.13 8.08 7.18
System.Tests.Perf_Byte.TryParse(value: "0") 1.13 11.58 10.29
System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 0) 1.12 6.05 5.38 several?
System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: List) 1.12 434.94 387.03
System.Tests.Perf_Int32.ParseHex(value: "7FFFFFFF") 1.12 22.46 19.99 several?
System.Collections.CtorDefaultSize.ImmutableArray 1.12 6.26 5.58
System.Collections.IterateForEach.Stack(Size: 512) 1.12 1766.74 1575.85
System.Collections.Concurrent.IsEmpty.Queue(Size: 0) 1.12 6.40 5.71
System.Tests.Perf_Int32.Parse(value: "12345") 1.12 16.78 14.97
System.Tests.Perf_Int32.ParseSpan(value: "2147483647") 1.12 23.53 21.00
System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests.WriteAsyncThenRead 1.12 84.10 75.09
System.Tests.Perf_String.IndexOfAny 1.12 14.78 13.21 several?
System.Buffers.Tests.RentReturnArrayPoolTests.SingleSerial(RentalSize: 409 1.12 35.34 31.59
System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert: 1.12 128.88 115.23
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_Config 1.12 41.25 36.89
System.Collections.IndexerSet.Dictionary(Size: 512) 1.11 6394.40 5743.69
System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired 1.11 169.64 152.39
System.Collections.Sort.Array_Comparison(Size: 512) 1.11 314825.89 283355.13
System.Tests.Perf_UInt64.Parse(value: "12345") 1.11 16.72 15.07
System.Threading.Tasks.ValueTaskPerfTest.Await_FromCompletedTask 1.11 31.27 28.19
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 1.11 1392.59 1258.67
System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: List) 1.11 398.86 360.87
System.Collections.IterateForEach.ImmutableList(Size: 512) 1.11 44633.66 40386.98
System.Linq.Tests.Perf_Enumerable.OrderByThenBy(input: IEnumerable) 1.10 4988.15 4519.03
System.Collections.Concurrent.IsEmpty.Bag(Size: 0) 1.10 13.61 12.35
System.Collections.CtorDefaultSize.ImmutableQueue 1.10 4.78 4.34 several?
System.Tests.Perf_Array.ArrayRetrieve2D 1.10 2308026.95 2098529.30 several?
System.Collections.CtorFromCollection.ImmutableList(Size: 512) 1.10 13519.12 12301.63
System.Text.Json.Tests.Perf_Booleans.WriteBooleans(Formatted: True, SkipValidati 1.10 2416131.16 2198617.86
System.Collections.ContainsTrue.Span(Size: 512) 1.10 34526.20 31502.44
System.Collections.IterateForEach.ImmutableQueue(Size: 512) 1.10 4312.09 3936.15
System.Text.Json.Tests.Perf_Ctor.Ctor(Formatted: True, SkipValidation: False) 1.10 24.02 21.94
System.Collections.IterateForEach.ConcurrentStack(Size: 512) 1.09 3014.02 2754.80
System.Memory.Slice.SpanStartLength 1.09 5.44 4.97 several?
System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEn 1.09 585.43 535.37
System.Linq.Tests.Perf_Enumerable.SelectToArray(input: List) 1.09 255.41 233.77
SeekUnroll.Test(boxedIndex: 19) 1.09 1656663500.00 1517181700.00
PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodLongHierarchy 1.09 360098.23 329940.01 bimodal
System.Collections.CreateAddAndClear.Span(Size: 512) 1.09 1674.77 1534.79
System.IO.Tests.Perf_StreamWriter.WritePartialCharArray(writeLength: 100) 1.09 461206900.00 422663700.00
System.Tests.Perf_Int32.ParseHex(value: "80000000") 1.09 21.88 20.06
System.Collections.CtorDefaultSize.SortedList 1.09 10.53 9.67
System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512) 1.09 6962.87 6399.79
System.Collections.ContainsTrue.Queue(Size: 512) 1.09 474541.43 436887.67 multimodal
PerfLabTests.DelegatePerf.MulticastDelegateCombineInvoke 1.09 165848500.00 152800700.00
System.Collections.ContainsFalse.Stack(Size: 512) 1.09 966752.13 890788.24
System.Collections.IndexerSet.IList(Size: 512) 1.08 7063.79 6512.08
System.Collections.CopyTo.Array(Size: 2048) 1.08 159.97 147.53
System.Buffers.Tests.ReadOnlySequenceTests.SliceTenSegments 1.08 32.56 30.06
System.Memory.ReadOnlySequence.Slice_Repeat_StartPosition_And_EndPosition(Segmen 1.08 55.52 51.33
System.Collections.AddGivenSize.List(Size: 512) 1.08 1803.43 1668.10
System.Memory.ReadOnlyMemory.ToArray(Size: 512) 1.08 86.81 80.37
System.Text.Json.Tests.Perf_Ctor.Ctor(Formatted: False, SkipValidation: False) 1.08 23.65 21.89
System.Collections.CtorFromCollectionNonGeneric.Stack(Size: 512) 1.08 10603.25 9821.36
LinqBenchmarks.Order00ManualX 1.08 155893350.00 144747200.00
System.Collections.CtorFromCollectionNonGeneric.Stack(Size: 512) 1.08 31676.41 29417.04
System.Tests.Perf_Guid.ParseExactD 1.08 88.28 81.99
System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSiz 1.07 2272.50 2115.58
System.Text.Json.Tests.Perf_Ctor.Ctor(Formatted: False, SkipValidation: True) 1.07 23.84 22.24
System.Buffers.Text.Tests.Utf8ParserTests.TryParseSByte(value: -128) 1.07 7.11 6.64
System.Text.Json.Tests.Perf_Ctor.Ctor(Formatted: True, SkipValidation: True) 1.07 23.72 22.18
System.Tests.Perf_UInt32.Parse(value: "12345") 1.07 16.08 15.04
MicroBenchmarks.Serializers.Xml_FromStream.Xml 1.07 2598.55 2430.62 several?
System.Tests.Perf_String.Trim(s: "Test ") 1.07 15.11 14.14
System.Linq.Tests.Perf_Enumerable.Except(input: IEnumerable) 1.07 3677.96 3442.40
System.Collections.ContainsFalse.ImmutableSortedSet(Size: 512) 1.07 313652.16 293580.17
SeekUnroll.Test(boxedIndex: 27) 1.07 2156615600.00 2019078300.00
System.Collections.ContainsTrue.HashSet(Size: 512) 1.07 6987.89 6550.29
System.Collections.TryGetValueFalse<String, String>.ImmutableDictionary(Size: 51 1.07 31943.18 29944.76
System.Tests.Perf_String.TrimEnd(s: "Test ") 1.07 13.14 12.33
System.Tests.Perf_Byte.TryParse(value: "255") 1.07 13.18 12.37
System.Memory.Slice.ReadOnlyMemoryStart 1.06 5.74 5.39
System.Tests.Perf_String.ToUpper(s: "TEST") 1.06 11.13 10.46
System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 1844674407370955 1.06 20.73 19.48 several?
System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: -2147483648) 1.06 86.58 81.42
System.Threading.Tests.Perf_CancellationToken.CreateLinkedTokenSource3 1.06 173.62 163.36
System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615") 1.06 35.92 33.83
System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32 1.06 20.45 19.26
MicroBenchmarks.Serializers.Json_FromStream.JsonNet_ 1.06 78639.60 74153.97
System.Collections.ContainsKeyTrue<String, String>.SortedDictionary(Size: 512) 1.06 294712.14 277988.17
System.Collections.CreateAddAndClear.Dictionary(Size: 512) 1.06 13283.24 12531.58
System.Memory.Span.IndexOfValue(Size: 512) 1.06 9.82 9.27
System.Memory.ReadOnlySpan.Trim(input: "abcdefg") 1.06 5.79 5.46
System.Collections.IterateForEach.SortedDictionary(Size: 512) 1.06 9570.64 9047.09
LinqBenchmarks.Where00ForX 1.06 454666000.00 429885150.00
System.Collections.CtorDefaultSize.List 1.06 10.97 10.37

@michellemcdaniel michellemcdaniel merged commit fe1c355 into dotnet:master Aug 14, 2019
@adamsitnik adamsitnik deleted the updateBDN branch August 15, 2019 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@billwert billwert Awaiting requested review from billwert

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adamsitnik @billwert @michellemcdaniel