Improving P95+ latency

[richlander]

Improving P95+ latency

We see an increasing number of large internet-facing sites and services being hosted on .NET. While there is a lot of legitimate focus on the requests per second (RPS) metric, we find that very few big site owners ask us about that or even need to satisfy throughput approaching 1000 RPS (1000 RPS == 86M requests per day). We hear a lot more about latency, specifically about improving P95 or P99 latency. Often, the number of machines or cores that are provisioned for (and biggest cost driver of) a site are chosen based on achieving a specific P95 metric (for example, 200ms), as opposed to a lower P50 metric (for example, 50ms). We think of latency as being the true “money metric”.

We want .NET to be a platform that makes it cheaper to host your applications with each new release. In order to achieve that, it's important that latency is both good (low) but also predictable. That's the whole point of measuring P95+ latency. We have increased our focus on predictably consistent performance, reducing performance cliffs and outliers, with an emphasis on P95+ latency.

An equally important theme is predictable performance. Some of these epics are more related to that, then specifically targeting P95 latency. We're going to be a bit lazy here, and mix the two topics.

GC

• Card mark stealing for better work balance in Server GC
• Optimize decommitting GC heap memory pages
• Pinned object heap to reduce heap fragmentation caused by pinning
  Reduce GC pause times in specific situations, like Array.Copy, Array.Sort or object unboxing

Runtime

• Improve call counting mechanism used by tiered JIT compilation to smooth out performance during startup
• Casting in a loop may cause long GC pause times
• Dynamic expansion of internal generic dictionary that eliminate performance cliffs hit by generic code
• Fix for solving lock contention issue in GC statics scanning
• GC polling in unboxing JIT helpers
• Buffer::BlockCopy may spend too long without GC polling
• Calling System.Math floating point operations in a loop causes long GC pause times
• Inlined GC poll for methods marked with SuppressGCTransitionAttribute

Libraries

• Very high latency for GC when using (lots of) ThreadLocal

[danmoseley]

All that remains here for 5.0 of list above is the issue fixed by #39474. That may or may not make 5.0.

[danmoseley]

Closing as we have completed this for 5.0.