Dynamic PGO startup improvements in NET 8

[EgorBo]

Meta issue to track improvements around startup for Dynamic PGO (Non-PGO might also benefit from some of these), I assume it makes sense to track them separately from the general "Dynamic PGO Improvements" issue to leave that for general/CQ improvements.

Planned for .NET 8

• Instrument only hot Tier0 and R2R code PGO: Add new tiers #70941
• General startup improvements
  • Emit less jump-stubs on x64 #77639

Planned for Jan-Mar'23

• Reduce startup overhead from PGO
  • PGO: Reduce number of block counters #81731
  • Optimize JIT_ClassProfile32 #82014
  • Don't instrument methods with [Intrinsic] #81983
  • Remove CPU cache contention for instrumented code #81932
  • JIT: in minimal profiling mode, ignore impact of conditional throws #81972
  • JIT: scalable profile counter mode #84427
• JIT: consider running some optimizations in Tier0 #9120
  • [PR] Introduce OptLevel() in jit #77465
  • Enable more intrinsics for Tier0 #83002
  • [PR] Allow some intrinsics in Tier0 #77357
  • Enable constant folding in Tier0 #82412
• Measure startup time of real world applications with dynamic PGO
  • Determine the actual apps to use
  • Run the tests and gather startup time data
  • Analyze the data

Under consideration

• Prioritize methods in the promotion queue.
  • Don't delay call-counting-installation because of R2R'd code #77635
    -- Pending on David Wrighton's API work Fix the multicore performance of mapping from an instruction pointer to a MethodDesc #79021
• Improve CallCountingStubs Make call-counting, class probes, block counters cache-friendly #72387
  • Put cells inside CallCountingStubs to avoid double indirect loads Make call-counting, class probes, block counters cache-friendly #72387 (comment)
  • Consider improving mutlithread/NUMA experience where multiple threads poll the same location of memory at the same time
• Tiered JIT: redundant compilations #76402 Inline tiny methods in Tier0 too to avoid redundant compilations for simple/auto properties and don't occupy a seat in the promotion queue. A fairly simple prototype for this reduced number of jitted functions from 250k to 200k for BingSNR. We can start from allowing inlining for empty or return CNS kinds of methods.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Meta issue to track improvements around startup for Dynamic PGO (Non-PGO might also benefit from some of these), I assume it makes sense to track them separately from the general "Dynamic PGO Improvements" issue to leave that for general/CQ improvements.

Planned for .NET 8

• Allow some optimizations in Tier0 which result in TP improvements: enable simple intrinsic expansion in Tier0, enable optimizations which might help to reduce number of class probes we emit during instrumentation

• Fix issues in optimized + instrumented code, enable edge profiling

• Instrument only hot Tier0 and R2R code PGO: Add new tiers #70941
  Pros:

  • We don't waste time/memory on instrumentation for methods which will never be promoted (a general rule that 75% of methods never make it to Tier1)
  • We no longer give up on R2R code and it also benefits from Dynamic PGO - this resulted in nice wins for TE benchmarks

  Cons:

  • Increased working set: as it turns out it's not really an issue. Even for a huge BingSNR service Tier1 contributes 10Mb (+stubs) to the working set which is ~8Gb
  • Increased number of methods waiting in the promotion queue (see below)

• Prioritize methods in the promotion queue. We recently found out that Dynamic PGO suffers a lot if hot methods with loops aren't placed on top of the queue for quicker promotion so such methods stuck in tier0 with slow class probes inside loops for too long - here is a good example: Optimize CheckSample #76520 (comment)

  • Give methods with loops higher priority
  • Give super small methods and R2R lower priority
  • Keep counting calls after they reach 30 to rely on that for priority - Optimize CheckSample #76520 (comment)

• Inline tiny methods in Tier0 too to avoid redundant compilations for simple/auto properties and don't occupy a seat in the promotion queue. A fairly simple prototype for this reduced number of jitted functions from 250k to 200k for BingSNR. We can start from allowing inlining for empty or return CNS kinds of methods.

Under consideration

• Improve CallCountingStubs Make call-counting, class probes, block counters cache-friendly #72387
  • Put cells inside CallCountingStubs to avoid double indirect loads Make call-counting, class probes, block counters cache-friendly #72387 (comment)
  • Consider improving mutlithread/NUMA experience where multiple threads poll the same location of memory at the same time

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr, User Story
Milestone:	8.0.0

[EgorBo]

Current state:

Json benchmark seems pretty good while Platform-plaintext suffers a lot from instrumentation (10x slower)

[EgorBo]

We've finished the work we planned to do for .NET 8.0 here, the rest is the "best effort"