Investigate why crossgen works slower with TieredCompilation/PGO

[EgorBo]

I was measuring crossgen2.exe -O SPC.dll (actually, the exact command we use for build Clr.NativeCoreLib -c Release) and noticed a few problems:

Mode	Time to prejit SPC.dll, seconds
TC=1 (Default)	4.81
TC=0	4.25
TC=1, CCDelayMS=0	3.78
TC=1, PGO=1	5.29
TC=1, PGO=1, CCDelayMS=0	3.93

Legend:

• TC - DOTNET_TieredCompilation (1 by default)
• PGO - DOTNET_TieredPGO (0 by default)
• CCDelayMS - DOTNET_TC_CallCountThreshold (100 by default)

The difference is quite noticeable so worth investigating - numbers are quite stable across multiple runs.
Judging by the effect from DOTNET_TC_CallCountThreshold we're having some contention for call counting stub installation/promotion to tier1.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I was measuring crossgen2.exe -O SPC.dll (actually, the exact command we use for build Clr.NativeCoreLib -c Release) and noticed a few problems:

Mode	Time to prejit SPC.dll, seconds
TC=0	4.23
TC=1	4.71
TC=1, PGO=1	5.29

The difference is quite noticeable so worth investigating.
Few observations so far - it seems there is a huge benefit from increasing call counting threshold for R2R'd code to e.g 1000 - will file a PR with that since @davidwrighton made IsReadyToRun(PCODE) VM API cheap now. It's needed because we don't want to re-jit R2R to InstrumentedTier too early.

Investigating this in VTune now, e.g. here is a VTune comparison for TC=1,PGO=1 vs TC=1,PGO=0:

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-

[EgorBo]

Fun-fact: DOTNET_TC_CallCountingDelayMs=1 makes TC=1 (default) faster than TC=0. So apparently there is a huge contention to install call counting stubs

[EgorBo]

cc @noahfalk @kouvel

[kunalspathak]

From https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362_RunKind%3dcrossgen_scenarios%2fCrossgen2%20Throughput%20-%20Single%20-%20System.Private.CoreLib.html

The regression is seen after enabling dynamic PGO in #86225

@davidwrighton @mangod9

[AndyAyersMS]

As mentioned offline we also ought to start measuring with the NAOT'd crossgen2.

[MichalStrehovsky]

#89489 disabled tiering to work around (matching the workaround used in ILC already) so if we still do TP measurements in the non-shipping configuration of crossgen2, there's going to be an improvement.