FOH (Frozen Object Heap) work items #76151

EgorBo · 2022-09-25T12:38:21Z

dotnet-issue-labeler · 2022-09-25T12:38:24Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2022-09-25T12:40:21Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

.NET 8.0

Allocate string literals on FOH Allocate string literals on frozen segments #49576
Allocate Type objects on FOH Allocate RuntimeType objects on Frozen Object Heap #75573
- NativeAOT support
Optimize static readonly fields holding frozen objects to const handles JIT: import static readonly fields holding frozen objects as const handles #76112
Don't use write-barriers for FOH objects JIT: Don't use write barriers for frozen objects #76135
Allocate Array.Empty<> on FOH
static field boxes Allocate RuntimeType objects on Frozen Object Heap #75573 (comment)
Diagnostic support Diagnostic support for NonGC heap diagnostics#4156 (and thread in Allocate string literals on frozen segments #49576)
Testing on real-world workloads (1P/3P)
- How many objects end up in FOH, what's the ratio between SOH and FOH typically.
- Standalone GC
- Hard heap limit
- Performance impact/savings

Potential good/bad ideas

Always allocate new T[0] on FOH. Technically, it violates ECMA but we already do the same for:

bool same = ReferenceEquals(new string(new char[0]), new string(new char[0])); // true

However, it's unlikely that it's worth the potential risks.

Optimize str == "" as just cmp reg, 0xFOHREFERENCE - it turns out that it's still possible to produce new empty strings
Cache boxed objects and allocated them on FOH (e.g. boxed True, boxed False like these)
So, for example:

object o = myInt;

will be optimized in codegen to:

if ((uint)myInt < 100)
    o = 0xBAADF00D (/*boxed zero on FOH*/ * myInt * 24
else
    o = box(myInt)

Same for booleans, etc. Can't find the related issue but AFAIR it raised some ECMA-related concerns (around box to always emit a new object)

Author:	EgorBo
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	8.0.0

jkotas · 2022-09-25T13:09:28Z

Potential good/bad ideas

Same ideas have been discussed multiple times. E.g.: #7079 (comment) . They have been always punted so far.

(These ideas are not FOH specific. They can be implemented without FOH by paying for one extra indirection that is a minuscule cost.)

stephentoub · 2022-09-25T13:11:18Z

Always allocate new T[0] on FOH

We ourselves have places we don't use Array.Empty because the use is relying on reference equality/inequality.

EgorBo · 2022-09-25T13:43:04Z

We ourselves have places we don't use Array.Empty because the use is relying on reference equality/inequality.

Thanks, good to know, updated the item.

Same ideas have been discussed multiple times. E.g.: #7079 (comment) . They have been always punted so far.

Right, it's just that since we now have FOH we don't need to fragmentate POH/SOH with that pinned cache. I agree that the idea is dangerous, although I still think it could be an opt-in feature for some large projects (e.g. for those who had to do it by hands today), the implementation should be relatively simple.

jkotas · 2022-09-25T14:06:45Z

I still think it could be an opt-in feature for some large projects

Doubt it. Dangerous opt-ins do not work for large projects. It is impossible (prohibitively expensive) to audit their code to validate that it is safe to opt-in.

EgorBo · 2022-09-25T14:07:29Z

I still think it could be an opt-in feature for some large projects

Doubt it. Dangerous opt-ins do not work for large projects. It is impossible (prohibitively expensive) to audit their code to validate that it is safe to opt-in.

fair enough!

GSPP · 2022-09-28T13:25:58Z

Optimize str == "" as just cmp reg, 0xFOHREFERENCE - it turns out that it's still possible to produce new empty strings so

I'm curious how that is possible. If memory serves correctly, @jkotas stated on this issue tracker a long time ago that the policy is to have the empty string be a singleton.

EgorBo · 2022-09-28T14:53:39Z

Optimize str == "" as just cmp reg, 0xFOHREFERENCE - it turns out that it's still possible to produce new empty strings so

I'm curious how that is possible. If memory serves correctly, @jkotas stated on this issue tracker a long time ago that the policy is to have the empty string be a singleton.

That is exactly what is written in that item, isn't? I just decided to put some ideas from the past to, probably, inspire for new ones.

jkotas · 2022-09-28T15:37:41Z

I'm curious how that is possible.

For example:

var s = string.Copy("");

Console.WriteLine(s == "");
Console.WriteLine(ReferenceEquals(s, ""));

EgorBo · 2022-09-28T15:47:00Z

Ah, the question was "how is it possible to create a unique empty string" and I read it as "how the optimization you propose is possible" 🙂

EgorBo · 2022-10-04T16:40:36Z

Idea: try to reserve memory for the first segment (4mb) next to coreclr in memory, the way we do currently for 4Gb for loader heap, I guess it won't hurt if we first try to reserve 4mb and then those 4Gb.

EgorBo · 2022-10-06T21:04:31Z

@jkotas am I correct that we can handle static readonly object SyncObj = new object(); (common idiom for synchronization objects) in JIT without too much efforts?

If we're in a static constructor and see pattern like

newobj instance void [System.Runtime]System.Object::.ctor()
stsfld object myReadonField

we can replace newobj with a helper that allocates on FOH, right?

ah, I assume only if constructor doesn't touch that field any more (doesn't try to set it multiple times)

EgorBo · 2022-10-06T21:19:19Z

Same for empty (or any size) arrays:

ldc.i4.0
newarr [System.Runtime]System.Int32
stsfld int32[] C::x

so we won't have to special-case Array.Empty.

So the strategy will be:

Import a cctor normally in JIT, make a list of static readonly fields which are set only once (and aren't address exposed/escape)
Check that cctor doesn't have backward branches (potential loops)
Replace normal allocator helpers with frozen ones.

Or actually, as an initial impl, it can be just:

stsfld
ret

jkotas · 2022-10-06T22:17:29Z

You can do this for non-collectible code only. Also, you can do the same thing for arrays of length 0. It would cover the Array.Empty case without special casing.

Or actually, as an initial impl, it can be just:

I am not sure what you mean by this.

EgorBo · 2022-10-06T22:39:41Z

I am not sure what you mean by this.

I meant that if a static constructor is complicated then

newarr/newobj
stsfld

pattern might be set in a loop so we'll end up allocating immortal frozen object each iteration, etc. So I mean that if we see

newarr/newobj
stsfld
ret

then it's definitely just a simple cctor that only sets one field 🙂 - I meant it as an initial quick/safe implementation. Array.Empty falls into that category

EgorBo · 2022-10-08T00:18:43Z

Idea: make pinning no-op for frozen objects since we plan to move static readonly arrays there.

MichalPetryka · 2022-10-08T00:55:46Z

Idea: provide higher alignment for arrays on FOH in order to achieve higher perf with SIMD operations.

EgorBo · 2022-10-08T01:32:29Z

Idea: provide higher alignment for arrays on FOH in order to achieve higher perf with SIMD operations.

Definitely will require changes on GC side where gc assumes fixed alignment for all objects when it iterates them, e.g:

void gc_heap::seg_set_mark_bits (heap_segment* seg)
{
    uint8_t* o = heap_segment_mem (seg);
    while (o < heap_segment_allocated (seg))
    {
        set_marked (o);
        o = o + Align (size(o));
    }
}

jkotas · 2022-10-08T03:33:01Z

I would also say that this is very niche. NativeMemory.AlignedAlloc and GC.AllocateArray variant that allocates on POH with specific alignment should sufficiently cover this space.

Note that the embedding of the object references from readonly fields can be also extended to pinned objects allocated on POH.

readonly static int[] a = GC.AllocateArray(..., pinned: true); // This can be treated the same way as if the array was allocated on FOH

EgorBo · 2022-10-08T08:11:26Z

You can do this for non-collectible code only

@jkotas do I understand it correctly that in order to detect that, I need to add an additional argument to my custom alloc helper - StackCrawlMarkHandle to then get Assembly object inside the alloc via stack walking and find out whether it's collectible or not?

Basically, call Assembly.GetExecutingAssembly().IsCollectible.

However, it's probably easier to just pass cctor's type handle to that helper

UPD: we don't support tiered compilation for collectible assemblies so in theory we can emit custom allocs only for Tier1 (promoted from tier0). But that might be not future-proof.

jkotas · 2022-10-08T11:43:47Z

Stack crawling to find caller is an anti-pattern. You should do it all at JIT time, something like:

Add CORJIT_FLAG_FROZEN_ALLOCATIONS_ALLOWED that is set for cctors in non-collectible code
Add CORINFO_HELP_NEW_FROZEN and CORINFO_HELP_NEWARRAY_1_FROZEN JIT helpers that the JIT replaces allocations eligible for freezing with.
Add READYTORUN_HELPER_NewFrozenObject and READYTORUN_HELPER_NewFrozenArray helpers for R2R that map to the above JIT helpers, so that R2R code does this optimization too and allows the eventual tiered-compilation to benefit.

MichalPetryka · 2022-10-08T12:03:21Z

Stack crawling to find caller is an anti-pattern. You should do it all at JIT time, something like:

Add CORJIT_FLAG_FROZEN_ALLOCATIONS_ALLOWED that is set for cctors in non-collectible code

Add CORINFO_HELP_NEW_FROZEN and CORINFO_HELP_NEWARRAY_1_FROZEN JIT helpers that the JIT replaces allocations eligible for freezing with.

Add READYTORUN_HELPER_NewFrozenObject and READYTORUN_HELPER_NewFrozenArray helpers for R2R that map to the above JIT helpers, so that R2R code does this optimization too and allows the eventual tiered-compilation to benefit.

Can't R2R assemblies be loaded into collectible ALCs?

jkotas · 2022-10-08T12:12:25Z

Can't R2R assemblies be loaded into collectible ALCs?

They cannot today:

runtime/src/coreclr/vm/readytoruninfo.cpp

Line 527 in fec0221

DoLog("Ready to Run disabled - collectible module");

. And even if it was allowed, this scheme would be still compatible with it. We would substitute the frozen allocation helper with a regular allocation helper when loading the R2R image into collectible context.

EgorBo · 2022-10-10T22:41:29Z

@jkotas @Maoni0 you asked for numbers - I downloaded BingSNR, updated to net8 and ran its benchmarks locally with bombardier (that simulates load):
338000 objects were allocated on FOH, total size is 17Mb (=5 frozen segments since size of a segment is hard-coded to 4Mb).

And here are the general performance counters:

[System.Runtime]
    % Time in GC since last GC (%)                                 1
    Allocation Rate (B / 1 sec)                                8,160
    CPU Usage (%)                                                  0
    Exception Count (Count / 1 sec)                                0
    GC Committed Bytes (MB)                                    2,587.238
    GC Fragmentation (%)                                           5.389
    GC Heap Size (MB)                                          2,033.518
    Gen 0 GC Count (Count / 1 sec)                                 0
    Gen 0 Size (B)                                         8,245,392
    Gen 1 GC Count (Count / 1 sec)                                 0
    Gen 1 Size (B)                                        17,739,352
    Gen 2 GC Count (Count / 1 sec)                                 0
    Gen 2 Size (B)                                            1.4631e+09
    IL Bytes Jitted (B)                                   21,699,370
    LOH Size (B)                                              6.1376e+08
    Monitor Lock Contention Count (Count / 1 sec)                  0
    Number of Active Timers                                       14
    Number of Assemblies Loaded                                5,167
    Number of Methods Jitted                                 237,451
    POH (Pinned Object Heap) Size (B)                      2,897,848
    ThreadPool Completed Work Item Count (Count / 1 sec)           6
    ThreadPool Queue Length                                        0
    ThreadPool Thread Count                                        3
    Time spent in JIT (ms / 1 sec)                                 0
    Working Set (MB)                                           7,083.553

My env vars in setup.bat:

REM fix for .NET 7 to use segments instead of regions due to overcommit in net7rc1
SET xxCOMPlus_GCName=clrgc.dll
SET DOTNET_ReadyToRun=0
SET DOTNET_TieredCompilation=1

NOTE: I commented GCName to enable gc regions (just to test net8).

jkotas · 2022-10-10T22:48:59Z

total size is 17Mb (=5 frozen segments since size of a segment is hard-coded to 4Mb).

Should we double the frozen segment reservations (ie 4mb, 8mb, 16mb, ...) to have a fewer of these?

EgorBo · 2022-10-10T22:51:22Z

total size is 17Mb (=5 frozen segments since size of a segment is hard-coded to 4Mb).

Should we double the frozen segment reservations (ie 4mb, 8mb, 16mb, ...) to have a fewer of these?

Makes sense!

Btw, it's impressive how much GC Regions save for the service, GC Heap with regions: 2Gb, without 3.5-4.5Gb

jkotas · 2022-10-10T22:53:03Z

SET DOTNET_ReadyToRun=0

I do not think that Bing is disabling R2R. Was there a particular reason you have disabled it?

EgorBo · 2022-10-10T22:55:02Z

SET DOTNET_ReadyToRun=0

I do not think that Bing is disabling R2R. Was there a particular reason you have disabled it?

It was an unrelated experiment - just wanted to see how much methods it jits in total, the biggest number I saw was 238k methods (21.8Mb IL Jitted) - now plan to check the actual size of native code in memory.

Maoni0 · 2022-10-10T23:59:46Z

thanks for the data!

Btw, it's impressive how much GC Regions save for the service, GC Heap with regions: 2Gb, without 3.5-4.5Gb

♥

I would also say that this is very niche. NativeMemory.AlignedAlloc and GC.AllocateArray variant that allocates on POH with specific alignment should sufficiently cover this space.

exactly, you don't need the GC support for this.

fewer FOH segs would be good.

EgorBo · 2022-11-02T17:55:35Z

More insights from the bing service - among those 17Mb of objects in the FOH, 9mb of them are RuntimeType and 8mb are string literals, the longest string literal is 24kb

stephentoub · 2022-11-02T18:00:04Z

the longest string literal is 24kb

So... curious...

Some sort of lookup table?

EgorBo · 2022-11-02T18:24:01Z

the longest string literal is 24kb

So... curious...

Some sort of lookup table?

The content looks like a huge set of coordinates (comma-separated floating points) for that one 🙂

mjsabby · 2022-11-05T16:28:35Z

I'm familiar with this service, and the FOH should be significantly larger than 17MB, it should be in the order of ~800MB or so. It contains all the RESX strings, and significant other object graphs. Perhaps we can chat offline to see why you're not seeing that size.

One question I did have is, will this collection of work regress performance of file-backed segments in the FOH?

EgorBo · 2022-11-05T17:29:34Z

@mjsabby

I'm familiar with this service, and the FOH should be significantly larger than 17MB

I was only counting objects in FOH segments added by runtime. Potentially, for your service we can surface a configuration knob to set the initial size e.g. 20Mb to have only just one extra frozen segment (currently it's 3).

One question I did have is, will this collection of work regress performance of file-backed segments in the FOH?

I'd not expect any regressions. Perhaps, only if you're not on GC Regions as we had to enable scanning for in-range FOH segments there (#76251)

EgorBo · 2023-06-09T15:57:21Z

All items except dotnet/diagnostics#4156 are closed, so closing it for .NET 8.0 and leave dotnet/diagnostics#4156 open (working on it now)

ghost added the untriaged New issue has not been triaged by the area owner label Sep 25, 2022

EgorBo added this to the 8.0.0 milestone Sep 25, 2022

ghost removed the untriaged New issue has not been triaged by the area owner label Sep 25, 2022

EgorBo added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 25, 2022

EgorBo self-assigned this Sep 25, 2022

EgorBo mentioned this issue Oct 11, 2022

Double the size of new frozen segments #76870

Merged

EgorBo mentioned this issue Nov 1, 2022

Allocate boxed static structs on Frozen Object Heap #77737

Merged

EgorBo mentioned this issue Apr 29, 2023

Allocate Array.Empty<> on a frozen segment (NonGC heap) #85559

Merged

jkotas mentioned this issue Aug 14, 2023

Diagnostic support for NonGC heap dotnet/diagnostics#4156

Open

21 tasks

EgorBo closed this as completed Jun 9, 2023

ghost locked as resolved and limited conversation to collaborators Jul 9, 2023

FOH (Frozen Object Heap) work items #76151

FOH (Frozen Object Heap) work items #76151

Comments

EgorBo commented Sep 25, 2022 • edited Loading

.NET 8.0

dotnet-issue-labeler bot commented Sep 25, 2022

ghost commented Sep 25, 2022

.NET 8.0

Potential good/bad ideas

jkotas commented Sep 25, 2022

stephentoub commented Sep 25, 2022

EgorBo commented Sep 25, 2022

jkotas commented Sep 25, 2022

EgorBo commented Sep 25, 2022

GSPP commented Sep 28, 2022

EgorBo commented Sep 28, 2022

jkotas commented Sep 28, 2022

EgorBo commented Sep 28, 2022

EgorBo commented Oct 4, 2022

EgorBo commented Oct 6, 2022 • edited Loading

EgorBo commented Oct 6, 2022 • edited Loading

jkotas commented Oct 6, 2022 • edited Loading

EgorBo commented Oct 6, 2022 • edited Loading

EgorBo commented Oct 8, 2022

MichalPetryka commented Oct 8, 2022

EgorBo commented Oct 8, 2022

jkotas commented Oct 8, 2022 • edited Loading

EgorBo commented Oct 8, 2022 • edited Loading

jkotas commented Oct 8, 2022 • edited Loading

MichalPetryka commented Oct 8, 2022

jkotas commented Oct 8, 2022

EgorBo commented Oct 10, 2022 • edited Loading

jkotas commented Oct 10, 2022

EgorBo commented Oct 10, 2022

jkotas commented Oct 10, 2022

EgorBo commented Oct 10, 2022 • edited Loading

Maoni0 commented Oct 10, 2022

EgorBo commented Nov 2, 2022

stephentoub commented Nov 2, 2022 • edited Loading

EgorBo commented Nov 2, 2022

mjsabby commented Nov 5, 2022

EgorBo commented Nov 5, 2022 • edited Loading

EgorBo commented Jun 9, 2023

EgorBo commented Sep 25, 2022 •

edited

Loading

EgorBo commented Oct 6, 2022 •

edited

Loading

EgorBo commented Oct 6, 2022 •

edited

Loading

jkotas commented Oct 6, 2022 •

edited

Loading

EgorBo commented Oct 6, 2022 •

edited

Loading

jkotas commented Oct 8, 2022 •

edited

Loading

EgorBo commented Oct 8, 2022 •

edited

Loading

jkotas commented Oct 8, 2022 •

edited

Loading

EgorBo commented Oct 10, 2022 •

edited

Loading

EgorBo commented Oct 10, 2022 •

edited

Loading

stephentoub commented Nov 2, 2022 •

edited

Loading

EgorBo commented Nov 5, 2022 •

edited

Loading