Inconsistent .NET GC process memory usage reporting

[GPSnoopy]

TL;DR

GC / dotnet-gcdump say process uses 400-500MB. Task Manager says 2.0-2.5GB. "Missing" >1GB memory is clearly filled with garbage managed strings. Whisky Tango Foxtrot ?

Context

We have a C# service that reads in the metadata of ~150K parquet files using ParquetSharp and creates an in-memory index of all this data. The reading is multithreaded and spread across 14/28 cores (physical/logical), with the data coming from a file share. Since there is a lot of data repetition, I have spent some time analyzing and improving the general memory usage (i.e. mostly by sharing immutable instances of arrays instead of maintaining separate copies when the data turns out to be identical).

This is an ASP.NET Core 3.1 application, running as x64 on Windows 10 via JetBrains Rider (as I'm developing and testing it).

All memory measurements and analysis are done after calling:

    GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
    GC.Collect();

The Issue

The reason why I'm raising this ticket is that I've never been able to match the memory usage reported by Task Manager (Details Tab) and the GC memory stats.

• GC.GetTotalMemory(forceFullCollection: true) returns about ~400MB.
• GC.GetGCMemoryInfo().HeapSizeBytes returns about ~400MB as well (usually slightly larger than the previous value).
• dotnet-gcdump + eeheap gives GC Heap Size as ~500MB (spread across 28 heaps - which seems to match the number of logical cores).
• Task Manager reports about 2.0-2.5GB for the dotnet process.
• process.PagedMemorySize64 reports around 2GB.
• dotnet-dump (not gcdump) creates a ~2.5GB file.

Investigation

At first I suspected a memory leak in the native components of ParquetSharp, but couldn't find anything there. Also switching to non-server GC reduces the process size to ~1.5GB (I reverted back to server GC afterwards).

In desperation I just opened in Rider the 2.5GB memory dump produced by dotnet-dump and viewed it as a text file. A good chunk of the file happened to be C# strings. In fact, they were clearly garbage strings from string.Split() operations; our custom Parquet metadata containing a lot of semi-colon-separated lists.

Doing the string splitting and parsing using ReadOnlySpan views to avoid creating loads of temporary string reduced the total process memory usage from 2.0-2.5GB to 1.0GB. The values returned by GC.GetTotalMemory() and `GC.GetGCMemoryInfo().HeapSizeBytes remain virtually unchanged.

I think I can conclude at that point that:

• The missing memory was clearly populated by managed objects.
• The GC APIs and tools I used did not report that memory.
• There is still 0.5GB of unaccounted memory.

Issue and Questions

This leaves with a lot of questions. The three top ones being:

• Am I fundamentally misunderstanding GC memory usage reporting and tools like dotnet-gcdump ?
• If so, how do I measure and inspect the "missing" memory using standard .NET APIs and tools?
• If not, do we have a memory leak in .NET Core GC?

TODO

I'll see if I can reproduce this behaviour in a small demo application. Worth testing on .NET 5.0 as well.

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

TL;DR

GC / dotnet-gcdump say process uses 400-500MB. Task Manager says 2.0-2.5GB. "Missing" >1GB memory is cleared filled with garbage managed strings. Whisky Tango Foxtrot ?

Context

We have a C# service that reads in the metadata of ~150K parquet files using ParquetSharp and creates an in-memory index of all this data. The reading is multithreaded and spread across 14/28 cores (physical/logical), with the data coming from a file share. Since there is a lot of data repetition, I have spent some time analyzing and improving the general memory usage (i.e. mostly by sharing immutable instances of arrays instead of maintaining separate copies when the data turns out to be identical).

This is an ASP.NET Core 3.1 application, running as x64 on Windows 10 via JetBrains Rider (as I'm developing and testing it).

All memory measurements and analysis are done after calling:

    GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
    GC.Collect();

The Issue

The reason why I'm raising this ticket is that I've never been able to match the memory usage reported by Task Manager (Details Tab) and the GC memory stats.

• GC.GetTotalMemory(forceFullCollection: true) returns about ~400MB.
• GC.GetGCMemoryInfo().HeapSizeBytes returns about ~400MB as well (usually slightly larger than the previous value).
• dotnet-gcdump + eeheap gives GC Heap Size as ~500MB (spread across 28 heaps - which seems to match the number of logical cores).
• Task Manager reports about 2.0-2.5GB for the dotnet process.
• process.PagedMemorySize64 reports around 2GB.
• dotnet-dump (not gcdump) creates a ~2.5GB file.

Investigation

At first I suspected a memory leak in the native components of ParquetSharp, but couldn't find anything there. Also switching to non-server GC reduces the process size to ~1.5GB (I reverted back to server GC afterwards).

In desperation I just opened in Rider the 2.5GB memory dump produced by dotnet-dump and viewed it as a text file. A good chunk of the file happened to be C# strings. In fact, they were clearly garbage strings from string.Split() operations; our custom Parquet metadata containing a lot of semi-colon-separated lists.

Doing the string splitting and parsing using ReadOnlySpan views to avoid creating loads of temporary string reduced the total process memory usage from 2.0-2.5GB to 1.0GB. The values returned by GC.GetTotalMemory() and `GC.GetGCMemoryInfo().HeapSizeBytes remain virtually unchanged.

I think I can conclude at that point that:

• The missing memory was clearly populated by managed objects.
• The GC APIs and tools I used did not report that memory.
• There is still 0.5GB of unaccounted memory.

Issue and Questions

This leaves with a lot of questions. The three top ones being:

• Am I fundamentally misunderstanding GC memory usage reporting and tools like dotnet-dcdump ?
• If so, how do I measure and inspect the "missing" memory using standard .NET APIs and tools?
• If not, do we have a memory leak in .NET Core GC?

TODO

I'll see if I can reproduce this behaviour in a small demo application. Worth testing on .NET 5.0 as well.

Author:	GPSnoopy
Assignees:	-
Labels:	area-GC-coreclr, tenet-performance, untriaged
Milestone:	-

[davidfowl]

cc @noahfalk @Maoni0

[noahfalk]

I can suggest a few potential issues to look out for + some other tools that might help track down the mystery memory if it isn't one of my guesses.

1. For the GC heap there are different useful measures of size:

• GC.GetGCMemoryInfo().HeapSizeBytes - This is a number of bytes being used for .NET objects on the GC heap
• GC.GetGCMemoryInfo().TotalCommittedBytes - This is the amount of VM that the GC has committed. Some of it is used to store .NET objects, some is used for the GC's private data structures, and some could be memory that the GC used for objects in the past and/or anticipates using for objects in the near future.
  Checking Committed instead of HeapSize might show you a chunk of memory that your current experiments are excluding.

2. Another issue you might be seeing depends on the point in time the size was calculated. As a .NET app runs the heap grows for a time storing new allocations and then at some point GC happens to reclaim the portion of memory that isn't referenced. This means just prior to GC the heap can be larger and then just after GC it is smaller. All of the numbers being reported by GetGCMemoryInfo() were calculated just after the GC ran when the heap was small. If there is substantial variation before/after GC then we'd probably expect TaskManager VM usage varying over time.

If neither of these account for it a few other tools that might shed more light:

• VMMap - great tool to get a top level breakdown of process VM usage.
• SOS !eeheap without the -gc argument to see other memory regions .NET may have allocated on your behalf

One more useful resource is the GC memdoc. Hopefully some of this will help things add up, but if the memory remains a mystery let us know what you found and we'll figure out where to look next.

[GPSnoopy]

I've attached a demo running on .NET5:
DotNetGcStressTest_v1.zip

Results on a R5950X 128GB:

Server GC

Generating garbage:
- Kept 100 objects alive

Memory statistics:
- GC GetTotalMemory: 73 MiB
- GC Info HeapSizeBytes: 73 MiB
- GC Info TotalCommittedBytes: 10,257 MiB
- GC Info TotalAvailableMemoryBytes: 130,987 MiB
- GC Info FragmentedBytes: 0 MiB
- GC Info MemoryLoadBytes: 20,957 MiB
- GC Info HighMemoryLoadThresholdBytes: 125,747 MiB
- Process PagedMemorySize: 11,218 MiB
- Process PagedSystemMemorySize: 0 MiB
- Process PrivateMemorySize: 11,218 MiB
- Process VirtualMemorySize: 2,151,659 MiB
- Process WorkingSet: 10,644 MiB
- Process PeakPagedMemorySize: 15,809 MiB
- Process PeakWorkingSet: 15,221 MiB
- Task Manager Working Set: 10,625 MiB

Completed in 27.6 seconds

Workstation GC

Generating garbage:
- Kept 100 objects alive

Memory statistics:
- GC GetTotalMemory: 73 MiB
- GC Info HeapSizeBytes: 73 MiB
- GC Info TotalCommittedBytes: 244 MiB
- GC Info TotalAvailableMemoryBytes: 130,987 MiB
- GC Info FragmentedBytes: 0 MiB
- GC Info MemoryLoadBytes: 17,028 MiB
- GC Info HighMemoryLoadThresholdBytes: 125,747 MiB
- Process PagedMemorySize: 281 MiB
- Process PagedSystemMemorySize: 0 MiB
- Process PrivateMemorySize: 281 MiB
- Process VirtualMemorySize: 2,102,750 MiB
- Process WorkingSet: 256 MiB
- Process PeakPagedMemorySize: 24,716 MiB
- Process PeakWorkingSet: 24,618 MiB
- Task Manager Working Set: 238 MiB

Completed in 135.3 seconds

TotalCommittedBytes seems to be the closest to what I was looking for, as it does account for the "missing" memory. This property was not available in .NET 3.1.

The Workstation GC seems to be fairly good at releasing the working set pages back to the OS, but the Server GC hogs 10GB even though only 73MB are actually used by the GC. Is this intentional? Is there a way to force the GC to release the memory that is unused?

[cshung]

The Workstation GC seems to be fairly good at releasing the working set pages back to the OS, but the Server GC hogs 10GB even though only 73MB are actually used by the GC. Is this intentional? Is there a way to force the GC to release the memory that is unused?

Check this out: #48601

[Maoni0]

this is most likely due to how we handle the ephemeral segments which is described here. committed bytes can be roughly approximated to the SizeBeforeBytes field in GCGenerationInfo that you can also get with GetGCMemoryInfo - GenerationInfo[0].

[GPSnoopy]

I think I'm happy to close this ticket. The API to report the committed memory has been added in .NET 5, so it's not like I'm left wondering whether we have a native memory leak in our code anymore.

Thanks for the link @Maoni0 , that's a very useful GC documentation.