Ionide v5: Large memory footprint for FSAutoComplete processes

[theimowski]

fsautocomplete.dll and fsautocomplete.backgroundservices.dll dotnet processes consume large amounts of memory.
I've grabbed a snapshot using dotMemory and pasted screenshot of the overview tab - not sure whether it's helpful as is, but I'd be happy to provide more info if necessary.
Wasn't able to do same for backgroundservices process as dotMemory hanged.
I don't think I saw such footprints for the same workspace using ionide v4.

Screenshots

dotnet fsautocomplete.dll (PID 21400):

Machine info

• OS: Windows 10
• .NET SDK version: 5.0.101
• Ionide version: 5.0.3

[open-collective-bot]

Hey @theimowski 👋,

Thank you for opening an issue. We will get back to you as
soon as we can. Also, check out our OpenCollective and consider
backing us.

https://opencollective.com/ionide

PS.: We offer backer support for all backers. Don't forget to add backer label when you start backing us 😄

[Krzysztof-Cieslak]

5.1 should include some performance fixes.

However, as far as I know, 5.0 itself didn't introduce any performance regressions (5.1 fixes stuff that was around for ages). The only thing that has changed is that background service is now enabled by default. If you haven't got it enabled in 4.X then the memory usage was obviously lower (as there was no background service)

[cartermp]

@theimowski if your code isn't sharable for profiling, you can also try to collect a trace with the jetbrains tools (or perfview if you're a weirdo like me and @Krzysztof-Cieslak) to give a more detailed breakdown of memory usage over the 30s or so sample of actions taken. Traces like that tend to yield a lot more useful information about what the cause of certain problems could be.

[Krzysztof-Cieslak]

+1 for PerfView.

[theimowski]

I've just upgraded to 5.1 - will observe how it behaves for me and if necessary share perfview results

[theimowski]

With 5.1 the process can eat up over 20 gigs of RAM and whole CPU:

Trying to grab a heap dump with PerfView but it times out after 100 seconds:

 98.7s: Making  GC Heap Progress...
 99.7s: Making  GC Heap Progress...
100.0s: Timed out after 100 seconds
100.2s: Shutting down ETW session
100.7s: Making  GC Heap Progress...
101.5s: Making  GC Heap Progress...
HeapDump Error: ETL file shows the start of a heap dump but not its completion.


Completed: Dumping GC Heap to C:\Users\theimowski\Downloads\dotnet.gcDump   (Elapsed Time: 142.368 sec)
Error: HeapDump failed with exit code 1

Unfortunately can't either take snapshot with dotMemory - it hangs, possibly due to my PC running out of resources

I'll keep trying

[Krzysztof-Cieslak]

Is it happening just after start? Or is that some kind of memory leak happening over time? How big is the solution?

It will be kinda hard to say anything useful without having context here ;-)

[theimowski]

It's happening over time, so seems rather like a memory leak, the solution is 8 F# + 4 C# projects.
When starting the workspace the memory footprint is "sane" - below 1 GB.
After working for an hour or two the memory usage would grow to couple GB.
And it happened 2 times for me already that after a few hours it would go above 20 GB.

[theimowski]

Here's a perfview gcdump when the process was at ~2GB:

• 250K Max Dump Objs: https://drive.google.com/file/d/1gvG7NyEwLVtnHZV6Qd1cFqzfH3LrpYf3/view?usp=sharing
• 1000K: https://drive.google.com/file/d/1KvPMRm2YkX3rU9GOMU3PjrxgVpaG-Lri/view?usp=sharing
• 10000K: https://drive.google.com/file/d/1pI-snMwr2NGt1EAhVM6EyRQRH-S3a_gV/view?usp=sharing

[theimowski]

Closing - v5.2 seems to work much better 👍

[vijoc]

I seem to have stumbled on a similar issue, if not the same. This is with the latest version of Ionide (5.3.2).

Opening my solution with a few F# projects (essentially the SAFE template with just an additional project), one editor open, and background service disabled, the memory usage stabilizes around a reasonable level. However, running dotnet paket install causes the FSAC process to keep churning away at something, using a lot of CPU and increasing amounts of memory.

The first screenshot is taken some moments after running the dotnet paket install, while the second one is several minutes later. Nothing was done inside VS Code during this time.

This would seem to suggest some issue possibly related to caching and/or some interop issue with how Paket handles things. I'm not too familiar with the inner workings of Paket nor Ionide so I don't know what the issue might be.

I'm happy to provide further information if that would be helpful.

[cartermp]

@vijoc This would definitely imply that there's a project system issue here, since that is the component responsible for forwarding references / doing design-time builds. High CPU usage would be expected for a dotnet paket install run since there's a lot of CPU work to be done, but that's quite a lot of memory usage.

Since you're on Windows, could you do the following?

1. Create a new issue with some context about what you're doing (if you have a sharable repro, that's great!)
2. Use PerfView -> Collect and create an ETL trace for when you run dotnet paket install and let it run, then continue to collect for a bit after the command is finished just so you can capture what happens afterwards
3. Take a process dump of the .NET Host process with so much RAMP usage
4. Upload both and give links in the issue

These are the best way to look at performance, since they can tell folks exactly what's going on in your system.

[pkese]

I'm experiencing a similar situation with ionide.ionide-fsharp-5.3.2/bin/fsautocomplete.dll --background-service-enabled going from about 2 GB to 24GB of memory after running dotnet paket install + dotnet restore.

After that, things stabilize and stay around 24GB, even if I repeat the same steps again.

My solution contains about 25 projects and 250 source files.
But I'm on Linux and I'm not sure what memory profiling tools are available.

[baronfel]

you're in luck! https://github.com/dotnet/diagnostics works perfectly on linux. I'd suggest using dotnet-dump to take a memory dump and submit it here for investigation (or just look in it yourself!) and dotnet-trace to take CPU traces. I'll say that there are massive perf improvements coming to project management in the next release, so this should be less of an issue quite soon.

[lenscas]

Sadly, vscode + ionide is still using a lot memory for me (Well, relatively spoken as this laptop only has 8GB). It is pretty much impossible for me to use as a result, as even with just vscode open my laptop quickly runs out of memory.

If I start the project up then {ionide_folder}/bin/autocomplete.dll starts with using 1.2 GB. Then after I only did a few changes (added 2 new lines, saved the file, removed the 2 new lines again and saved) memory usage goes to 3.3 GB, though it already increased after the first save I just didn't look by how much.

Below you find screenshots of my task manager of the before and after (dotnet is the mentioned process), as well as the memory dumps of the before and after and links to the repo that I work the most on.

Please tell me if there is anything else I can provide to help diagnose this issue.

Before the edits

After the edits

memory dumps:
https://drive.google.com/file/d/1k4f4St76AmpZS7cLLq-Y9gd7cnJsWJRj/view?usp=sharing

repo:
https://github.com/lenscas/card_game/tree/master/godot_client

As my repo uses both generated code and Godot I made a copy of it with the generated files and Godot's .dll's already in place at
https://github.com/lenscas/ionide_memory_bug

[baronfel]

Briefly commented on this in Discord, but it's going to be hard to say anything conclusive from those dumps. 100% that kind of reproduction is what helps to triage this sort of thing, though!

Looking into the types returned, without knowing the exact call paths/GC roots, I see lots of the following types:

• TypeDefs
• MethInfos
• Maps of names to entity
• Maps of string to entity
• Tokens
• Name resolution items
• DisplayEnv

These don't surprise me at all. There are tons of areas of the FCS APIs where lists/maps of these items are computed or returned, and they can be quite large.

However, the FCS used in the current shipped FSAC is 39.x. 40.x is out, and we have ongoing work to upgrade to it. New versions of FCS often have memory/performance enhancements, so the thing to do that's actionable to solve this would be:

• wait for FSAC with FCS 40 to ship (there's an open-but-stalled MR currently)
• rerun this thread + collect dumps
• especially gcroot information would be useful, we might be 'anchoring' a GC root in FSAC that's keeping a whole ton of object alive

[boeremak]

This seems like the right place.

I have the same issue as mentioned above, I have a pretty small application: 3 files with some recursive functions and the dotnet process will easily use 800MB at startup.

If I type one line like: let b = 1 it will spike to 2,7GB in a couple of seconds.

I have 32GB of RAM but this seems a bit excessive.

I am running VS Code with .Net 6.1 SDK on Ubuntu.

Summary
The process dotnet (with pid 115373) is using approximately 2.8 GB of memory.
It is using 2.8 GB privately, and a further 63.4 MB that is, or could be, shared with other programs.
Dividing up the shared memory between all the processes sharing that memory we get a reduced shared memory usage of 15.8 MB. Adding that to the private usage, we get the above mentioned total memory footprint of 2.8 GB.

Any idea of things I might try?

[MarcoBuess]

Has this ever been adressed? I'm still seeing the same problem. Here is my Environment:

Operatingsystem: Windows 10 (10.0.19042)
VSCode: 1.63.2
Ionide: 5.10.2
Net-SDK: 6.0.101
FSharp/FSI: F# Interactive version 12.0.0.0 for F# 6.0

A relatively small .fsx script causes the .NET Host process to spike up to 3 GB of memory.

[cartermp]

@boeremak @MarcoBuess If you can take a memory dump and trace of a reproducible action that leads to high memory/CPU usage and upload it that would be great. Otherwise it's just guesswork as to what's the cause.

[MarcoBuess]

@cartermp Am I supposed to take the dump with dotnet-dump as described above? Where am I supposed to upload the dumps? As a further clarification, the problem literally happens, as soon as the file is opened in vscode. You can watch the .NET Host process grabbing more and more memory in Task Manager.

[cartermp]

A link to a fileshare could probably work (they're usually too large to accept for GitHub)

[MarcoBuess]

So even though I didn't take the dumps yet I found something rather interesting. I was using the following code snipped to load a json config using Microsoft.Extensions.Configuration

let loadConfig =
    ConfigurationBuilder()
        .SetBasePath(__SOURCE_DIRECTORY__)
        .AddJsonFile("config.json").Build()

Changing to code to this:

let loadConfig () =
    ConfigurationBuilder()
        .SetBasePath(__SOURCE_DIRECTORY__)
        .AddJsonFile("config.json").Build()

Made the memory usage of the .NET Host process go down from ~1.8 GB to ~420MB. This raises the question, if I'm doing something fundamentally wrong here?

[cartermp]

@MarcoBuess Interesting. Is this just with an idle script file, or does this also include executing it?

[MarcoBuess]

@cartermp This is with an idle script file. If you use paket load-scripts with the #load directive the issue occurs as well. Using

#load @".paket/load/net6.0/main.group.fsx"

The .NET Host processes memory spikes immediately to 1,5GB+ memory when opening the script. This doesn't happen if you use #r "paket: nuget..." to bring the dependencies into scope.

PS: Loading the script with the mentioned adjustments plus using #r "paket: nuget..." will still lead to .NET Host memory usage of ~1GB, which is still much imo, but not as bad.

[nadamgm]

I'm also experiencing this issue on my manjaro laptop with 8 gigs of ram. Process starts up with a reasonable memory footprint then eventually the process eats up the laptop memory and the pc crashes. My project is a single f# project with a couple of fsx scripts to fetch and organize data from the internet.

[boeremak]

I don't what has changed, but memory consumption is really down right now. Like orders of GBs.

I don't know who to thank for this, but know it is much appreciated.

Maybe a bit too fast, it is still at a couple of GBs. Seems lower though.

[lenscas]

Oh, that does indeed sounds great!

Thanks to everyone involved. Can't wait to test it out!

[MarcoBuess]

Can confirm. I updated Ionide to v7.0.0 this morning and it reduced the memory usage on my test project by ~2GB. From what I observed though, the issue was triggered as soon as it try's to load Intellisense.

PS: The issue not only occurs in .fsx files but also in full blown projects.

PSS: After working on the project for a couple of hours. Memory usage is up to ~3.5GB for .NET Host again. I'm working with the graph-sdk but the code is only about 180 lines.

[MarcoBuess]

I just came accross this OmniSharp issue. Starting the codebase with --disable-renderer-accessibility indeed solves the issue for me. Would you guys mind to confirm?

[drhumlen]

I keep getting ram usage of >5GB with Ionide on a single project. I frequently have to quit VsCode and reopen because my 16gb MBP m1 pro laptop runs out of memory as a result. And that's despite not running anything from within VsCode. (I run my F# project directly from Iterm2 (a terminal app))