node takes up too much committed memory

[beck-8]

version and command

.\subspace-node-windows-x86_64-skylake-gemini-3h-2024-may-15.exe `
  run `
  --chain gemini-3h `
  --base-path D:\subspace-node `
  --farmer `
  --name "beck" `
  --in-peers 320 `
  --out-peers 80 2>&1 | tee -a node.log

It often causes the farmer to out of mem when he re-Ps

2024-06-01T02:14:34.024015Z  INFO {farm_index=4}: subspace_farmer::reward_signing: Successfully signed reward hash 0xc176dd7a853215a6b54c4ab398f89f4de631413127f7fa9e85ef2f6a5d296fb5
2024-06-01T04:57:11.202509Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.137.1/tcp/30533/p2p/12D3KooWSFez94wC7LNzPfrPvTKoKgjJAdPijoAJeAtNjveYzrqv
2024-06-01T05:17:49.301069Z  WARN hickory_proto::xfer::dns_exchange: failed to associate send_message response to the sender
2024-06-01T05:17:49.301121Z  WARN hickory_proto::xfer::dns_exchange: failed to associate send_message response to the sender
2024-06-01T05:17:49.301131Z  WARN hickory_proto::xfer::dns_exchange: failed to associate send_message response to the sender
2024-06-01T05:20:25.844538Z  INFO {farm_index=2}: subspace_farmer::reward_signing: Successfully signed reward hash 0x808c0d6ae4acf1cffadbb7b52587558deb4b9a802ae363638b57ba1b22566c2d
2024-06-01T06:11:19.564299Z  INFO {farm_index=5}: subspace_farmer::reward_signing: Successfully signed reward hash 0x2df9d009521545ef9278610144010c48486ce31fc07d008471c9102e058c5479
2024-06-01T06:49:13.377104Z  INFO {farm_index=3}: subspace_farmer::reward_signing: Successfully signed reward hash 0x939449a23bf0755e79fb1eb88e8578a8eaa3637968df014eb22c1c7b5400bebf
2024-06-01T07:45:47.409121Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/240e:391:eda:e911:fcd7:4ebc:6f25:4ec9/tcp/30533/p2p/12D3KooWSFez94wC7LNzPfrPvTKoKgjJAdPijoAJeAtNjveYzrqv
2024-06-01T07:45:47.428933Z  INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/240e:391:eda:e911:d435:e49c:53b8:2490/tcp/30533/p2p/12D3KooWSFez94wC7LNzPfrPvTKoKgjJAdPijoAJeAtNjveYzrqv
2024-06-01T09:00:41.887484Z  INFO {farm_index=5}: subspace_farmer::reward_signing: Successfully signed reward hash 0xd8449e4081806d38ee34e30e0e6dd89af102d722dfab80eb3f703550e97cd30a
2024-06-01T09:13:39.225825Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) sector_index=283
2024-06-01T09:13:39.225904Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) sector_index=485
2024-06-01T09:14:14.544154Z  WARN libp2p_kad::handler: New inbound substream to peer exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream. peer=PeerId("12D3KooWJFgeT1tco7WYD19fUUzPQeJ6YGfmNEZ9JWvvtaijQ3UX")
2024-06-01T09:14:14.544208Z  WARN libp2p_kad::handler: New inbound substream to peer exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream. peer=PeerId("12D3KooWJFgeT1tco7WYD19fUUzPQeJ6YGfmNEZ9JWvvtaijQ3UX")
2024-06-01T09:16:12.321827Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) sector_index=444
memory allocation of 2621440 bytes failed

Because the submitted memory occupied too much, I increased some virtual memory. From the monitoring chart, I can see that when the virtual memory becomes less, he will apply to increase the virtual memory (I set 2G-16G)

The increase in physical memory in the last picture is a change caused by me using the rammap tool to clean up some memory.

I will update the latest version later and continue to try.

[nazar-pc]

The logs are rom node, not farmer and the issue is most likely farmer indeed as found in https://forum.subspace.network/t/high-ram-usage-for-may-15th-release/3952?u=nazar-pc

Working on it, but I don't think we had a tracking issue for it, so this will be the one now.

[beck-8]

I'm not sure if the system's memory is empty when the farmer memory overflows. The monitoring of my win was just added last night, I will continue to observe it.

[beck-8]

Once again, I saw that node submitted a large amount of memory usage.

However, I understand it as the buffer/cache memory of Linux. In theory, it should be released when it needs to be used. I found that the Windows mechanism does not work like this. It is equivalent to direct OOM when win finds that there is no available memory (excluding spare memory)

[nazar-pc]

Windows is really bad with this stuff for sure, I fought it before, but wins are alernated with loses. There will be a workaround in the next release though, see #2816 and forum thread with experimental build.

[nazar-pc]

Let me close this since workaround was merged and will be included in the next release. As to long-term solution we'll have to figure something out, but we have a TODO about it in the code.

[beck-8]

version: gemini-3h-2024-jun-11

I just want to submit some information. I don't know if this is a problem.
I found that node still takes up a lot of commit memory as I mentioned before.There is not much physical memory used directly. I'm not sure if this is a Windows problem, but I'm sure this situation will cause Windows to overflow memory, because Windows only looks at whether there is enough virtual memory. My solution is to add a large number of page files (swap)

That is to say, subspace-node currently occupies 32G of memory (I don’t know if my expression is correct)

[nazar-pc]

Virtual memory is a completely different concept from physical memory. Many applications are using many times more virtual memory than physical (web browsers for example), this doesn't necessarily mean there is an issue.

[beck-8]

Yes, but the problem with Windows is that virtual memory defaults to the size of physical memory, unlike Linux, which can be infinitely large.

[nazar-pc]

Try to use a node with Snap sync (start with new database using --sync snap). This will reduce the size of the database and should reduce memory usage as well. I created an upstream issue to not use memory-mapped I/O for the database (paritytech/parity-db#239), though I don't have high hopes it will actually happen.