[sgen] Fixes for max-heap-size option (mono/mono#18446)

* [sgen] Don't subtract twice the nursery size from max_heap_size

We were subtracting the nursery size when initializing max_heap_size as well as adding it to the allocated_heap when allocating the nursery (alloc_nursery).

* [sgen] Trigger collection if failing to allocate los section

If the memory governor says we are exceeding the max_heap_size by allocating a los section (1MB) for a new los object, attempt to clear some space by triggering a major collection, before completely bailing out.


Commit migrated from mono/mono@d9db79b

src/mono/mono/sgen/sgen-los.c

@@ -304,8 +304,15 @@ get_los_section_memory (size_t size)
 		return randomize_los_object_start (free_chunks, obj_size, size, LOS_CHUNK_SIZE);
 	}
 
-	if (!sgen_memgov_try_alloc_space (LOS_SECTION_SIZE, SPACE_LOS))
-		return NULL;
+	if (!sgen_memgov_try_alloc_space (LOS_SECTION_SIZE, SPACE_LOS)) {
+		/*
+		 * We failed to allocate a section due to exceeding max_heap_size.
+		 * Trigger a major collection and try again.
+		 */
+		sgen_ensure_free_space (LOS_SECTION_SIZE, GENERATION_OLD);
+		if (!sgen_memgov_try_alloc_space (LOS_SECTION_SIZE, SPACE_LOS))
+			return NULL;
+	}
 
 	section = (LOSSection *)sgen_alloc_os_memory_aligned (LOS_SECTION_SIZE, LOS_SECTION_SIZE, (SgenAllocFlags)(SGEN_ALLOC_HEAP | SGEN_ALLOC_ACTIVATE), NULL, MONO_MEM_ACCOUNT_SGEN_LOS);
 

src/mono/mono/sgen/sgen-memory-governor.c

@@ -519,7 +519,7 @@ sgen_memgov_init (size_t max_heap, size_t soft_limit, gboolean debug_allowance,
 		sgen_env_var_error (MONO_GC_PARAMS_NAME, "Setting to minimum.", "`max-heap-size` must be at least 4 times as large as `nursery size`.");
 		max_heap = SGEN_DEFAULT_NURSERY_SIZE * 4;
 	}
-	max_heap_size = max_heap - SGEN_DEFAULT_NURSERY_SIZE;
+	max_heap_size = max_heap;
 
 	sgen_gc_info.total_available_memory_bytes = max_heap;