Skip to content

Commit

Permalink
grub: Set soft memory limits to 80% of hard limits.
Browse files Browse the repository at this point in the history 
By default, we now set the soft memory limits to 80% of the hard memory limits
for EVE cgroups. This adjustment sets the target values for memory reclamation
when it's triggered by the Kernel.
Updated the default values for dom0_mem, eve_mem, and ctrd_mem in the
documentation and configuration files to reflect this change.

Signed-off-by: Nikolay Martyanov <nikolay@zededa.com>
  • Loading branch information
@OhmSpectator @eriknordmark
OhmSpectator authored and eriknordmark committed Sep 30, 2024
1 parent 2f2f905 commit da2c37b
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 26 deletions.
6 changes: 3 additions & 3 deletions docs/BOOTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,15 +99,15 @@ in grub.cfg with graphical GRUB menu to get the device to boot again.
1. Cgroup related options. Please follow the [example](README.md#eve-cgroups) to write
these values in the correct GRUB variables:
1. `dom0_mem` option of `hv_dom0_mem_settings` variable - memory limit for eve cgroup
(default is `set_global hv_dom0_mem_settings "dom0_mem=800M,max:800M"`)
(default is `set_global hv_dom0_mem_settings "dom0_mem=640M,max:800M"`)
2. `dom0_max_vcpus` option of `hv_dom0_cpu_settings` variable - cpu limit for eve cgroup
(default is `set_global hv_dom0_cpu_settings "dom0_max_vcpus=1"`)
3. `eve_mem` - memory limit for cgroups with services of EVE
(default is `set_global hv_eve_mem_settings "eve_mem=650M,max:650M"`)
(default is `set_global hv_eve_mem_settings "eve_mem=520M,max:650M"`)
4. `eve_max_vcpus` option of `hv_eve_mem_settings` variable - cpu limit for cgroups with services of EVE
(default is `set_global hv_eve_cpu_settings "eve_max_vcpus=1""`)
5. `ctrd_mem` option of `hv_ctrd_mem_settings` variable - memory limit for cgroups with containerd-shims of EVE
(default is `set_global hv_ctrd_mem_settings "ctrd_mem=400M,max:400M"`)
(default is `set_global hv_ctrd_mem_settings "ctrd_mem=320M,max:400M"`)
6. `ctrd_max_vcpus` option of `hv_ctrd_cpu_settings` variable - cpu limit for cgroups with containerd-shims of EVE
(default is `set_global hv_ctrd_cpu_settings "ctrd_max_vcpus=1"`)
2. Installer options. These options are used during [the installation](DEPLOYMENT.md) process and may be set by adding
Expand Down
26 changes: 12 additions & 14 deletions docs/MEMORY-SETTINGS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,12 @@ We use explicit memory settings for several of the cgroups. Namely, we set
memory limits for the `eve`, `eve/containerd`, `pillar`, and all the sub-cgroups
of the `eve-user-apps` cgroup.

By default, we set the soft memory limits to 80% of the hard memory limits. It's
done to give the Kernel target value for the memory usage of the cgroup.
When memory rebalancing is needed, the kernel will try to keep the memory usage
of the cgroup at the soft limit. The rebalancing is triggered in different
situations, for example, when the system is under memory pressure.

For the EVE-related cgroups, we set the memory limits in the kernel command line
arguments. For the user applications, we set the memory in runtime using the
values of RAM.
Expand Down Expand Up @@ -142,12 +148,12 @@ If the hard limit is not specified, the soft limit is used as the hard limit.
The mapping of the `<cmd_component_name>` to the cgroup is not straightforward
and is misleading. The following table shows the mapping:

| Component | cgroup | default value |
|-------------|-------------------|---------------|
| `dom0` | `eve` | 800M |
| `ctrd` | `eve/containerd/` | 400M |
| `eve` | `eve/services/` | 650M |
| `eve` | `eve/services/*` | 650M |
| Component | cgroup | default soft limit | default hard limit |
|-------------|-------------------|--------------------|--------------------|
| `dom0` | `eve` | 640M | 800M |
| `ctrd` | `eve/containerd/` | 320M | 400M |
| `eve` | `eve/services/` | 520M | 650M |
| `eve` | `eve/services/*` | 520M | 650M |

That way, the `dom0_mem` setting sets the memory limits for the `eve` cgroup,
the `ctrd_mem` setting sets the memory limits for the `eve/containerd` cgroup,
Expand Down Expand Up @@ -308,14 +314,6 @@ components in EVE OS. For each setting, we provide guidelines on when it makes
sense to increase or decrease the values, as well as the potential impact of
setting them too high or too low.

### Note on soft and hard limits

Currently, the soft and hard limits are set to the same value. Theoretically,
the soft limit should be set to a value that is lower than the hard limit, and
it can be used to handle the memory pressure in the system more effectively.
However, in practice, we have not tested this configuration yet and do not
have precise recommendations on how to set the soft limit.

### dom0_mem

The `dom0_mem` setting controls the memory allocated to the EVE OS's system
Expand Down
6 changes: 3 additions & 3 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -366,10 +366,10 @@ Memory and CPU limits of `eve`, `eve/services` and `eve/containerd` cgroups can
Example:

```text
set_global hv_dom0_mem_settings "dom0_mem=800M,max:800M"
set_global hv_dom0_mem_settings "dom0_mem=640M,max:800M"
set_global hv_dom0_cpu_settings "dom0_max_vcpus=1 dom0_vcpus_pin"
set_global hv_eve_mem_settings "eve_mem=650M,max:650M"
set_global hv_eve_mem_settings "eve_mem=520M,max:650M"
set_global hv_eve_cpu_settings "eve_max_vcpus=1"
set_global hv_ctrd_mem_settings "ctrd_mem=400M,max:400M"
set_global hv_ctrd_mem_settings "ctrd_mem=320M,max:400M"
set_global hv_ctrd_cpu_settings "ctrd_max_vcpus=1"
```
12 changes: 6 additions & 6 deletions pkg/grub/rootfs.cfg
View file
Original file line number Diff line number Diff line change
Expand Up @@ -154,13 +154,13 @@ function set_generic {
set_global hv_eve_cpu_settings "eve_max_vcpus=1"
#temporarily increase memory settings for kubevirt
if [ "$eve_flavor" = "kubevirt" ]; then
set_global hv_dom0_mem_settings "dom0_mem=8000M,max:8000M"
set_global hv_eve_mem_settings "eve_mem=6500M,max:6500M"
set_global hv_ctrd_mem_settings "ctrd_mem=4000M,max:4000M"
set_global hv_dom0_mem_settings "dom0_mem=6400M,max:8000M"
set_global hv_eve_mem_settings "eve_mem=5200M,max:6500M"
set_global hv_ctrd_mem_settings "ctrd_mem=3200M,max:4000M"
else
set_global hv_dom0_mem_settings "dom0_mem=800M,max:800M"
set_global hv_eve_mem_settings "eve_mem=650M,max:650M"
set_global hv_ctrd_mem_settings "ctrd_mem=400M,max:400M"
set_global hv_dom0_mem_settings "dom0_mem=640M,max:800M"
set_global hv_eve_mem_settings "eve_mem=520M,max:650M"
set_global hv_ctrd_mem_settings "ctrd_mem=320M,max:400M"
fi
set_global hv_ctrd_cpu_settings "ctrd_max_vcpus=1"
set_global hv_platform_tweaks "smt=false"
Expand Down

0 comments on commit da2c37b

Please sign in to comment.