Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move /var/log to RAM for Mellanox SN2700, Nokia 7215 and Dell S6100 #15077

Merged
merged 2 commits into from
Jun 26, 2023

Conversation

prgeor
Copy link
Contributor

@prgeor prgeor commented May 15, 2023

Why I did it

Move the /var/log on RAM. This is to prevent too many disk write on /var/log when mounted on disk.

Work item tracking

Microsoft ADO (number only): 17955517

How I did it

Pass kernel cmdline option "log_inram=on"

How to verify it

  1. Mellanox SN2700
root@str-msn2700-02:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           791M   15M  776M   2% /run
root-overlay     15G   12G  2.9G  80% /
/dev/sda3        15G   12G  2.9G  80% /host
tmpfs           790M   12M  779M   2% /var/log
tmpfs           3.9G  107M  3.8G   3% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/f50948841bee041368bf7c0546ceab4c71f05951fb0ed5ae70411f28dde68907/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/c45de6c53e7185631a37e87686dd296b2585425f638aa92c720c90eae038480c/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/f5bc87d5c2965b21b222f09dd57fe0fc798e518101d7ecd25d170b7662ae3e80/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/b2f435a256b930da4897d8a096095dcce183a6efa55b5b637187a654db0585ee/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/5c3588e42b29fd0516a164c00de621b7a00236ecbb240c4d0b3903ec706c220d/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/5a4a2a2602fb4ed1d1df90c3916076f595b4d8bc18eb465dd23e33f354adcfb8/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/9926f7378de9223fd3e88c8f59d888ad178e2ca23fa978f372e9838f10b7b803/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/130abaf95cffc06d952adacb6aa54a2f5e7c54c81fa8c15184389e25a7884328/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/aeef95cf5af6e20909a4cfd6c696176cc5dcb31dd456cc8acbbd3d59d47333d7/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/ef9bb94012b9fe987e55c9b73141296da8081d258d0d134922776c3c4b3ec551/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/cf425d372b347fd68569f128e1771e5a70dbf504b2f013304d60bcef6dfbd0da/merged
overlay          15G   12G  2.9G  80% /var/lib/docker/overlay2/7a2592cdac5c7369a6a98e07dbf1c2d96d29634e7d7b593617c50cc7e09e5cb3/merged
root@str-msn2700-02:~# 
root@str-msn2700-02:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       3.0Gi       3.3Gi       133Mi       1.5Gi       4.4Gi
Swap:             0B          0B          0B
root@str-msn2700-02:~# 


Dell S6100

root@str-s6100-acs-5:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           794M   15M  780M   2% /run
root-overlay     14G  9.9G  3.6G  74% /
/dev/sda4        14G  9.9G  3.6G  74% /host
tmpfs           793M   13M  781M   2% /var/log
tmpfs           3.9G   60K  3.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/f94441208fba5df49b0b8f0b49c699475ed0fd07673ab4a3eb574869b8e17c83/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/1c3dc3b582599602aec0dbd78945560f330f6244d2e218750622b3814dc53ed3/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/ab5b96e72e323fff5168abc69f8599fa244410d856dbd10cdbf73c99a4fe8d67/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/0e6e3adaba6bb1d2684da444661e540030d588ef498466b7d8ff773ce263a2ea/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/1218ed8bfa7a17c8927b20005d45f5e1e4a634e653d5c5c2057ac54713dc3387/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/b31486f665e5c929966185397990553fee6b41b515cbef28c945096673ac9bef/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/b984fa70f30bd1bac92bdf8d36542ed4433b4dabc33f7bb1f0a17a5eaee90f3e/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/b7866a1462768f3564b832187837c7a5e3d493b8084204e59610960cc5f6bc19/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/113bbbe88ee8452f4310b02a1343cfb4f1beb5fedf68a7d810ff5b5d7457c9f0/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/e7cc383186c6f9acecf2031c0c1f0870b8a7f63e1918b8359afa7a13d3c28963/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/c5d269100da205981c51e70e9e86facf69487f99e234dcdac822b8ab01af3d6a/merged
overlay          14G  9.9G  3.6G  74% /var/lib/docker/overlay2/463874ab78b2e45a34cf4d3d1cd2e45ff18c0abbf37be62d2c8559dce38d6219/merged
root@str-s6100-acs-5:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       2.1Gi       4.1Gi        69Mi       1.6Gi       5.3Gi
Swap:             0B          0B          0B
root@str-s6100-acs-5:~# 

Nokia-7215

root@str-2-7215-acs-4:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.5G     0  1.5G   0% /dev
tmpfs           303M   14M  289M   5% /run
root-overlay     15G  7.2G  7.2G  51% /
/dev/sda2        15G  7.2G  7.2G  51% /host
tmpfs           302M  7.5M  295M   3% /var/log
tmpfs           1.5G   60K  1.5G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/617e49b8b8e4368db2b3b2fb3e3204e80ec572fe7981d67ad2116d9c3e4472f3/merged
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/c94b855482fc14aa1f032b0c8dc035b02f37ad9e4341cb5a8d22f14e14c63824/merged
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/2d8c6ee95b212bbc8376d15916723128455678f2a3c88f382b451bec88297341/merged
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/92114013a19dc19f30505ba645f961d50e093365422a9b22116ced1fa88ded2b/merged
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/c8e79a8403863887666324f163a4b6633c40c8b349402b3a0f40ba7e51adb28b/merged
overlay          15G  7.2G  7.2G  51% /var/lib/docker/overlay2/27fd4a51859f3febd345a8551a0b4686d696c205048e1d595b76114385a68949/merged
root@str-2-7215-acs-4:~# 

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

@prgeor prgeor changed the title Move /var/log on tmpfs Move /var/log to RAM May 15, 2023
@@ -1 +1,2 @@
VAR_LOG_SIZE=4096
extra_cmdline_linux="logs_inram=on"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am worries on this platform, it doesn't have too much mem either. Do you have a standard to judge if we need loginram for a platform?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Blueve 7215 has around 3GB of total memory. So /var/log size on tmpfs will be max 10% of total memory which comes to around 300MB. Currently it has 2.2GB free:-

root@str-2-7215-acs-4:~# free -h
              total        used        free      shared  buff/cache   available
Mem:          3.0Gi       580Mi       1.9Gi        57Mi       500Mi       2.2Gi
Swap:            0B          0B          0B
root@str-2-7215-acs-4:~# 

Copy link
Contributor

@Blueve Blueve May 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But var log size was set to 4GB... you can refer to LN1: VAR_LOG_SIZE=4096, 300M log size is too small and might easy to trigger rotate issue

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Blueve checked with Guohan...seems fine. We need to live with this limitation of less memory but cannot use disk for too much writes

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Blueve , what rotation issue?

@prgeor prgeor force-pushed the var_log_inram branch 3 times, most recently from 2475ec1 to 03e79bd Compare May 31, 2023 23:37
@lguohan
Copy link
Collaborator

lguohan commented Jun 21, 2023

@prgeor , i cannot find where 300MB are set? can you point out?

@prgeor prgeor changed the title Move /var/log to RAM Move /var/log to RAM for Mellanox SN2700, Nokia 7215 and Dell S6100 Jun 21, 2023
@prgeor
Copy link
Contributor Author

prgeor commented Jun 21, 2023

@prgeor , i cannot find where 300MB are set? can you point out?

@lguohan the calculation is in this set_tmpfs_log_partition_size()

@prgeor
Copy link
Contributor Author

prgeor commented Jun 21, 2023

@lguohan @yxieca please see if this can be merged

@prgeor
Copy link
Contributor Author

prgeor commented Jul 12, 2023

@yxieca please cherry pick to 202205.

@prgeor
Copy link
Contributor Author

prgeor commented Jul 12, 2023

@StormLiangMS please cherry pick to 202305

mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this pull request Jul 12, 2023
…onic-net#15077)

* add ONIE_PLATFORM_EXTRA_CMDLINE_LINUX to kernel bootparam
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202205: #15804

@prgeor
Copy link
Contributor Author

prgeor commented Jul 27, 2023

@qiluo-msft could you cherry pick to 202012?

vaibhavhd added a commit to sonic-net/sonic-mgmt that referenced this pull request Jul 27, 2023
Fix the tests failing on newly addded platforms where logs are moved to tmpfs.

Related PR: sonic-net/sonic-buildimage#15077

Fixes: sonic-net/sonic-buildimage#15895
This change updates the check from hwsku match to platform match
@mssonicbld
Copy link
Collaborator

@prgeor PR conflicts with 202012 branch

@liuh-80
Copy link
Contributor

liuh-80 commented Aug 2, 2023

@prgeor, this PR has conflict with 202012, please create manual cherry-pick PR.

qiluo-msft pushed a commit that referenced this pull request Sep 15, 2023
…l S6100

### Why I did it
The commit by prgeor that was merged to master had conflict with 202012 branch and needed to be manually cherrypicked.


##### Work item tracking
- Microsoft ADO **(number only)**: 25086124

#### How I did it
Manually cherrypicked changes from PR #15077 

#### How to verify it
Built image with these changes and checked the filesystem:


**Mellanox-SN2700**
```
admin@str-msn2700-02:~$ show ver | grep -i "sonic software version"
SONiC Software Version: SONiC.202012-16544.362113-6980ffe77
admin@str-msn2700-02:~$
admin@str-msn2700-02:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           793M  8.4M  785M   2% /run
root-overlay     15G  6.4G  7.9G  45% /
/dev/sda3        15G  6.4G  7.9G  45% /host
tmpfs           792M  4.7M  788M   1% /var/log
tmpfs           3.9G  104M  3.8G   3% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup

admin@str-msn2700-02:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.7Gi       2.5Gi       3.9Gi       120Mi       1.4Gi       4.9Gi
Swap:            0B          0B          0B

```

**Nokia-M0-7215**

```
SONiC Software Version: SONiC.202012-16544.362113-6980ffe77
Platform: armhf-nokia_ixs7215_52x-r0
HwSKU: Nokia-M0-7215
ASIC: marvell

admin@str2-7215-acs-1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.5G     0  1.5G   0% /dev
tmpfs           304M  7.3M  296M   3% /run
root-overlay     15G  5.0G  9.5G  35% /
/dev/sda2        15G  5.0G  9.5G  35% /host
tmpfs           303M  728K  303M   1% /var/log
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup

admin@str2-7215-acs-1:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          3.0Gi       632Mi       1.6Gi       9.0Mi       772Mi       2.3Gi
Swap:            0B          0B          0B

```

**Dell S6100**

```
SONiC Software Version: SONiC.202012-16544.362113-00aac5392
Platform: x86_64-dell_s6100_c2538-r0
HwSKU: Force10-S6100
ASIC: broadcom
ASIC Count: 1
Uptime: 19:50:04 up 2 min,  1 user,  load average: 3.38, 1.66, 0.65

admin@str-s6100-acs-4:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           797M  7.8M  789M   1% /run
root-overlay     14G  4.7G  8.8G  35% /
/dev/sda4        14G  4.7G  8.8G  35% /host
tmpfs           796M  6.7M  790M   1% /var/log
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup

admin@str-s6100-acs-4:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       1.6Gi       4.9Gi        14Mi       1.3Gi       5.9Gi
Swap:            0B          0B          0B
admin@str-s6100-acs-4:~$

```

### Tested branch (Please provide the tested image version)

- [x] SONiC.202012-16544.362113-6980ffe77

#### Description for the changelog
Move /var/log to RAM for Mellanox SN2700, Nokia 7215 and Dell S6100 for the 202012 image.
sonic-otn pushed a commit to sonic-otn/sonic-buildimage that referenced this pull request Sep 20, 2023
…onic-net#15077)

* add ONIE_PLATFORM_EXTRA_CMDLINE_LINUX to kernel bootparam
assrinivasan added a commit to assrinivasan/sonic-buildimage that referenced this pull request Oct 4, 2023
lguohan pushed a commit that referenced this pull request Oct 11, 2023
… S6100 (#15077)" (#16775)

This reverts commit 05f326e.

Microsoft ADO 25355843:
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this pull request Oct 14, 2023
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this pull request Oct 17, 2023
mssonicbld pushed a commit that referenced this pull request Oct 17, 2023
… S6100 (#15077)" (#16775)

This reverts commit 05f326e.

Microsoft ADO 25355843:
AharonMalkin pushed a commit to AharonMalkin/sonic-mgmt that referenced this pull request Jan 25, 2024
…et#9094)

Fix the tests failing on newly addded platforms where logs are moved to tmpfs.

Related PR: sonic-net/sonic-buildimage#15077

Fixes: sonic-net/sonic-buildimage#15895
This change updates the check from hwsku match to platform match
stephenxs added a commit to stephenxs/sonic-buildimage that referenced this pull request Jun 4, 2024
…and Dell S6100 (sonic-net#15077)" (sonic-net#16775)"

This reverts commit 61683d9.

Signed-off-by: Stephen Sun <stephens@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants