[Celestica-E1031] Enable CPU watchdog

[lizhijianrd]

Why I did it

Enable CPU watchdog on Celestica-E1031.

Work item tracking

• Microsoft ADO (number only): 24536684

How I did it

Add a system service cpu_wdt to enable CPU watchdog and send keep-alive signal to watchdog periodically.

How to verify it

Build SONiC image and installed on physical device. Can see the cpu_wdt work as expected.

$ sudo systemctl status cpu_wdt.service
● cpu_wdt.service - CPU WDT
     Loaded: loaded (/lib/systemd/system/cpu_wdt.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-08-09 14:37:12 UTC; 1min 11s ago
   Main PID: 324656 (cpu_wdt)
      Tasks: 2 (limit: 2329)
     Memory: 2.1M
     CGroup: /system.slice/cpu_wdt.service
             ├─324656 /bin/bash /usr/local/bin/cpu_wdt start
             └─325540 sleep 60

Aug 09 14:37:12 e1031-1 systemd[1]: Started CPU WDT.
Aug 09 14:37:12 e1031-1 cpu_wdt[324657]: Enable CPU WDT..
Aug 09 14:37:13 e1031-1 cpu_wdt[324675]: CPU WDT has been enabled with 180 seconds timeout
Aug 09 14:37:13 e1031-1 cpu_wdt[324676]: Enable keep alive messaging every 60 seconds

When I stopped cpu_wtd service, it will disarm watchdog before exit:

$ sudo systemctl stop cpu_wdt.service
$ sudo systemctl status cpu_wdt.service
● cpu_wdt.service - CPU WDT
     Loaded: loaded (/lib/systemd/system/cpu_wdt.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2023-08-09 14:47:24 UTC; 18s ago
    Process: 324656 ExecStart=/usr/local/bin/cpu_wdt start (code=exited, status=0/SUCCESS)
   Main PID: 324656 (code=exited, status=0/SUCCESS)

Aug 09 14:37:12 e1031-1 cpu_wdt[324657]: Enable CPU WDT..
Aug 09 14:37:13 e1031-1 cpu_wdt[324675]: CPU WDT has been enabled with 180 seconds timeout
Aug 09 14:37:13 e1031-1 cpu_wdt[324676]: Enable keep alive messaging every 60 seconds
Aug 09 14:46:23 e1031-1 systemd[1]: Stopping CPU WDT...
Aug 09 14:46:23 e1031-1 cpu_wdt[324656]: Terminated
Aug 09 14:46:23 e1031-1 cpu_wdt[331991]: Caught SIGTERM - exiting...
Aug 09 14:47:24 e1031-1 cpu_wdt[332812]: Watchdog disarmed successfully
Aug 09 14:47:24 e1031-1 cpu_wdt[332826]: CPU WDT has been disabled!
Aug 09 14:47:24 e1031-1 systemd[1]: cpu_wdt.service: Succeeded.
Aug 09 14:47:24 e1031-1 systemd[1]: Stopped CPU WDT.

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211
• 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[prgeor]

@lizhijianrd how did you test this? Did the system reboot if watchdog is not ticked? Can you issue echo c > /proc/sysrq-trigger and see if system finally rebooted?

[prgeor]

@lizhijianrd after reboot of system due to watchdog timeout, is the "show reboot-cause" showing "watchdog" ?

[lizhijianrd]

Hi @prgeor I tried to set the watchdog time to 60 seconds and keepalive interval to 120 seconds, the watchdog was not ticked on time and triggered reboot successfully. After the reboot, I can see the reboot-cause is Watchdog (None).

Per your suggestion, I also tried echo c > /proc/sysrq-trigger several times. It triggers the system reboot immediately and the reboot-cause is Unknown. I'm not sure is this the expected behavior?

[Blueve]

We request to backport 202012 only because:

1. 201811 already have cpu_wdt service
2. later releases are no longer supported on this platform

[mssonicbld]

Cherry-pick PR to 202012: #16193