[systemd] ASIC status based service bringup on VOQ chassis #7477
Conversation
|
@mprabhu-nokia, today on multi-ASIC device SNMP for example has a dependency on SWSS and requires all the SWSS instances to start before it can start. |
You are right, looks like all these services (may be some of these don't run on supervisor) have a dependency. @SuvarnaMeenakshi I see lldp has multiinstance swss as dependency and snmp does not, what would this mean? May be just that lldp itself is a per namespace service. |
Yes, that is right. lldp is per namespace service, so it depends only on that namespace specific swss service. |
Thanks. @SuvarnaMeenakshi and @arlakshm, do we know the hard dependencies for starting snmp, telemetry etc only if swss is started? Is it just with respect to speeding up boot times for critical services? |
| return | ||
|
|
||
| def enable_feature(asic_name, feature_names, feature_suffixes): | ||
| start_cmds = [] |
There was a problem hiding this comment.
Could this service be used to call any platform specific script to generate any required config like config_db.json? I see on multi-asic system, there is no service which creates config_db json file. Now sonic-cfggen requires database services to be running before it may be called to generate config_db json.
This service, bootstrap_asic, starts after database services and is responsible to start the swss and syncd. That makes it in a good spot to let platforms generate any required configs. Could there be a platform hook provided here for platforms to do any specific initialization?
|
Please divide the PR into two parts:
|
|
Are the UT logs enclosed to conversation older/prior to latest commit (3rd one)? |
|
Chassis Subgroup 5/26: |
mprabhu-nokia
changed the title
|
@judyjoseph - could you please review and approve. |
mprabhu-nokia
force-pushed
the
nokia-bootstrap-asic
branch
from
a4ea55d to
9a4ed4c
Compare
mprabhu-nokia
force-pushed
the
nokia-bootstrap-asic
branch
2 times, most recently
from
6fd26b3 to
e57788e
Compare
|
This pull request introduces 1 alert when merging bde49e42d932a6265538a1936f8211ce226aa0fb into f39fc88 - view on LGTM.com new alerts:
|
|
@judyjoseph - can you please merge asap please, needed for 202106 release |
Each asic dependent service will wait in the ExecStart part of systemd invocation for a REDIS notification that respective ASIC is now online.
Check if kvm-tests pass without changes
check_asic_status() modified Additional review comments
module key update from module_name to name
Skip additional checks for fixed-platforms
Simplify the logic in the lldp, syncd, swss, teamd scripts. asic_status checks can be done only for supervisor cards
mprabhu-nokia
force-pushed
the
nokia-bootstrap-asic
branch
from
5208cb2 to
05ca42e
Compare
| #!/bin/bash | ||
|
|
||
| is_chassis_supervisor() { | ||
| if [ -f /etc/sonic/chassisdb.conf ]; then |
There was a problem hiding this comment.
Need to find a different way to identify it is supervisor or linecard as this file /etc/sonic/chassisdb.conf is present in both sup and linecard
There was a problem hiding this comment.
@judyjoseph @abdosi
Quick clarification - The check is correct. In database-chassis.service, that is what is used to start the chassis-database only on the Supervisor. Since the file is not present on the line-card, chassis-database does not start there.
[Unit]
Description=database-chassis container
Requires=docker.service
ConditionPathExists=/etc/sonic/chassisdb.conf
After=docker.service
After=config-chassisdb.service
Requires=config-chassisdb.service
StartLimitIntervalSec=1200
StartLimitBurst=3
There was a problem hiding this comment.
Ok, yes - there is a similar reference here https://github.com/Azure/sonic-buildimage/blob/c508a10c08636c78a3fb4d7454cc93769b8fbf92/files/image_config/config-chassisdb/config-chassisdb#L40.
So even though the file /usr/share/sonic/device/$platform/chassisdb.conf, is present in both supervisor and linecard, the variable "start_chassis_db=1" is set in supervisor which is used to copy the file to /etc/sonic/chassisdb.conf
@mprabhu-nokia, @vganesan-nokia , @shyam77git, @anamehra -- Could you confirm this approach is followed in different chassis models ?
There was a problem hiding this comment.
@shyam77git, @anamehra can you confirm and sign-off these changes.
There was a problem hiding this comment.
At the outset, this seems OK i.e. having and checking chassisdb.conf presence on/for Supervisor.
Anand is validating this further on modular chassis to see how it goes
There was a problem hiding this comment.
Instead of /etc/sonic/chassisdb.conf, may we have be some other mechanism like an entry in DEVICE_METADATA to distinguish between Supervisor and other cards? Non VOQ chassis may not need a chassis db.
There was a problem hiding this comment.
As suggested in PR #8065 (in review comments) we can have a variable like "supervisor=1" in platform_env.conf.
There was a problem hiding this comment.
Sure, once that gets committed, we can take it up as part of a bug fix PR. Don't want to block this PR with additional dependencies now. Let me know if that is ok to proceed.
|
@shyam77git @anamehra Could one of you please review this? |
|
@ngoc-do - can you please review from Arista team as well, thanks |
| logger.log_error('Detected no asics on this platform for service {}'.format(service)) | ||
| sys.exit(1) | ||
|
|
||
| # Connect to STATE_DB and subscribe to chassis-module table notifications |
There was a problem hiding this comment.
Connect to CHASSIS_STATE_DB?
Changes to allow starting per asic services like swss and syncd only if the platform vendor codedetects the asic is detected and notified. The systemd services ordering we want is database->database@->pmon->swss@->syncd@->teamd@->lldp@ There is also a requirement that management, telemetry, snmp dockers can start even if all asic services are not up. Why I did it For VOQ chassis, the fabric cards will have 1-N asics. Also, there could be multiple removable fabric cards. On the supervisor, swss and syncd containers need to be started only if the fabric-card is in Online state and respective asics are detected by the kernel. Using systemd, the dependent services can be in inactive state. How I did it Introduce a mechanism where all ASIC dependent service wait on its state to be published via PMON to REDIS. Once the subscription is received, the service proceeds to create respective dockers. For fixed platforms, systemd is unchanged i.e. the service bring up and docker creation happens in the start()/ExecStartPre routine of the .sh scripts. For VOQ chassis platform on supervisor, the service bringup skips docker creation in the start() routine, but does it in the wait()/ExecStart routine of the .sh scrips. Management dockers are decoupled from ASIC docker creation.
…#7477) Changes to allow starting per asic services like swss and syncd only if the platform vendor codedetects the asic is detected and notified. The systemd services ordering we want is database->database@->pmon->swss@->syncd@->teamd@->lldp@ There is also a requirement that management, telemetry, snmp dockers can start even if all asic services are not up. Why I did it For VOQ chassis, the fabric cards will have 1-N asics. Also, there could be multiple removable fabric cards. On the supervisor, swss and syncd containers need to be started only if the fabric-card is in Online state and respective asics are detected by the kernel. Using systemd, the dependent services can be in inactive state. How I did it Introduce a mechanism where all ASIC dependent service wait on its state to be published via PMON to REDIS. Once the subscription is received, the service proceeds to create respective dockers. For fixed platforms, systemd is unchanged i.e. the service bring up and docker creation happens in the start()/ExecStartPre routine of the .sh scripts. For VOQ chassis platform on supervisor, the service bringup skips docker creation in the start() routine, but does it in the wait()/ExecStart routine of the .sh scrips. Management dockers are decoupled from ASIC docker creation.
Changes to allow starting per asic services like swss and syncd only if the platform vendor codedetects the asic is detected and notified. The systemd services ordering we want is database->database@->pmon->swss@->syncd@->teamd@->lldp@
There is also a requirement that management, telemetry, snmp dockers can start even if all asic services are not up.
Why I did it
For VOQ chassis, the fabric cards will have 1-N asics. Also, there could be multiple removable fabric cards. On the supervisor, swss and syncd containers need to be started only if the fabric-card is in Online state and respective asics are detected by the kernel. Using systemd, the dependent services can be in inactive state.
How I did it
Introduce a mechanism where all ASIC dependent service wait on its state to be published via PMON to REDIS. Once the subscription is received, the service proceeds to create respective dockers.
For fixed platforms, systemd is unchanged i.e. the service bring up and docker creation happens in the start()/ExecStartPre routine of the .sh scripts.
For VOQ chassis platform on supervisor, the service bringup skips docker creation in the start() routine, but does it in the wait()/ExecStart routine of the .sh scrips.
Management dockers are decoupled from ASIC docker creation.
Other options that were tried/discussed are below.
Option1
Introduce a new bootstrap service. Using systemd service options like "Before" and "Type=notify". The service will be kept waiting using "EXTEND_TIMEOUT_USEC" option for the notify socket.
Relying on systemd After=, Before= hard dependency features to make sure that swss@->syncd@ services come up only after the asic has been detected as Online. However, services like snmp have After=swss which creates a hard dependency on the service to be started.
We have to remove After=swss/syncd tag from snmp, telemetry, mgmt-framework services.
Option2
Use mask/unmask to control asic services
For swss/syncd services, on every boot, systemd-generator will mask these services only on the Supervisor card. When asic is detected by pmon, bootstrap-asic service will unmask swss/syncd for that asic.
(With this option, we may not require a per-asic bootstrap_asic.service. We can just have a single one)
How to verify it
Before asic1 is detected and notified
Notify asic0 presence via REDIS
(will come from PMON as part of sonic-net/sonic-platform-daemons#175)
After asic0 is detected and notified
Which release branch to backport (provide reason below if selected)
Description for the changelog
A picture of a cute animal (not mandatory but encouraged)