Multi-ASIC implementation #3888

SuvarnaMeenakshi · 2019-12-12T20:11:06Z

- What I did
To support SONiC on multi-asic platforms, the approach is to create SONiC docker stack for each ASIC. Each SONiC docker stack is separated by network namespaces for each ASIC.
Each network namespace will comprise of key dockers :
database
swss
syncd
bgp
teamd
interfaces-config
lldp

- How I did it
Main changes done to support SONiC on multi-asic platform are:

systemd service template for multi-instance service is added to start N number of systemd services where N is the number of ASICs in the platform.
Changes in FRR config to support BGP between the ASIC namespaces. To support this, a new type is added in DEVICE_METADATA called "InternalFrontend" and "InternalBackend" to differentiate between a front-end ASIC and back-end ASIC.
Change in docker_img_ctl.j2 to support starting dockers with a specific ASIC instance number which will be passed to the <docker_name>.sh script by the systemd service.
Apart from multiple instances of database docker, a global database docker will exist in the host namespace.

- How to verify it
Build a multi-asic VS image.
Follow instruction (in README.vsd) to bring up multi-asic VS image by adding the number of ASICs in device/platform/asic.conf and also adding a specific topology.sh in HWSKU directory to create a virtual ASIC connectivity in the VS.
Once the VS image comes up, ensure the dockers come up and verify BGP sessions between ASICs.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

* create systemd target to start/stop all containers in a specific namespace at once Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add multi-instance option to redis-cli * create database template service * change docker_image_ctl to use new redis sockets * increase database service timeout duration to 120s Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add instance number option and unix socket option to docker_image_ctl for swss * change swss service to service template * add instance number option and unix socket option to swss.sh * force all redis-cli calls through redis-cli wrapper Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add syncd service template * add instance number argument for syncd script

* add rsyslog service template compatible with network namespaces * change rsyslog_config to use database instance unix socket Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit e871f11)

* add wrapper for 'sudo ip netns exec' command Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 85745b7)

* increase from 2GB to 8GB to prevent OOM crashes * necessary because multi-ASIC devices have increased memory requirements Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 845778e)

* add bgp service template * create /etc/sonic/frr* for multiple instances of bgp docker container [frr docker]: add multi-ASIC support to bgpd.conf * add `redistribute connected` option to advertise routes to local interfaces * add `all` option to next-hop-self * add route-map to hide internal routes * add route-map to allow duplicate frontend router id Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 4ea01b7)

* change teamd service file to template * force teamd to restart when swss does Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 0297279)

* set submodule to point at newest commit Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit ebab66d)

* create service template * enable template instances in sonic_debian_extension * use iproute2 in interfaces-config script if working with network namespaces * make swss multi-instance dependent on multi-instance interfaces config Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 08af0db)

* use 'asic' as namespace prefix instead of 'namespace' in accordance with PR sonic-net#3269 Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit c1444e3)

lguohan · 2020-01-29T19:26:26Z

@pavel-shirshov , can you check the bgp template?

lguohan · 2020-01-29T19:30:14Z

files/image_config/rsyslog/rsyslog-config@.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Update rsyslog configuration


i do not understand why when need multi-instance for rsyslog.

Removed multiple instances for rsyslog service. Changes required for rsyslog service support for multi-asic platform will be added in a different PR.

lguohan · 2020-01-29T19:31:27Z

files/scripts/swss.sh

@@ -83,7 +79,11 @@ start_peer_and_dependent_services() {
    check_warm_boot

    if [[ x"$WARM_BOOT" != x"true" ]]; then
-        /bin/systemctl start ${PEER}
+        if [[ ! -z $DEV ]]; then
+                       /bin/systemctl start ${PEER}@$DEV


need consistent spacing.

abdosi · 2020-01-30T22:22:04Z

files/build_templates/multi_instance/bgp@.service.j2

@@ -0,0 +1,16 @@
+[Unit]
+Description=BGP container
+#Requires=updategraph.service


@SuvarnaMeenakshi @lguohan : we don't need this dependency for Multi Instance ? Single Instance still have this dependency.

support for updategraph and config-setup service for multi-asic platform will be added in a separate PR. As updategraph service is not updated to support multiple config_db, this dependency cannot be added here.

qiluo-msft · 2020-01-31T02:19:33Z

Please solve the conflicts #Closed

dockers/docker-database/base_image_files/redis-cli

different namespaces.

database instances

database instances.

to talk to different database instances

for multi-asic platform

platform

files/build_templates/per_namespace/syncd.service.j2

SuvarnaMeenakshi · 2020-03-19T18:30:39Z

retest vsimage please

paulnice

Looks good for me. Please wait for others.

lguohan · 2020-03-20T02:57:00Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-20T17:21:46Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-21T01:18:42Z

retest vsimage please

lguohan · 2020-03-24T07:10:22Z

pending on vsimage test passing.

SuvarnaMeenakshi · 2020-03-26T18:18:35Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-31T06:15:04Z

retest vsimage please

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

This reverts commit 2e87a16.

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

This reverts commit 2e87a16.

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

wendani · 2020-05-09T05:35:52Z

Multi-ASIC is not part of the 201911 release https://github.com/Azure/SONiC/wiki/Release-Progress-Tracking-201911

Why do we include it in 201911 @abdosi ?

Lawrence Lee and others added 14 commits December 12, 2019 10:48

[files]: add namespace synchronization target

504c632

* create systemd target to start/stop all containers in a specific namespace at once Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

[database docker]: add multi-instance support

8571864

* add multi-instance option to redis-cli * create database template service * change docker_image_ctl to use new redis sockets * increase database service timeout duration to 120s Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

[files]: add multi-instance support for syncd

193f182

* add syncd service template * add instance number argument for syncd script

[files]: add multi-instance support to logging

aa44fda

* add rsyslog service template compatible with network namespaces * change rsyslog_config to use database instance unix socket Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit e871f11)

[files]: add namespace wrapper

f4844e8

* add wrapper for 'sudo ip netns exec' command Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 85745b7)

[platform/vs]: add memory to virtual switch

a13a8d8

* increase from 2GB to 8GB to prevent OOM crashes * necessary because multi-ASIC devices have increased memory requirements Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 845778e)

[build_templates]: add multi-instance support to teamd

9328339

* change teamd service file to template * force teamd to restart when swss does Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 0297279)

[sonic-utilities]: update sonic-utilities

04b4887

* set submodule to point at newest commit Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit ebab66d)

[files]: change namespace name

d5c4a0d

* use 'asic' as namespace prefix instead of 'namespace' in accordance with PR sonic-net#3269 Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit c1444e3)

Remove namespace docker related files.

83ca4de

[build_templates]: remove namespace service dependency

6e3b519

lguohan requested a review from qiluo-msft January 29, 2020 19:05

lguohan reviewed Jan 29, 2020

View reviewed changes

abdosi reviewed Jan 30, 2020

View reviewed changes

qiluo-msft self-assigned this Jan 31, 2020

jleveque suggested changes Jan 31, 2020

View reviewed changes

dockers/docker-database/base_image_files/redis-cli Outdated Show resolved Hide resolved

SuvarnaMeenakshi added 8 commits February 11, 2020 18:04

Merge remote-tracking branch 'remotes/origin/master' into sumeenak_pr_2

87220d9

[redis-cli/sonic-cfggen]: Add wrappers to talk to database instance in

81b024e

different namespaces.

[swss.sh]: Modified to use redis-cli wrapper to talk to multiple

1875e6b

database instances

[syncd.sh]: Updated to use redis-cli wrapper to talk to multiple

3363c0d

database instances.

[docker_img_ctl.j2]: Updated to use redis-cli/sonic-cfggen wrappers

95fb5f5

to talk to different database instances

[database service]: changes made to include global database service

382f13e

for multi-asic platform

[swss/syncd services]: Added dependency to topology.service for VS

c20bf43

platform

[interfaces-config.sh]: Made changes to support multi-asic platform

a76eb01

jleveque suggested changes Mar 18, 2020

View reviewed changes

files/build_templates/per_namespace/syncd.service.j2 Outdated Show resolved Hide resolved

[syncd.servic]: Minor update in template file to make it concise.

fa25e1a

paulnice approved these changes Mar 19, 2020

View reviewed changes

jleveque approved these changes Mar 19, 2020

View reviewed changes

qiluo-msft approved these changes Mar 19, 2020

View reviewed changes

lguohan approved these changes Mar 20, 2020

View reviewed changes

yxieca approved these changes Mar 21, 2020

View reviewed changes

SuvarnaMeenakshi merged commit 4b8067e into sonic-net:master Mar 31, 2020

abdosi added the Request for 201911 Branch label Apr 2, 2020

abdosi pushed a commit that referenced this pull request Apr 2, 2020

Multi-ASIC implementation (#3888)

2e87a16

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

abdosi added the Included in 201911 Branch label Apr 2, 2020

nazariig mentioned this pull request Apr 3, 2020

[mellanox] SONiC fails to start on t0 #4367

Closed

abdosi added a commit that referenced this pull request Apr 3, 2020

Revert "Multi-ASIC implementation (#3888)"

249265a

This reverts commit 2e87a16.

SuvarnaMeenakshi mentioned this pull request Apr 5, 2020

[sonic-netns-exec]: use "$@" to reflects all positional parameters as they were set initially #4375

Merged

abdosi removed the Included in 201911 Branch label Apr 7, 2020

abdosi added the Included in 201911 Branch label Apr 15, 2020

abdosi pushed a commit that referenced this pull request Apr 15, 2020

Multi-ASIC implementation (#3888)

0099305

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Multi-ASIC implementation (sonic-net#3888)

aa616e8

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Revert "Multi-ASIC implementation (sonic-net#3888)"

485b9ff

This reverts commit 2e87a16.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Multi-ASIC implementation (sonic-net#3888)

eaa1d49

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

SuvarnaMeenakshi deleted the pr_2_multiasic branch December 14, 2020 09:28

arlakshm mentioned this pull request Aug 4, 2021

[redis-cli][multiasic]redis-cli supports namespace #8243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-ASIC implementation #3888

Multi-ASIC implementation #3888

SuvarnaMeenakshi commented Dec 12, 2019 •

edited

Loading

lguohan commented Jan 29, 2020

lguohan Jan 29, 2020

SuvarnaMeenakshi Feb 26, 2020

lguohan Jan 29, 2020

SuvarnaMeenakshi Feb 24, 2020

abdosi Jan 30, 2020

SuvarnaMeenakshi Feb 26, 2020

qiluo-msft commented Jan 31, 2020 •

edited

Loading

SuvarnaMeenakshi commented Mar 19, 2020

paulnice left a comment

lguohan commented Mar 20, 2020

SuvarnaMeenakshi commented Mar 20, 2020

SuvarnaMeenakshi commented Mar 21, 2020

lguohan commented Mar 24, 2020

SuvarnaMeenakshi commented Mar 26, 2020

SuvarnaMeenakshi commented Mar 31, 2020

wendani commented May 9, 2020

		@@ -0,0 +1,13 @@
		[Unit]
		Description=Update rsyslog configuration

Multi-ASIC implementation #3888

Multi-ASIC implementation #3888

Conversation

SuvarnaMeenakshi commented Dec 12, 2019 • edited Loading

lguohan commented Jan 29, 2020

lguohan Jan 29, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Feb 26, 2020

Choose a reason for hiding this comment

lguohan Jan 29, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Feb 24, 2020

Choose a reason for hiding this comment

abdosi Jan 30, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Feb 26, 2020

Choose a reason for hiding this comment

qiluo-msft commented Jan 31, 2020 • edited Loading

SuvarnaMeenakshi commented Mar 19, 2020

paulnice left a comment

Choose a reason for hiding this comment

lguohan commented Mar 20, 2020

SuvarnaMeenakshi commented Mar 20, 2020

SuvarnaMeenakshi commented Mar 21, 2020

lguohan commented Mar 24, 2020

SuvarnaMeenakshi commented Mar 26, 2020

SuvarnaMeenakshi commented Mar 31, 2020

wendani commented May 9, 2020

SuvarnaMeenakshi commented Dec 12, 2019 •

edited

Loading

qiluo-msft commented Jan 31, 2020 •

edited

Loading