From dc050628150e0dcc8546efe71c54e05fd9a5c187 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 22 Dec 2021 12:44:16 -0800 Subject: [PATCH 01/13] Create Interface-Link-bring-up-sequence.md --- .../Interface-Link-bring-up-sequence.md | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 doc/sfp-cmis/Interface-Link-bring-up-sequence.md diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md new file mode 100644 index 0000000000..02a683b896 --- /dev/null +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -0,0 +1,118 @@ +# Feature Name +Deterministic Approach for Interface Link bring-up sequence + +# High Level Design Document +#### Rev 0.2 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Abbreviation](#abbreviation) + * [References](#references) + * [Requirement](#requirement) + +# List of Tables + * [Table 1: Definitions](#table-1-definitions) + * [Table 2: References](#table-2-references) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:----------------------------------:|------------------------------| +| 0.1 | 08/16/2021 | Shyam Kumar | Initial version +| 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows | + + +# About this Manual +Its a high-level design document describing the need to have determinstic approach for +Interface link bring-up sequence and workflows for use-cases around it + +# Abbreviation + +# Table 1: Definitions +| **Term** | **Definition** | +| -------------- | ------------------------------------------------ | +| pmon | Platform Monitoring Service | +| xcvr | Transceiver | +| xcvrd | Transceiver Daemon | +| CMIS | Common Management Interface Specification | +| gbsyncd | Gearbox (External PHY) docker container | +| DPInit | Data-Path Initialization | +| QSFP-DD | QSFP-Double Density (i.e. 400G) optical module | + +# References + +# Table 2 References + +| **Document** | **Location** | +|---------------------------------------------------------|---------------| +| CMIS v5 | [CMIS5p0.pdf](http://www.qsfp-dd.com/wp-content/uploads/2021/05/CMIS5p0.pdf) | + +# Requirement + +## Problem Definition + + 1. Presently in SONiC, there is no synchronization between 400G Datapath Init operation and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios. + For 400G optics module, The Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state. + Fundamentally it means - have a deterministic approach to bring-up the interface. + + Also, this problem is mentioned ‘as outside-the-scope’ of ‘CMIS Application Initialization’ high-level design document + https://github.com/ds952811/SONiC/blob/0e4516d7bf707a36127438c7f2fa9cc2b504298e/doc/sfp-cmis/cmis-init.md#outside-the-scope + + 2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the optics laser for 100G/400G. + This will lead to power wastage and un-necessary fan power consumption to keep the module temperature in operating range + + +## Background/Context + + Per the ‘QSFPDD spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors', + need to follow following bring-up seq to enable port/interface with 400G optics in LC/chassis: + + a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd + + b) Enable port on PHY (bring-up port, serdes on the PHY ; enable signals) : gbsyncd + - Wait for signal to stabilize on PHY + + c) Enable optical module (turn laser on/ enable tx) : xcvrd or platform bootstrap/infra + + In boards not having PHY, 2nd point not needed but 1st & 3rd sequence to be followed. + + ### Clause from QSFP-DD (CMIS4 spec) + + ### Cluase from QSFP-DD (CMIS5 spec) + +# Objective + +The 400G optics data path initialization to be invoked only ‘after the NPU is initialized and Tx is enabled’ for the port. +For the External PHY based board, also need to ensure that ‘PHY is initialized, and Tx is enabled on the PHY’. + +# Proposal + +- xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB +- Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) +- xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API +- Recommendation is to follow this proposal for all the known interface speeds - 400G/100G/40G/25G/10G + - 400G - as mentioned above the CMIS spec to be followed + - 100G/40G/25G/10G - deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network + - This synchronization will also benefit native 10G SFPs interfaces as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) + +# Proposed Work-Flows +Please refer to the flow/sequence diagrams which covers the following required use-cases + - LC boot-up + - optics initialization + - admin enable, disable configurations + +# Optics Initialization + (at platform bootstrap layer) + + [LC boot-up sequence - optics INIT (platform bootstrap).pdf](https://github.com/Azure/SONiC/files/7765096/LC.boot-up.sequence.-.optics.INIT.platform.bootstrap.pdf) + +# Applying 'interface admin startup' configuration + +[LC boot-up sequence - 'admin enable' Config gets applied.pdf](https://github.com/Azure/SONiC/files/7765098/LC.boot-up.sequence.-.admin.enable.Config.gets.applied.pdf) + +# Applying 'interface admin shutdown' configuration + +[LC boot-up sequence - 'admin disable' Config gets applied.pdf](https://github.com/Azure/SONiC/files/7765100/LC.boot-up.sequence.-.admin.disable.Config.gets.applied.pdf) + + From 323b161349a0c0d6d2319d66026f38536c266990 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 22 Dec 2021 18:23:48 -0800 Subject: [PATCH 02/13] Update Interface-Link-bring-up-sequence.md --- .../Interface-Link-bring-up-sequence.md | 74 ++++++++++++------- 1 file changed, 46 insertions(+), 28 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 02a683b896..f6f968168c 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -10,7 +10,11 @@ Deterministic Approach for Interface Link bring-up sequence * [About This Manual](#about-this-manual) * [Abbreviation](#abbreviation) * [References](#references) - * [Requirement](#requirement) + * [Problem Definition](#problem-definition) + * [Background](#background) + * [Objective](#objective) + * [Proposal](#proposal) + * [Proposed Work-Flows](#proposed-work-flows) # List of Tables * [Table 1: Definitions](#table-1-definitions) @@ -48,46 +52,57 @@ Interface link bring-up sequence and workflows for use-cases around it |---------------------------------------------------------|---------------| | CMIS v5 | [CMIS5p0.pdf](http://www.qsfp-dd.com/wp-content/uploads/2021/05/CMIS5p0.pdf) | -# Requirement -## Problem Definition +# Problem Definition - 1. Presently in SONiC, there is no synchronization between 400G Datapath Init operation and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios. - For 400G optics module, The Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state. - Fundamentally it means - have a deterministic approach to bring-up the interface. +1. Presently in SONiC, there is no synchronization between 400G Datapath Init operation and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios. + + For 400G optics module, the Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state. + + Fundamentally it means - have a deterministic approach to bring-up the interface. + + Also, this problem is mentioned ‘as outside-the-scope’ of ‘CMIS Application Initialization’ high-level design document + **(https://github.com/ds952811/SONiC/blob/0e4516d7bf707a36127438c7f2fa9cc2b504298e/doc/sfp-cmis/cmis-init.md#outside-the-scope)** - Also, this problem is mentioned ‘as outside-the-scope’ of ‘CMIS Application Initialization’ high-level design document - https://github.com/ds952811/SONiC/blob/0e4516d7bf707a36127438c7f2fa9cc2b504298e/doc/sfp-cmis/cmis-init.md#outside-the-scope - - 2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the optics laser for 100G/400G. +2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the optics laser for 100G/400G. This will lead to power wastage and un-necessary fan power consumption to keep the module temperature in operating range - -## Background/Context +# Background Per the ‘QSFPDD spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors', need to follow following bring-up seq to enable port/interface with 400G optics in LC/chassis: - - a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd + a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd b) Enable port on PHY (bring-up port, serdes on the PHY ; enable signals) : gbsyncd - Wait for signal to stabilize on PHY - c) Enable optical module (turn laser on/ enable tx) : xcvrd or platform bootstrap/infra - In boards not having PHY, 2nd point not needed but 1st & 3rd sequence to be followed. + In boards not having PHY, #b) not needed but #a) and #c) sequence to be followed. + + ## Clause from QSFP-DD (CMIS4.0 spec) - ### Clause from QSFP-DD (CMIS4 spec) - - ### Cluase from QSFP-DD (CMIS5 spec) + Excerpt from CMIS4.0 spec providing detailed reasoning for the above-mentioned bring-up sequence + + ![61f5b485-cf3b-4ca8-beac-9102b6feabfe](https://user-images.githubusercontent.com/69485234/147173702-f124fc9d-ef27-4816-b1a1-b4a44a5833a7.PNG) + + + ## Clause from QSFP-DD (CMIS5.0 spec) + + Excerpt from CMIS5.0 spec providing detailed reasoning for the above-mentioned bring-up sequence + + ![96a35dc5-618f-418c-9593-5639a90f1b28](https://user-images.githubusercontent.com/69485234/147173164-5ad0123c-479a-4774-b3ee-12a81fdd7d7e.PNG) + # Objective -The 400G optics data path initialization to be invoked only ‘after the NPU is initialized and Tx is enabled’ for the port. -For the External PHY based board, also need to ensure that ‘PHY is initialized, and Tx is enabled on the PHY’. +Have a determistic approach for Interface link bring-up sequence i.e. below sequence to be followed: + 1. Initialize and enable NPU Tx and Rx path + 2. For system with 'External' PHY: Initialize and enable PHY Tx and Rx on both line and host sides; ensure host side link is up + 3. Then only perform optics data path initialization/activation/Tx enable (for 400G) and Tx enable (for 100G) # Proposal +Recommend following this high-level work-flow sequence to accomplish the Objective: - xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB - Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) - xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API @@ -98,21 +113,24 @@ For the External PHY based board, also need to ensure that ‘PHY is initialized # Proposed Work-Flows Please refer to the flow/sequence diagrams which covers the following required use-cases - - LC boot-up - - optics initialization - - admin enable, disable configurations + - Transceiver initialization + - admin enable configurations + - admin disable configurations -# Optics Initialization +# Transceiver Initialization (at platform bootstrap layer) - [LC boot-up sequence - optics INIT (platform bootstrap).pdf](https://github.com/Azure/SONiC/files/7765096/LC.boot-up.sequence.-.optics.INIT.platform.bootstrap.pdf) + ![LC boot-up sequence - optics INIT (platform bootstrap)](https://user-images.githubusercontent.com/69485234/147166795-5665670d-dd2b-4b6f-976c-eabcc65d5448.png) # Applying 'interface admin startup' configuration -[LC boot-up sequence - 'admin enable' Config gets applied.pdf](https://github.com/Azure/SONiC/files/7765098/LC.boot-up.sequence.-.admin.enable.Config.gets.applied.pdf) +![LC boot-up sequence - 'admin enable' Config gets applied](https://user-images.githubusercontent.com/69485234/147166867-56f3e82d-1b1c-4b7a-a867-5470ee6050e7.png) + # Applying 'interface admin shutdown' configuration -[LC boot-up sequence - 'admin disable' Config gets applied.pdf](https://github.com/Azure/SONiC/files/7765100/LC.boot-up.sequence.-.admin.disable.Config.gets.applied.pdf) +![LC boot-up sequence - 'admin disable' Config gets applied](https://user-images.githubusercontent.com/69485234/147166884-92c9af48-2d64-4e67-8933-f80531d821b4.png) + + From 736b9a8d72300d64b275233e1c2a0bbd32912bf9 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 19 Jan 2022 11:15:48 -0800 Subject: [PATCH 03/13] Update Interface-Link-bring-up-sequence.md --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index f6f968168c..85892d903b 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -106,9 +106,13 @@ Recommend following this high-level work-flow sequence to accomplish the Objecti - xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB - Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) - xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API -- Recommendation is to follow this proposal for all the known interface speeds - 400G/100G/40G/25G/10G +- Recommendation is to follow this proposal for all the known interfaces - 400G/100G/40G/25G/10G. Reason being: - 400G - as mentioned above the CMIS spec to be followed - - 100G/40G/25G/10G - deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network + - 100G/40G/25G/10G - + - deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network + - there is a possibility of interface link flaps with non-quiescent optical modules if this 'deterministic approach' is not followed + - It helps bring down the optical module laser when interface is adminstiratively shutdown. Per the workflow here, this is acheived by xcvrd listening to host_tx_ready field from PORT_TABLE of STATE_DB. Turning the laser off would reduce the power consumption and avoid any lab hazard + - Additionally provides uniform workflow across all interface types instead of just 400G - This synchronization will also benefit native 10G SFPs interfaces as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) # Proposed Work-Flows From 28a67ee0c5730857a56b3d824a30eadf794a5319 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 19 Jan 2022 13:26:47 -0800 Subject: [PATCH 04/13] Update Interface-Link-bring-up-sequence.md --- .../Interface-Link-bring-up-sequence.md | 26 ++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 85892d903b..15ad204602 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -24,11 +24,12 @@ Deterministic Approach for Interface Link bring-up sequence | Rev | Date | Author | Change Description | |:---:|:-----------:|:----------------------------------:|------------------------------| | 0.1 | 08/16/2021 | Shyam Kumar | Initial version -| 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows | +| 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows +| 0.3 | 01/19/2022 | Shyam Kumar | Addressed review-comments | # About this Manual -Its a high-level design document describing the need to have determinstic approach for +This is a high-level design document describing the need to have determinstic approach for Interface link bring-up sequence and workflows for use-cases around it # Abbreviation @@ -110,9 +111,10 @@ Recommend following this high-level work-flow sequence to accomplish the Objecti - 400G - as mentioned above the CMIS spec to be followed - 100G/40G/25G/10G - - deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network + e.g. If there is a PHY device in between, and this 'deterministic approach' is not followed, PHY may adapt to a bad signal or interface flaps may occur when the optics tx/rx enabled during PHY initialization. - there is a possibility of interface link flaps with non-quiescent optical modules if this 'deterministic approach' is not followed - It helps bring down the optical module laser when interface is adminstiratively shutdown. Per the workflow here, this is acheived by xcvrd listening to host_tx_ready field from PORT_TABLE of STATE_DB. Turning the laser off would reduce the power consumption and avoid any lab hazard - - Additionally provides uniform workflow across all interface types instead of just 400G + - Additionally provides uniform workflow (from SONiC NOS) across all interface types instead of just 400G - This synchronization will also benefit native 10G SFPs interfaces as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) # Proposed Work-Flows @@ -136,5 +138,23 @@ Please refer to the flow/sequence diagrams which covers the following required ![LC boot-up sequence - 'admin disable' Config gets applied](https://user-images.githubusercontent.com/69485234/147166884-92c9af48-2d64-4e67-8933-f80531d821b4.png) +# Out of Scope +Following items are not in the scope of this document. They would be taken up separately +1. xcvrd restart + - If the xcvrd goes for restart, then all the DB events will be replayed. + Here the Datapath init/activate (for 400G), tx-disable register set (for 100G), will be a no-op if the optics is already in that state + +2. syncd/gbsyncd/swss docker container restart + - Cleanup scenario - the host_tx_ready field in STATE-DB should be updated to “False” to respective ports that a PHY/NPU interface with + - +3. CMIS API feature is not part of this design and the APIs will be used in this design. For CMIS HLD, Please refer to: + https://github.com/Azure/SONiC/blob/9d480087243fd1158e785e3c2f4d35b73c6d1317/doc/sfp-cmis/cmis-init.md + +4. Error handling of SAI attributes + a) At present, If there is a set attribute failure, orch agent will exit. + Refer the error handling API : https://github.com/Azure/sonic-swss/blob/master/orchagent/orch.cpp#L885 + b) Error handling for SET_ADMIN_STATUS attribute will be added in future. + c) A propabale way to handle the failure is to set a error handling attribute to respective container syncd/GBsyncd with attribute that is failed. + The platform layer knows the error better and it will try to recover. From b802395e168013d42ad9ecefeff3a29bb27c02e6 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 19 Jan 2022 13:30:26 -0800 Subject: [PATCH 05/13] Update Interface-Link-bring-up-sequence.md Address review comments, added more details to 'Proposal' section, added 'Out of Scope' section --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 15ad204602..35891d2597 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -142,14 +142,11 @@ Please refer to the flow/sequence diagrams which covers the following required Following items are not in the scope of this document. They would be taken up separately 1. xcvrd restart - If the xcvrd goes for restart, then all the DB events will be replayed. - Here the Datapath init/activate (for 400G), tx-disable register set (for 100G), will be a no-op if the optics is already in that state - + Here the Datapath init/activate (for 400G), tx-disable register set (for 100G), will be a no-op if the optics is already in that state 2. syncd/gbsyncd/swss docker container restart - Cleanup scenario - the host_tx_ready field in STATE-DB should be updated to “False” to respective ports that a PHY/NPU interface with - - 3. CMIS API feature is not part of this design and the APIs will be used in this design. For CMIS HLD, Please refer to: https://github.com/Azure/SONiC/blob/9d480087243fd1158e785e3c2f4d35b73c6d1317/doc/sfp-cmis/cmis-init.md - 4. Error handling of SAI attributes a) At present, If there is a set attribute failure, orch agent will exit. Refer the error handling API : https://github.com/Azure/sonic-swss/blob/master/orchagent/orch.cpp#L885 From 4c735ca12ffd7c36ffaf5a9ba728c6941e31191e Mon Sep 17 00:00:00 2001 From: jaganbal-a <97986478+jaganbal-a@users.noreply.github.com> Date: Thu, 20 Jan 2022 20:56:52 -0500 Subject: [PATCH 06/13] Update Interface-Link-bring-up-sequence.md --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 35891d2597..ac3273ea05 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -99,7 +99,7 @@ Interface link bring-up sequence and workflows for use-cases around it Have a determistic approach for Interface link bring-up sequence i.e. below sequence to be followed: 1. Initialize and enable NPU Tx and Rx path 2. For system with 'External' PHY: Initialize and enable PHY Tx and Rx on both line and host sides; ensure host side link is up - 3. Then only perform optics data path initialization/activation/Tx enable (for 400G) and Tx enable (for 100G) + 3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules) # Proposal From d9d1ba6cc7e109196a0915735e3adbd14ef0cd23 Mon Sep 17 00:00:00 2001 From: jaganbal-a <97986478+jaganbal-a@users.noreply.github.com> Date: Sat, 22 Jan 2022 14:39:05 -0500 Subject: [PATCH 07/13] Update Interface-Link-bring-up-sequence.md (#2) * Update Interface-Link-bring-up-sequence.md * Update Interface-Link-bring-up-sequence.md * Update Interface-Link-bring-up-sequence.md * Update Interface-Link-bring-up-sequence.md * Update Interface-Link-bring-up-sequence.md --- .../Interface-Link-bring-up-sequence.md | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index ac3273ea05..982af7d21b 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -2,7 +2,7 @@ Deterministic Approach for Interface Link bring-up sequence # High Level Design Document -#### Rev 0.2 +#### Rev 0.3 # Table of Contents * [List of Tables](#list-of-tables) @@ -25,7 +25,7 @@ Deterministic Approach for Interface Link bring-up sequence |:---:|:-----------:|:----------------------------------:|------------------------------| | 0.1 | 08/16/2021 | Shyam Kumar | Initial version | 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows -| 0.3 | 01/19/2022 | Shyam Kumar | Addressed review-comments | +| 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments | # About this Manual @@ -56,22 +56,22 @@ Interface link bring-up sequence and workflows for use-cases around it # Problem Definition -1. Presently in SONiC, there is no synchronization between 400G Datapath Init operation and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios. +1. Presently in SONiC, there is no synchronization between Datapath Init operation of CMIS complaint optical module and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios. - For 400G optics module, the Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state. + For CMIS-compliant active (optical) modules, the Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state. Fundamentally it means - have a deterministic approach to bring-up the interface. Also, this problem is mentioned ‘as outside-the-scope’ of ‘CMIS Application Initialization’ high-level design document **(https://github.com/ds952811/SONiC/blob/0e4516d7bf707a36127438c7f2fa9cc2b504298e/doc/sfp-cmis/cmis-init.md#outside-the-scope)** -2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the optics laser for 100G/400G. +2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the opticcal module Tx/laser. This will lead to power wastage and un-necessary fan power consumption to keep the module temperature in operating range # Background - Per the ‘QSFPDD spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors', - need to follow following bring-up seq to enable port/interface with 400G optics in LC/chassis: + Per the ‘CMIS spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors', + need to follow following bring-up seq to enable port/interface with CMIS compliant optical modules in LC/chassis: a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd b) Enable port on PHY (bring-up port, serdes on the PHY ; enable signals) : gbsyncd @@ -80,14 +80,14 @@ Interface link bring-up sequence and workflows for use-cases around it In boards not having PHY, #b) not needed but #a) and #c) sequence to be followed. - ## Clause from QSFP-DD (CMIS4.0 spec) + ## Clause from CMIS4.0 spec Excerpt from CMIS4.0 spec providing detailed reasoning for the above-mentioned bring-up sequence ![61f5b485-cf3b-4ca8-beac-9102b6feabfe](https://user-images.githubusercontent.com/69485234/147173702-f124fc9d-ef27-4816-b1a1-b4a44a5833a7.PNG) - ## Clause from QSFP-DD (CMIS5.0 spec) + ## Clause from CMIS5.0 spec Excerpt from CMIS5.0 spec providing detailed reasoning for the above-mentioned bring-up sequence @@ -96,10 +96,10 @@ Interface link bring-up sequence and workflows for use-cases around it # Objective -Have a determistic approach for Interface link bring-up sequence i.e. below sequence to be followed: +Have a determistic approach for Interface link bring-up sequence for all interfaces types i.e. below sequence to be followed: 1. Initialize and enable NPU Tx and Rx path 2. For system with 'External' PHY: Initialize and enable PHY Tx and Rx on both line and host sides; ensure host side link is up - 3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules) + 3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules) # Proposal @@ -107,15 +107,16 @@ Recommend following this high-level work-flow sequence to accomplish the Objecti - xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB - Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) - xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API -- Recommendation is to follow this proposal for all the known interfaces - 400G/100G/40G/25G/10G. Reason being: - - 400G - as mentioned above the CMIS spec to be followed - - 100G/40G/25G/10G - +- Recommendation is to follow this proposal for all the known interfaces types- 400G/100G/40G/25G/10G. Reason being: + - CMIS complaint optical modules:- + All CMIS complaint optical modules will follow this approach as recommended in the CMIS spec. + - SFF complaint optical modules:- - deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network e.g. If there is a PHY device in between, and this 'deterministic approach' is not followed, PHY may adapt to a bad signal or interface flaps may occur when the optics tx/rx enabled during PHY initialization. - there is a possibility of interface link flaps with non-quiescent optical modules if this 'deterministic approach' is not followed - It helps bring down the optical module laser when interface is adminstiratively shutdown. Per the workflow here, this is acheived by xcvrd listening to host_tx_ready field from PORT_TABLE of STATE_DB. Turning the laser off would reduce the power consumption and avoid any lab hazard - - Additionally provides uniform workflow (from SONiC NOS) across all interface types instead of just 400G - - This synchronization will also benefit native 10G SFPs interfaces as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) + - Additionally provides uniform workflow (from SONiC NOS) across all interface types with or without module presence. + - This synchronization will also benefit SFP+ optical modules as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) # Proposed Work-Flows Please refer to the flow/sequence diagrams which covers the following required use-cases @@ -142,7 +143,7 @@ Please refer to the flow/sequence diagrams which covers the following required Following items are not in the scope of this document. They would be taken up separately 1. xcvrd restart - If the xcvrd goes for restart, then all the DB events will be replayed. - Here the Datapath init/activate (for 400G), tx-disable register set (for 100G), will be a no-op if the optics is already in that state + Here the Datapath init/activate for CMIS compliant optical modules, tx-disable register set (for SFF complaint optical modules), will be a no-op if the optics is already in that state 2. syncd/gbsyncd/swss docker container restart - Cleanup scenario - the host_tx_ready field in STATE-DB should be updated to “False” to respective ports that a PHY/NPU interface with 3. CMIS API feature is not part of this design and the APIs will be used in this design. For CMIS HLD, Please refer to: From 87ae10fe4c1292b7a8ff25cb68f8a2af317fa87c Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Tue, 25 Jan 2022 15:08:40 -0800 Subject: [PATCH 08/13] Update Interface-Link-bring-up-sequence.md Further review comments addressed --- .../Interface-Link-bring-up-sequence.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 982af7d21b..dde52c092f 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -21,11 +21,12 @@ Deterministic Approach for Interface Link bring-up sequence * [Table 2: References](#table-2-references) # Revision -| Rev | Date | Author | Change Description | -|:---:|:-----------:|:----------------------------------:|------------------------------| +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:----------------------------------:|-----------------------------------| | 0.1 | 08/16/2021 | Shyam Kumar | Initial version | 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows -| 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments | +| 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments +| 0.4 | 01/26/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments # About this Manual @@ -76,7 +77,7 @@ Interface link bring-up sequence and workflows for use-cases around it a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd b) Enable port on PHY (bring-up port, serdes on the PHY ; enable signals) : gbsyncd - Wait for signal to stabilize on PHY - c) Enable optical module (turn laser on/ enable tx) : xcvrd or platform bootstrap/infra + c) Enable optical module (data path initializatio, turn laser on/ enable tx) : xcvrd In boards not having PHY, #b) not needed but #a) and #c) sequence to be followed. @@ -101,13 +102,13 @@ Have a determistic approach for Interface link bring-up sequence for all interfa 2. For system with 'External' PHY: Initialize and enable PHY Tx and Rx on both line and host sides; ensure host side link is up 3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules) -# Proposal +# Plan -Recommend following this high-level work-flow sequence to accomplish the Objective: +Plan is to follow this high-level work-flow sequence to accomplish the Objective: - xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB -- Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) +- Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status from syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled) - xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API -- Recommendation is to follow this proposal for all the known interfaces types- 400G/100G/40G/25G/10G. Reason being: +- Per the discussion and agreement in sonic-chassis workgroup and OCP community, plan is to follow this proposal for all the known interfaces types- 400G/100G/40G/25G/10G. Reason being: - CMIS complaint optical modules:- All CMIS complaint optical modules will follow this approach as recommended in the CMIS spec. - SFF complaint optical modules:- @@ -145,7 +146,8 @@ Following items are not in the scope of this document. They would be taken up se - If the xcvrd goes for restart, then all the DB events will be replayed. Here the Datapath init/activate for CMIS compliant optical modules, tx-disable register set (for SFF complaint optical modules), will be a no-op if the optics is already in that state 2. syncd/gbsyncd/swss docker container restart - - Cleanup scenario - the host_tx_ready field in STATE-DB should be updated to “False” to respective ports that a PHY/NPU interface with + - Cleanup scenario - Check if the host_tx_ready field in STATE-DB need to be updated to “False” for any use-case, either in going down or coming up path + - Discuss further on the possible use-cases 3. CMIS API feature is not part of this design and the APIs will be used in this design. For CMIS HLD, Please refer to: https://github.com/Azure/SONiC/blob/9d480087243fd1158e785e3c2f4d35b73c6d1317/doc/sfp-cmis/cmis-init.md 4. Error handling of SAI attributes From c272efa5a54bb79aad8e47ece84b4240f24be656 Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Fri, 28 Jan 2022 14:11:42 -0800 Subject: [PATCH 09/13] Update Interface-Link-bring-up-sequence.md Addressed further review comments --- .../Interface-Link-bring-up-sequence.md | 20 +++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index dde52c092f..db63a672ea 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -13,7 +13,8 @@ Deterministic Approach for Interface Link bring-up sequence * [Problem Definition](#problem-definition) * [Background](#background) * [Objective](#objective) - * [Proposal](#proposal) + * [Plan](#plan) + * [Pre-requisite](#pre-requisite) * [Proposed Work-Flows](#proposed-work-flows) # List of Tables @@ -27,6 +28,7 @@ Deterministic Approach for Interface Link bring-up sequence | 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows | 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments | 0.4 | 01/26/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments +| 0.5 | 01/28/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments # About this Manual @@ -52,6 +54,7 @@ Interface link bring-up sequence and workflows for use-cases around it | **Document** | **Location** | |---------------------------------------------------------|---------------| +| CMIS v4 | [QSFP-DD-CMIS-rev4p0.pdf](http://www.qsfp-dd.com/wp-content/uploads/2019/05/QSFP-DD-CMIS-rev4p0.pdf) | | CMIS v5 | [CMIS5p0.pdf](http://www.qsfp-dd.com/wp-content/uploads/2021/05/CMIS5p0.pdf) | @@ -119,11 +122,20 @@ Plan is to follow this high-level work-flow sequence to accomplish the Objective - Additionally provides uniform workflow (from SONiC NOS) across all interface types with or without module presence. - This synchronization will also benefit SFP+ optical modules as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx) +# Pre-requisite + +As mentioned above in 'Background' and 'Plan' sections, need to follow specified bring-up sequence. +Work flows are designed considering SONiC NOS operating in sync mode. + +In case SONiC NOS operates in async mode, then expected behavior is - the return status of the set ADMIN_STATE attribute update in ASIC-DB (syncd/GBsyncd) will be treated to set the host_tx_ready in Orchagent. + # Proposed Work-Flows + Please refer to the flow/sequence diagrams which covers the following required use-cases - Transceiver initialization - admin enable configurations - admin disable configurations + - No transceiver present # Transceiver Initialization (at platform bootstrap layer) @@ -139,6 +151,11 @@ Please refer to the flow/sequence diagrams which covers the following required ![LC boot-up sequence - 'admin disable' Config gets applied](https://user-images.githubusercontent.com/69485234/147166884-92c9af48-2d64-4e67-8933-f80531d821b4.png) +# No transceiver present +if transceiver is not present: + - All the workflows mentioned above will reamin same ( or get exercised) till host_tx_ready field update + - xcvrd will not perform any action on receiving host_tx_ready field update + # Out of Scope Following items are not in the scope of this document. They would be taken up separately @@ -157,4 +174,3 @@ Following items are not in the scope of this document. They would be taken up se c) A propabale way to handle the failure is to set a error handling attribute to respective container syncd/GBsyncd with attribute that is failed. The platform layer knows the error better and it will try to recover. - From 6c85773b1d84945dab1f30c5c81059ea4bbf42be Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 2 Feb 2022 12:44:48 -0800 Subject: [PATCH 10/13] Update Interface-Link-bring-up-sequence.md Added additional work-flow (Ability to enable/disable this feature) --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index db63a672ea..a7674a7277 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -29,6 +29,7 @@ Deterministic Approach for Interface Link bring-up sequence | 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments | 0.4 | 01/26/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments | 0.5 | 01/28/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments +| 0.6 | 02/02/2022 | Shyam Kumar | Added feature-enablement workflow # About this Manual @@ -132,11 +133,25 @@ In case SONiC NOS operates in async mode, then expected behavior is - the return # Proposed Work-Flows Please refer to the flow/sequence diagrams which covers the following required use-cases + - Enabling this feature - Transceiver initialization - admin enable configurations - admin disable configurations - No transceiver present +# Feature enablement + This feature (optics Interface Link bring-up sequence) would be enabled on per platform basis. + There could be cases where vendor(s)/platform(s) may take time to shift from existing codebase to the model (work-flows) described in this document. + In order to avoid any breakage and ensure gradual migration to this model, there would be new config to enable/disable this feature. + High-Level workflow: + 1. Have a new field for this 'link_bringup_sequence' feature in config file + 2. By default it would be set to FALSE + 3. xcvrd to subscribe to state-change of this feature + 4. Platform while booting up, will set this feature to TRUE (as part of platform bootstrap layer) + 5. xcvrd on receving state-change notification of this feature, would act accordingly i.e. + - if the state is found as FALSE, ignore host_tx_ready field state + - if the state is found as TRUE, register to host_tx_ready field update notifications + # Transceiver Initialization (at platform bootstrap layer) From 4a39c0845ceee1fc1cde16fdbf29e5af95d13fee Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 2 Feb 2022 17:41:10 -0800 Subject: [PATCH 11/13] Update Interface-Link-bring-up-sequence.md Added Sequence diagram (Workflow) to enable/disable this feature --- .../Interface-Link-bring-up-sequence.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index a7674a7277..4ef3da945e 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -142,20 +142,17 @@ Please refer to the flow/sequence diagrams which covers the following required # Feature enablement This feature (optics Interface Link bring-up sequence) would be enabled on per platform basis. There could be cases where vendor(s)/platform(s) may take time to shift from existing codebase to the model (work-flows) described in this document. - In order to avoid any breakage and ensure gradual migration to this model, there would be new config to enable/disable this feature. - High-Level workflow: - 1. Have a new field for this 'link_bringup_sequence' feature in config file - 2. By default it would be set to FALSE - 3. xcvrd to subscribe to state-change of this feature - 4. Platform while booting up, will set this feature to TRUE (as part of platform bootstrap layer) - 5. xcvrd on receving state-change notification of this feature, would act accordingly i.e. - - if the state is found as FALSE, ignore host_tx_ready field state - - if the state is found as TRUE, register to host_tx_ready field update notifications - + In order to avoid any breakage and ensure gradual migration of different platforms/vendors to this model, there would be new field (flag) in xcvrd to enable/disable this feature. + When xcvrd spawns on LC/board, it would invoke platform plugin to check with the platform (hwsku) whether this feature is yet supported on underlying platform (board/LC) or not + + Workflow : + ![Enabling 'Interface link bring-up sequence' feature(3)](https://user-images.githubusercontent.com/69485234/152266723-050377ce-d4de-4c67-a405-5acc66474d46.png) + + # Transceiver Initialization (at platform bootstrap layer) - ![LC boot-up sequence - optics INIT (platform bootstrap)](https://user-images.githubusercontent.com/69485234/147166795-5665670d-dd2b-4b6f-976c-eabcc65d5448.png) +![LC boot-up sequence - optics INIT (platform bootstrap)](https://user-images.githubusercontent.com/69485234/152261613-e20dcda9-2adc-42aa-a1f1-4b8a47dd32af.png) # Applying 'interface admin startup' configuration From fdc4e1534697de7f2b0f5bba16670e099efefaee Mon Sep 17 00:00:00 2001 From: jaganbal-a <97986478+jaganbal-a@users.noreply.github.com> Date: Wed, 2 Feb 2022 23:43:42 -0500 Subject: [PATCH 12/13] Update Interface-Link-bring-up-sequence.md (#3) Added breakout handling section. --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 4ef3da945e..3f26a27d69 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -2,7 +2,7 @@ Deterministic Approach for Interface Link bring-up sequence # High Level Design Document -#### Rev 0.3 +#### Rev 0.7 # Table of Contents * [List of Tables](#list-of-tables) @@ -30,6 +30,7 @@ Deterministic Approach for Interface Link bring-up sequence | 0.4 | 01/26/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments | 0.5 | 01/28/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments | 0.6 | 02/02/2022 | Shyam Kumar | Added feature-enablement workflow +| 0.7 | 02/02/2022 | Jaganathan Anbalagan | Added Breakout Handling # About this Manual @@ -130,6 +131,11 @@ Work flows are designed considering SONiC NOS operating in sync mode. In case SONiC NOS operates in async mode, then expected behavior is - the return status of the set ADMIN_STATE attribute update in ASIC-DB (syncd/GBsyncd) will be treated to set the host_tx_ready in Orchagent. +# Breakout Handling + - The new 'host_tx_ready' field of Port table in state-DB is created for every interface . + - Xcvrd processes the 'host_tx_ready' change event and is responsible to disable Tx/laser for all optical lanes or respective optical lane that belongs to the interface in case of breakout. + - Currently the logical mapping between the interface and optical lane is not present in xcvrd. Creating this logical mapping in xcvrd will address breakout interface handling. + # Proposed Work-Flows Please refer to the flow/sequence diagrams which covers the following required use-cases From 1bdd505e6a5d634d0746cf0454043a76745bdf5d Mon Sep 17 00:00:00 2001 From: Shyam <69485234+shyam77git@users.noreply.github.com> Date: Wed, 2 Feb 2022 20:50:07 -0800 Subject: [PATCH 13/13] Update Interface-Link-bring-up-sequence.md Updated 'Table of Contents' with 'Breakout handling ' item --- doc/sfp-cmis/Interface-Link-bring-up-sequence.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md index 3f26a27d69..aef8958462 100644 --- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md +++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md @@ -15,6 +15,7 @@ Deterministic Approach for Interface Link bring-up sequence * [Objective](#objective) * [Plan](#plan) * [Pre-requisite](#pre-requisite) + * [Breakout handling](#breakout-handling) * [Proposed Work-Flows](#proposed-work-flows) # List of Tables