Add PIT HLD document. #1014

clarklee-guizhao · 2022-06-07T16:02:33Z

Repo	PR title	State
sonic-buildimage	[sonic-pit] Add PIT(Platform Integration Test) feature, first commit with only framework and 2 test cases with 1 supportting platform.
sonic-buildimage	[sonic-pit]Add PIT(Platform Integration Test) feature, second part, add 6 test cases.

clarklee-guizhao · 2022-06-14T02:43:14Z

Test comments.

zhufangbo · 2022-06-14T02:46:02Z

test comments.

keboliu · 2022-06-20T09:28:19Z

doc/pit/Platform_Integration_Test_high_level_design.md

+
+Platform plugins are intended to provide abstraction for device function, in order to test full functionality, we extend some APIs for testing purpose. Most of them are not intended to be used in a running deployment , they are defined to have fine grain test only. Extended/added plugins are as follow:
+
+* device/$platform_name/plugins/sfputil.py


Is it possible to extend the platform APIs instead of extending the plugins? platform plugs are phased out, many platforms don't support them anymore.

Yes, in phase 2 we may support platform APIs. Platform plugins is supported to back our massive deployment and a lot of users are still using platform plugins, that's why our first step extends plugins rather than platform APIs.

would be good to clarify the plan in the document itself. Incremental deliverable is OK, but let's share the plan.

would be good to clarify the plan in the document itself. Incremental deliverable is OK, but let's share the plan.

Thanks, I'll add it to HLD.

keboliu · 2022-06-20T09:33:33Z

doc/pit/Platform_Integration_Test_high_level_design.md

+
+### 3.4.1 Test cases list generator
+
+Test case list generator is used to generate a list of test cases, it parses users' input options, product specific platform configuration, and generate test case list from all test case database. This is used in the daily regression test and delivery test scenario.


is it possible to reuse the current test framework in the sonic-mgmt repositoty? If the new cases can be added there it will also benefit those who use the sonic-mgmt test cases as SONiC daily regression.

Yes, it's possible to add to sonic-mgmt repository, with some extra effort. Anyone want to take this piece of job is more than welcome.
As PIT is proposed to be used by ODM vendor in development and manufacturing stage, that's an early stage of a device lifecycle, it is designed to be a stand-alone component without sonic-mgmt test framework(it's much difficult to get sonic-mgmt deployed in these scenario, as we focus on a per device basis).

I agree with Kebo. It's much easier to put the tests together.

I'm not following on why standalone test is difficult.

I agree with Kebo. It's much easier to put the tests together.

I'm not following on why standalone test is difficult.

The most important reason is ODM vendor cannot have access to SAI unless they pay extra money while we(users) have it, especially for new ASIC. In this situation, they cannot have the whole sonic running, this is a reality for most smaller ODM vendors.

jeff-yin · 2022-07-28T16:05:18Z

doc/pit/Platform_Integration_Test_high_level_design.md

+Figure 3-4
+
+Each test case has a configuration file, containing information such as test case name, description, type, and tag list.
+Each platform has a platform configuration file, containing platform specific (product specific) information such as CPU architecture, boot loader, CPU error monitor capability, peripheral related information (e.g. FAN number, PSU control options, etc.).


Is there an example of the platform configuration file?

I am concerned that this will be yet another configuration file that platform vendors will have to generate and maintain when bringing platforms in to SONiC. Can the platform configuration be derived from already-existing platform configs?
https://github.com/Azure/sonic-buildimage/blob/master/device/dell/x86_64-dell_s6100_c2538-r0/platform.json is an example of an existing platform config.

There is also the PDDF feature that was contributed by Broadcom that provides structured platform config data as well. Example:
https://github.com/Azure/sonic-buildimage/blob/master/device/accton/x86_64-accton_as7712_32x-r0/pddf/pddf-device.json

Hi Jeff,

Case configuration is for test case, not imply any implementation of the switch. an example of fan test case:
{
"name": "fan-test",
"description": "Check fan status",
"type": "auto",
"tags": ["manufacture", "delivery", "pa", "emc"]
}
As above, the fan test case is tagged with list, those tags are used to generate test case list for certain scenario. Like, "manufacture" means this case should be include in manufacturing test in manufacturing process.

PIT is totally different from PDDF/PDK stuff, PIT is a verify system.

clarklee-guizhao · 2022-10-27T15:38:49Z

PR1: sonic-net/sonic-buildimage#12514

clarklee-guizhao · 2022-10-27T15:39:15Z

PR2: sonic-net/sonic-buildimage#12530

zhangyanzhao · 2022-10-31T05:05:32Z

@clarklee-guizhao would you please add the code PRs by following example #806 ?

prgeor · 2022-11-04T18:33:47Z

@clarklee-guizhao is this HLD presented in the community?

clarklee-guizhao · 2022-11-14T08:17:01Z

@clarklee-guizhao is this HLD presented in the community?

Yes, this HLD was presented in June 2022, in 2 weekly meeings.

lihuay · 2022-11-21T18:38:02Z

doc/pit/Platform_Integration_Test_high_level_design.md

+![](images/PIT_subsystem_overview.png)
+Figure 2-1
+
+PIT runs in SONiC variant(which is called D4OS, aka. Datacenter Development Diagnostic and Delivery OS, SONiC based, tailored to focus on the developing process of a switch product, co-develop by ODM and users, added users' tools for delivery process, system default configuation, etc). It can be deployed in 3 stage of a switch device developing time axis. Development stage and manufacturing stage which take place in ODM side. Delivery stage which happen on the IDC users. The verified hardware drivers can be added and run in the user SONiC system without changes.


Please avoid creating new brand names. -- I'm referring to the D4OS name

Hi @lihuay Thanks for the advise. So, should I change back to branch master and redo this PR?

yes, you need redo the PR against the master branch since the current PR is not for any branches.

yes, you need redo the PR against the master branch since the current PR is not for any branches.

Please check this PR, changed the branch to master.
#1127

lihuay · 2022-11-21T18:39:41Z

doc/pit/Platform_Integration_Test_high_level_design.md

+
+In the blooming SONiC community, lots of users build their software on open hardware（especially white-box switches）. As the devices hardware are built by ODM vendor, not end users (users' SONiC development team), it's necessary to have a systematic method to verify whether the hardware, firmware (such as CPLD, FPGA, BMC, BIOS) and their drivers meet the users' requirement. At this point, PIT system (Platform Integration Test) is introduced for this purpose.
+
+PIT system could lighten up workload for both ODM and end-user because it not only automates hardware, firmware, driver verification efforts, but also it acheive this in a platform independent way, ias it provide a set of platform APIs to unify platform functions, so that hardware from different ODM have a unique view to the users. PIT is the first standard target to simplified SONiC porting on white-box switch in SONiC community. The simplified porting job may attract more and more user and ODM partners to join the SONiC community.


by "lighten up", do you mean "reduce"?

Yes, reduce, it's more suitable.

lihuay · 2022-11-21T18:40:28Z

doc/pit/Platform_Integration_Test_high_level_design.md

+
+In the blooming SONiC community, lots of users build their software on open hardware（especially white-box switches）. As the devices hardware are built by ODM vendor, not end users (users' SONiC development team), it's necessary to have a systematic method to verify whether the hardware, firmware (such as CPLD, FPGA, BMC, BIOS) and their drivers meet the users' requirement. At this point, PIT system (Platform Integration Test) is introduced for this purpose.
+
+PIT system could lighten up workload for both ODM and end-user because it not only automates hardware, firmware, driver verification efforts, but also it acheive this in a platform independent way, ias it provide a set of platform APIs to unify platform functions, so that hardware from different ODM have a unique view to the users. PIT is the first standard target to simplified SONiC porting on white-box switch in SONiC community. The simplified porting job may attract more and more user and ODM partners to join the SONiC community.


ias it provide ==> as it provides?

Hah, typo. I'll fix it. Thanks.

lihuay · 2022-11-21T18:41:53Z

doc/pit/Platform_Integration_Test_high_level_design.md

+
+# Scope
+
+This document describes a subsystem for Platform Integration Test, aka. PIT. PIT system concentrates on standardize and automate white-box switch hardware functionalities verification, along with their driver and firmware. As a result, it will be easier to port SONiC on a new white-box platform which is PIT-verified. Current scope covers all hardware components, such as CPU, memory, SSD, power system, fan, system sensors of various kind, and logical device firmware management, BMC subsystem etc.


how does PIT compares to PMON? what is the problem in PMON that we are trying to address with PIT?

@lihuay
That's a good question and I'm trying to summary it briefly as below:

Q1: PIT and PMON are different in several aspects.

PIT is a tool while PMON is a monitoring daemon
PIT is a tool for verification in the stage of development, manufacturing and acceptance test in delivery, it's not a running process(daemon) as PMON(regular container). PIT won't (and should not) be run the the stage of production.

PIT run before production deployment while PMON runs in production
PMON is created and works in the running stage for switch system without BMC, it is used for platform monitoring while PIT is not used as monitoring tool.

how does PIT compares to PMON? what is the problem in PMON that we are trying to address with PIT?
@lihuay
The second question:
What issue PIT is addressing?

PIT is trying to solve 1 core issue: users tells the ODM partners how they verify the hardware+driver+firmware. In this way, both users and ODMs workloads are reduced as many duplicated requirement discussion, development detail(hardware+driver functionality, testing requirement) can be expressed by test logic(in codes).

Signed-off-by: Li Hua <guizhao.lh@alibaba-inc.com>

Add PIT HLD document.

4c51926

keboliu reviewed Jun 20, 2022

View reviewed changes

jeff-yin reviewed Jul 28, 2022

View reviewed changes

clarklee-guizhao mentioned this pull request Sep 29, 2022

Sonic pit sonic-net/sonic-buildimage#12224

Closed

7 tasks

Merge branch 'master' into pit

bf1b049

This was referenced Oct 31, 2022

[sonic-pit] Add PIT(Platform Integration Test) feature, second part, … sonic-net/sonic-buildimage#12530

Merged

[sonic-pit] Add PIT(Platform Integration Test) feature, first commit … sonic-net/sonic-buildimage#12514

Open

clarklee-guizhao requested review from keboliu and jeff-yin and removed request for keboliu and jeff-yin October 31, 2022 07:00

prgeor self-assigned this Nov 4, 2022

zhangyanzhao requested a review from prgeor November 7, 2022 23:11

lihuay reviewed Nov 21, 2022

View reviewed changes

Fix a misleading word, 'lighten up' to 'reduce'

227869f

Signed-off-by: Li Hua <guizhao.lh@alibaba-inc.com>

clarklee-guizhao closed this by deleting the head repository Nov 23, 2022

clarklee-guizhao mentioned this pull request Nov 23, 2022

Add PIT HLD document , this PR replace previous PR#1014. #1127

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PIT HLD document. #1014

Add PIT HLD document. #1014

clarklee-guizhao commented Jun 7, 2022 •

edited

Loading

clarklee-guizhao commented Jun 14, 2022

zhufangbo commented Jun 14, 2022

keboliu Jun 20, 2022

clarklee-guizhao Jun 21, 2022

lihuay Nov 23, 2022

clarklee-guizhao Nov 23, 2022

keboliu Jun 20, 2022

clarklee-guizhao Jun 21, 2022

lihuay Nov 22, 2022

clarklee-guizhao Nov 23, 2022

jeff-yin Jul 28, 2022

clarklee-guizhao Sep 29, 2022

clarklee-guizhao commented Oct 27, 2022

clarklee-guizhao commented Oct 27, 2022

zhangyanzhao commented Oct 31, 2022

prgeor commented Nov 4, 2022

clarklee-guizhao commented Nov 14, 2022

lihuay Nov 21, 2022 •

edited

Loading

clarklee-guizhao Nov 22, 2022

lihuay Nov 22, 2022 •

edited

Loading

clarklee-guizhao Nov 23, 2022

lihuay Nov 21, 2022

clarklee-guizhao Nov 22, 2022

lihuay Nov 21, 2022

clarklee-guizhao Nov 22, 2022

lihuay Nov 21, 2022

clarklee-guizhao Nov 22, 2022

clarklee-guizhao Nov 22, 2022


		Platform plugins are intended to provide abstraction for device function, in order to test full functionality, we extend some APIs for testing purpose. Most of them are not intended to be used in a running deployment , they are defined to have fine grain test only. Extended/added plugins are as follow:

		* device/$platform_name/plugins/sfputil.py


		### 3.4.1 Test cases list generator

		Test case list generator is used to generate a list of test cases, it parses users' input options, product specific platform configuration, and generate test case list from all test case database. This is used in the daily regression test and delivery test scenario.


		In the blooming SONiC community, lots of users build their software on open hardware（especially white-box switches）. As the devices hardware are built by ODM vendor, not end users (users' SONiC development team), it's necessary to have a systematic method to verify whether the hardware, firmware (such as CPLD, FPGA, BMC, BIOS) and their drivers meet the users' requirement. At this point, PIT system (Platform Integration Test) is introduced for this purpose.

		PIT system could lighten up workload for both ODM and end-user because it not only automates hardware, firmware, driver verification efforts, but also it acheive this in a platform independent way, ias it provide a set of platform APIs to unify platform functions, so that hardware from different ODM have a unique view to the users. PIT is the first standard target to simplified SONiC porting on white-box switch in SONiC community. The simplified porting job may attract more and more user and ODM partners to join the SONiC community.


		# Scope

		This document describes a subsystem for Platform Integration Test, aka. PIT. PIT system concentrates on standardize and automate white-box switch hardware functionalities verification, along with their driver and firmware. As a result, it will be easier to port SONiC on a new white-box platform which is PIT-verified. Current scope covers all hardware components, such as CPU, memory, SSD, power system, fan, system sensors of various kind, and logical device firmware management, BMC subsystem etc.

Add PIT HLD document. #1014

Add PIT HLD document. #1014

Conversation

clarklee-guizhao commented Jun 7, 2022 • edited Loading

clarklee-guizhao commented Jun 14, 2022

zhufangbo commented Jun 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clarklee-guizhao commented Oct 27, 2022

clarklee-guizhao commented Oct 27, 2022

zhangyanzhao commented Oct 31, 2022

prgeor commented Nov 4, 2022

clarklee-guizhao commented Nov 14, 2022

lihuay Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihuay Nov 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clarklee-guizhao commented Jun 7, 2022 •

edited

Loading

lihuay Nov 21, 2022 •

edited

Loading

lihuay Nov 22, 2022 •

edited

Loading