[SDFAB-221] Make PTF tests more readable #268

dariusgrassi · 2021-06-02T21:34:48Z

Pull request building more readable PTF tests by creating a function generating test arguments in fabric_test.py which takes in high level parameters and returns dictionaries of various test configurations

- Including new function get_test_args - Reads in high level parameters from actual test and generates test test arguments - Makes tests more readable by simplifying repeated code

codecov · 2021-06-02T21:45:16Z

Codecov Report

Merging #268 (c903a30) into main (b6f8a8f) will not change coverage.
The diff coverage is n/a.

❗ Current head c903a30 differs from pull request most recent head 33ea668. Consider uploading reports for the commit 33ea668 to get more accurate results

@@            Coverage Diff            @@
##               main     #268   +/-   ##
=========================================
  Coverage     69.69%   69.69%           
  Complexity      467      467           
=========================================
  Files            37       37           
  Lines          3260     3260           
  Branches        333      333           
=========================================
  Hits           2272     2272           
  Misses          813      813           
  Partials        175      175

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6f8a8f...33ea668. Read the comment docs.

redesign - Created one giant for-loop for return dictionaries - Changed appending and returning a list into yielding dictionaries - Populate lists in order of for-loop complexity from top to bottom

ptf/tests/ptf/fabric_test.py

- Changed FabricSpgwDownlinkTest in test.py to match high-level param model - Fixed list naming model to match consistent style

working for FabricSpgwDownlinkTest - Changed self.get_test_args call to get_test_args (causing error in Jenkins)

specification - One high level variable called traffic_dir that takes in string input of model "source-device-destination" - Implementation then parses input into variables source, device, dest - vlan_conf_list and other such lists are now specified based off of this parameter rather than multiple variables like device_type and traffic_dir being separate

ptf/tests/ptf/fabric_test.py

ptf/tests/ptf/fabric.ptf/test.py

ptf/tests/ptf/fabric_test.py

Yi-Tseng · 2021-06-14T22:19:19Z

ptf/tests/ptf/fabric_test.py

+    """ SEND_REPORT_TO_SPINE """
+    # Check if INT is specified and a spine is specified in traffic_dir
+    # TODO: what about if source is a spine?
+    if int_test_type in INT_OPTIONS and is_next_hop_spine_list[0] != None:
+        send_report_to_spine_list = [False, True]
+    else:
+        send_report_to_spine_list = [None]


I believed that this will always be [False, True] if it is an INT test.

Consider a topology like a figure below, host1 sends packets to host2 (red arrow), here we have three traffic direction in this case: host1-leaf1-spine1, leaf1-spine1-leaf2, and spine1-leaf2-host2
Every device in this red path will generate an INT report and send it to the collector on the leaf1 (orange arrows)

For host1-leaf1-spine1, leaf1 sends the INT report to the collector directly

For leaf1-spine1-leaf2, spine1 sends the INT report to leaf1 and leaf1 will help forward it to the collector

For spine1-leaf2-host2, leaf2 sends the INT report to spine1, and spine1 will help to forward it to the collector via leaf1

Also, when sending packets from host2 to host1, the leaf2 will also send an INT report to the spine. (`host2-leaf2-spine1)

This tells us that send_report_to_spine_list will be [False, True] if traffic_dir is host-leaf-spine or spine-leaf-host

Another case is to have a remote cluster/POD with an INT collector.

For every type of traffic in the local culster (host-leaf-spine, leaf-spine-leaf, spine-leaf-host, and host-leaf-host), the device(leaf or spine) will send the INT report to another spine.

I didn't draw host-leaf-host traffic, but it is similar to host-leaf-spine case.

Firstly, thank you for this explanation, I appreciate the effort you went through to make your reasoning clear.

I understand the reasoning behind the first example. However, I'm confused by the second example. In the first one, you state that send_report_to_spine = [True, False] if traffic_dir is host-leaf-spine or spine-leaf-host. However, in this separate example, send_report_to_spine = [True, False] regardless of traffic_dir, as long as it is an INT test.

Which is it? Is send_report_to_spine = [True, False] just as long as it is an INT test, or is there a specific topology in which this is the case, and other topologies in which it is not?

Right, send_report_to_spine will always be [True, False] if it is an INT test since we need to consider all possible cases.
The traffic_dir won't really affect the value of send_report_to_spine

Co-authored-by: Yi Tseng <a86487817@gmail.com>

ptf/tests/ptf/fabric.ptf/test.py

…o readable-ptf

- Didn't call .items() when looping through vlan_confs dict

- Cleanup spacing and remove obvious comments - Remove loops for consistent singular test arguments - Check correct traffic direction devices with const option lists - Yield more test arguments

* Added allow list to get_test_args for-loop generation * Added new fields to print statement in for loop * Question about lines 150-156, packet attribute specifications

* Modify get_test_args to parse packet addresses and pass in loop * Modify SPGW test cases to include new address specifications

* Remove consideration of traffic direction from conditional

ptf/tests/common/fabric_test.py

arguments

* Modify get_test_args to accomodate for INT test tweaks * Clean unused paramaters from INT+SPGW test doRunTest function * Make pkt_addrs parameter in get_test_args not required since SPGW+INT tests don't use the convention for their packets

get_test_args

* Only access pkt_addrs dictionary for non-INT tests * Stop printing actual pkt while test is running by adding pkt after print * Change params in INT doRunTest to access tagged1 + tagged2 instead of just tagged

* Implement INT SPGW non-drop tests * Implement INT SPGW drop tests

ptf/tests/common/fabric_test.py

* Move OPTIONS constants from get_test_args to top of file with other constants * Add documentation for get_test_args function * Rename vague allow param to ue_recirculation * Fixed: include **kwargs in doRunTest for INT SPGW tests

Yi-Tseng

some comments, but lgtm overall

ptf/tests/common/fabric_test.py

* Implement conditional psc flag for SPGW tests * Add some more traffic directions for tests that weren't testing enough device types (i.e., include spine type for destination device for some tests)

…-tna into readable-ptf

* Some INT tests follow unreadable for-loop structure, convert these to readable format * Change default packet list to include VXLAN packet types

INT tests

Yi-Tseng · 2021-07-22T08:33:31Z

Here are some summaries after looking at all parameters carefully.

We ignored the SCTP type for SPGW tests since there won't be an SCTP packet in the GTP tunnel. So we missed some test cases in the PR, but I think that's correct.
In the main branch, we won't set send_report_to_spine to True when tagged[1] is True, which is not correct. This PR fixes this issue.

Here is the diff, compare with only common parameters:

The SCTP issue:

test.FabricSpgwDownlinkTest(main - pr):
  pkt_type=sctp, is_next_hop_spine=True, with_psc=True, vlan_conf=untag->untag
  pkt_type=sctp, is_next_hop_spine=True, with_psc=False, vlan_conf=tag->untag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=True, vlan_conf=untag->untag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=True, vlan_conf=tag->tag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=False, vlan_conf=untag->tag
  pkt_type=sctp, is_next_hop_spine=True, with_psc=True, vlan_conf=tag->untag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=True, vlan_conf=untag->tag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=False, vlan_conf=tag->tag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=False, vlan_conf=untag->untag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=True, vlan_conf=tag->untag
  pkt_type=sctp, is_next_hop_spine=False, with_psc=False, vlan_conf=tag->untag
  pkt_type=sctp, is_next_hop_spine=True, with_psc=False, vlan_conf=untag->untag
test.FabricSpgwUplinkRecircTest(main - pr):
  pkt_type=sctp, is_next_hop_spine=True, vlan_conf=untag->untag
  pkt_type=sctp, is_next_hop_spine=False, vlan_conf=untag->tag
  pkt_type=sctp, is_next_hop_spine=False, vlan_conf=tag->tag
  pkt_type=sctp, is_next_hop_spine=False, vlan_conf=untag->untag
  pkt_type=sctp, is_next_hop_spine=False, vlan_conf=tag->untag
  pkt_type=sctp, is_next_hop_spine=True, vlan_conf=tag->untag
test.FabricSpgwDownlinkToDbufTest(main - pr):
  pkt_type=sctp, vlan_conf=untag->tag
  pkt_type=sctp, vlan_conf=tag->untag
  pkt_type=sctp, vlan_conf=untag->untag
  pkt_type=sctp, vlan_conf=tag->tag
test.FabricSpgwDownlinkFromDbufTest(main - pr):
  pkt_type=sctp, vlan_conf=untag->tag
  pkt_type=sctp, vlan_conf=tag->untag
  pkt_type=sctp, vlan_conf=untag->untag
  pkt_type=sctp, vlan_conf=tag->tag

The send_report_to_spine=True issue:

test.FabricSpgwUplinkIntTest(pr - main):
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
test.FabricSpgwDownlinkIntTest(pr - main):
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
test.FabricSpgwIntUplinkDropTest(pr - main):
  drop_reason=133, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=133, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=133, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=133, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=133, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=133, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
test.FabricSpgwIntDownlinkDropTest(pr - main):
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=132, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=134, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
test.FabricIntLocalReportTest(pr - main):
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
test.FabricIntIngressDropReportTest(pr - main):
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  drop_reason=80, is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
test.FabricIntEgressDropReportTest(pr - main):
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=vxlan_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_psc_dl_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=gtp_psc_ul_icmp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=sctp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=gtp_tcp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=untag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True
  is_next_hop_spine=False, vlan_conf=tag->tag, is_device_spine=False, pkt_type=udp, send_report_to_spine=True

Yi-Tseng · 2021-07-22T18:37:05Z

@ccascone Can you take a look at the test coverage in my comment? Just want to make sure it is ok to skip SCP packet type in all SPGW tests. Thanks

ccascone · 2021-07-22T19:30:11Z

I won't be able to do a proper review until Sunday, but if understand your question we don't have a requirement to support GTPU encap/decap of SCTP packets. SCTP is used for control traffic between eNB and MME, UEs are not supposed to generate/receive SCTP.

ccascone · 2021-07-22T19:35:23Z

@Yi-Tseng please feel free to merge this, don't wait for me. If necessary, I can do a post-merge review.

Updating fabric_test.py

2395de3

- Including new function get_test_args - Reads in high level parameters from actual test and generates test test arguments - Makes tests more readable by simplifying repeated code

dariusgrassi changed the title ~~Updating fabric_test.py~~ [WIP] Making PTF tests more readable Jun 2, 2021

dariusgrassi added the WIP Do not merge! label Jun 2, 2021

dariusgrassi changed the title ~~[WIP] Making PTF tests more readable~~ [SDFAB-183] Making PTF tests more readable #110 Jun 2, 2021

dariusgrassi changed the title ~~[SDFAB-183] Making PTF tests more readable #110~~ [SDFAB-183] Making PTF tests more readable Jun 2, 2021

dariusgrassi changed the title ~~[SDFAB-183] Making PTF tests more readable~~ [WIP] Making PTF tests more readable Jun 2, 2021

Beginning updates to get_test_args with new fixes and Yi discussion

0a3457c

redesign - Created one giant for-loop for return dictionaries - Changed appending and returning a list into yielding dictionaries - Populate lists in order of for-loop complexity from top to bottom

Yi-Tseng reviewed Jun 4, 2021

View reviewed changes

ptf/tests/ptf/fabric_test.py Outdated Show resolved Hide resolved

dariusgrassi added 2 commits June 4, 2021 10:15

Implementing Yi syntax recommendation and setting up first test

9e56940

- Changed FabricSpgwDownlinkTest in test.py to match high-level param model - Fixed list naming model to match consistent style

Making minor fixes to actually test if high-level implementation is

e327af7

working for FabricSpgwDownlinkTest - Changed self.get_test_args call to get_test_args (causing error in Jenkins)

dariusgrassi self-assigned this Jun 7, 2021

Yi-Tseng reviewed Jun 14, 2021

View reviewed changes

ptf/tests/ptf/fabric_test.py Outdated Show resolved Hide resolved

Yi-Tseng reviewed Jun 14, 2021

View reviewed changes

Apply suggestions from code review

690e084

Co-authored-by: Yi Tseng <a86487817@gmail.com>

Yi-Tseng reviewed Jun 21, 2021

View reviewed changes

ptf/tests/ptf/fabric.ptf/test.py Outdated Show resolved Hide resolved

Darius Grassi added 8 commits July 6, 2021 11:08

Merge branch 'main' of https://github.com/dariusgrassi/fabric-tna int…

879d093

…o readable-ptf

Cleanup for readable test case and indentation fix

f16f245

Cleanup and remove is_device_spine_list iteration since unchanging

f8d6d81

Fix small bug in gen_test_args for-loop

22bc36f

- Didn't call .items() when looping through vlan_confs dict

Cleanup get_test_args and fix send_report_to_spine

5b7cc54

- Cleanup spacing and remove obvious comments - Remove loops for consistent singular test arguments - Check correct traffic direction devices with const option lists - Yield more test arguments

Change FabricSpgwUplinkTest to readable

4558971

Add allow to get_test_args and rework FabricSpgwUplinkRecircTest

0685b4e

* Added allow list to get_test_args for-loop generation * Added new fields to print statement in for loop * Question about lines 150-156, packet attribute specifications

Stop printing test args when running readable tests

0754265

dariusgrassi changed the title ~~[WIP] Making PTF tests more readable~~ [WIP] Make PTF tests more readable Jul 7, 2021

Darius Grassi added 4 commits July 7, 2021 14:03

Add unique addresses for packets per test since they vary

c84def0

* Modify get_test_args to parse packet addresses and pass in loop * Modify SPGW test cases to include new address specifications

Finalize rework of SPGW tests, push for Jenkins testing

f46d215

Fix send report to spine to fill list as long as INT test

cf3c38b

* Remove consideration of traffic direction from conditional

Add kwargs to doRunTest functions for SPGW dbuf tests

8b4a79b

Yi-Tseng reviewed Jul 8, 2021

View reviewed changes

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

Darius Grassi added 7 commits July 8, 2021 14:55

Implement unique tc_name and cleanup iteration approach

a1f884e

Cleanup and modify parameters for packet getattr() while generating test

50b701f

arguments

Attempt fix for INT test packet assignment method

a0975e7

Cleaning up implementation for INT+SPGW test and bug-fixing

be9d80c

get_test_args

Bug fix NoneType dictionary access for INT tests

776d685

* Only access pkt_addrs dictionary for non-INT tests * Stop printing actual pkt while test is running by adding pkt after print * Change params in INT doRunTest to access tagged1 + tagged2 instead of just tagged

Implement rest of INT SPGW tests

2cae975

* Implement INT SPGW non-drop tests * Implement INT SPGW drop tests

Yi-Tseng reviewed Jul 12, 2021

View reviewed changes

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

Yi-Tseng removed the WIP Do not merge! label Jul 12, 2021

Yi-Tseng changed the title ~~[WIP] Make PTF tests more readable~~ [SDFAB-221] Make PTF tests more readable Jul 12, 2021

Bug fix and implement Yi suggestions

7b120f4

* Move OPTIONS constants from get_test_args to top of file with other constants * Add documentation for get_test_args function * Rename vague allow param to ue_recirculation * Fixed: include **kwargs in doRunTest for INT SPGW tests

Yi-Tseng approved these changes Jul 13, 2021

View reviewed changes

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

ptf/tests/common/fabric_test.py Outdated Show resolved Hide resolved

Clean obvious comments and remove drop_test param from get_test_args

b8c4c78

Yi-Tseng requested a review from ccascone July 14, 2021 01:03

dariusgrassi and others added 8 commits July 19, 2021 10:31

Merge branch 'stratum:main' into readable-ptf

33912e7

Make some changes to address test cases not matching up

c711b6a

* Implement conditional psc flag for SPGW tests * Add some more traffic directions for tests that weren't testing enough device types (i.e., include spine type for destination device for some tests)

Merge branch 'readable-ptf' of https://github.com/dariusgrassi/fabric…

c48867c

…-tna into readable-ptf

Make output for modified tests more readable

d2dc0e0

Improve readability for test argument generation printing

67c5d82

Fix test coverage by modifying readable test traffic directions

b84e692

Add readable tests for some non-SPGW INT tests

e8e6d83

* Some INT tests follow unreadable for-loop structure, convert these to readable format * Change default packet list to include VXLAN packet types

Fix bug when not accounting for spgw_type being default value None for

a57cf4c

INT tests

Merge branch 'main' into readable-ptf

33ea668

Yi-Tseng merged commit d271414 into stratum:main Jul 22, 2021

dariusgrassi deleted the readable-ptf branch July 23, 2021 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SDFAB-221] Make PTF tests more readable #268

[SDFAB-221] Make PTF tests more readable #268

dariusgrassi commented Jun 2, 2021 •

edited

Loading

codecov bot commented Jun 2, 2021 •

edited

Loading

Yi-Tseng Jun 14, 2021

Yi-Tseng Jun 14, 2021

dariusgrassi Jul 6, 2021 •

edited

Loading

Yi-Tseng Jul 7, 2021 •

edited

Loading

Yi-Tseng left a comment •

edited

Loading

Yi-Tseng commented Jul 22, 2021

Yi-Tseng commented Jul 22, 2021

ccascone commented Jul 22, 2021

ccascone commented Jul 22, 2021

[SDFAB-221] Make PTF tests more readable #268

[SDFAB-221] Make PTF tests more readable #268

Conversation

dariusgrassi commented Jun 2, 2021 • edited Loading

codecov bot commented Jun 2, 2021 • edited Loading

Codecov Report

Yi-Tseng Jun 14, 2021

Choose a reason for hiding this comment

Yi-Tseng Jun 14, 2021

Choose a reason for hiding this comment

dariusgrassi Jul 6, 2021 • edited Loading

Choose a reason for hiding this comment

Yi-Tseng Jul 7, 2021 • edited Loading

Choose a reason for hiding this comment

Yi-Tseng left a comment • edited Loading

Choose a reason for hiding this comment

Yi-Tseng commented Jul 22, 2021

Yi-Tseng commented Jul 22, 2021

ccascone commented Jul 22, 2021

ccascone commented Jul 22, 2021

dariusgrassi commented Jun 2, 2021 •

edited

Loading

codecov bot commented Jun 2, 2021 •

edited

Loading

dariusgrassi Jul 6, 2021 •

edited

Loading

Yi-Tseng Jul 7, 2021 •

edited

Loading

Yi-Tseng left a comment •

edited

Loading